Available Parameters

Hi, what’s the roadmap looking for in terms of allowing > 1 completions so we don’t have to resend the same input multiple times to get some choice in possible outputs? Also, any chance on getting logit bias for helping to manipulate text? (e.g. if I’ve got a few shot that has a stop token at the end and I want to logit bias out the stop token to force the model to continue generating even if the stop token coming next is really likely)

2 Likes

Hi Matt!
The more-than-1-completion feature is in the works, and should be done very soon, in a few weeks at most.
We haven’t started implementing logit bias yet, but can dedicate cycles to it. Are you interested in that functionality because you want the generations to not end at the stop token, unless the stop token is exceptionally likely to come next?

2 Likes

On the logit bias, there’s 2 widely used useful cases. (probably more but these are the ones I’m thinking of off the top of my head)

  1. is giving instructions. E.g. either few shot or fine tune,
    {uniquetoken} tag {uniquetoken} for guiding generation, we’d want to logitbias out the unqiue token so text doesn’t generate the instructions.

  2. example1: …{uniquetoken}
    example2: …{uniquetoken}
    example3: …{uniquetoken}
    targetCompletion: …
    by logit biasing out the unique token we can force it to continue the section it’s in (e.g. suppose {uniquetoken} is \n, then it never completes the line).

Got it, thanks for explaining the use cases! So, you’re looking for a way to specify a token to ignore in the sampling algorithm, right? We’ll incorporate this feedback into our feature backlog and circle back when we can confirm a timeline!

yep, that’s exactly what we’re looking for.

3 Likes

Hey @mattbrockman!

Our multi-gen feature is now live. Let us know if you have any additional questions.

1 Like