Langchain Prompt Format

[0xDigest]

I have been working to integrate langchain with basaran and I am encountering an issue that I believe has to do with the prompt format. It seems that when langchain is posting to basaran, the prompt is a list and not a string. For example:

{"prompt": ["Sample data is test data\n\nQuestion: What is sample data?\nHelpful Answer:"], "model": "text-davinci-003", "temperature": 0.1, "max_tokens": 256, "top_p": 1, "frequency_penalty": 0, "presence_penalty": 0, "n": 1, "logit_bias": {}}

returns

{"id":"cmpl-3728b36ca4b7a2aac121df7f","object":"text_completion","created":1685652057,"model":"wizard-vicuna-13B","choices":[{"text":"","index":0,"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":1,"completion_tokens":256,"total_tokens":257}}

It sees the prompt as a single token and doesn't return anything. I am able to replicate the issue by changing the example to use a list. The model seems to take the single (empty?) token and generate text. For instance:

curl http://127.0.0.1/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{ "prompt": ["once upon a time,"], "echo": true }'

returns

{"id":"cmpl-ef0bc647b6de2f4986c728e8","object":"text_completion","created":1685652242,"model":"wizard-vicuna-13B","choices":[{"text":"Ahituv, Nima, 1974-\nIntroduction:","index":0,"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":1,"completion_tokens":17,"total_tokens":18}}

Would you consider this a langchain issue if the openAI API supports the call-- or am I missing something in my basaran setup?

[peakji]

Hi @0xDigest! Thank you for bringing this issue to our attention.

Upon verification, it has been confirmed that the prompt parameter in OpenAI's API can indeed be either a string or a list of strings. Therefore, this issue should be attributed to Basaran, and we will make compatibility adjustments in the upcoming version.

We are currently investigating the behavior of specifying the n parameter and multiple prompt strings in a single request, as it appears to be quite complex.

Additionally, does langchain really send multi-prompt requests when invoking the OpenAI API? If it is only using the list format with single prompt, I guess we can roll out a quick fix. 🤔️

[0xDigest]

Thanks for the quick reply-

Honestly, I've just started with langchain, I haven't seen any instances of multiple values in the prompt list, yet. I did go ahead and patch parse_options to add list to the dtypes tuple and changed the returned value to return the first value if it's a list So far, everything is working well.

In case I'm not being clear:
in __main__.py:

# Allow casting from int to float.
if dtype == float:
    dtypes = (int, float)
# Add list to the allowed dtypes
elif dtype == str:
    dtypes = (str, list)
else:
    dtypes = (dtype,)

# Use custom function to convert string to bool correctly.
if dtype == bool:
    dtype_fn = is_true
else:
    dtype_fn = dtype

# If an option appears in both the query parameters and the request
# body, the former takes precedence.
if key in request.args:
    options[key] = request.args.get(key, dtype(), type=dtype_fn)
elif payload and key in payload and isinstance(payload[key], dtypes):
    # if payload is a list get the first item
    if isinstance(payload[key], list):
        options[key] = str(payload[key][0])
    else:
        options[key] = dtype(payload[key])

[fardeon]

We've added a temporary workaround in v0.18.1: currently only the first prompt in the list will be used, and 400 Bad Request will be returned if the prompt list contains more than one values.

Complete support for multi-prompt requests will be added in the upcoming versions.

[0xDigest]

Thanks for the quick turnaround. I've updated and can confirm this works as expected.

[peakji]

Example responses for some multi-prompt requests:

multi-prompts.txt
multi-prompts-2.txt
multi-prompts-3.txt
multi-prompts-stream.txt
multi-prompts-stream-2.txt
multi-prompts-stream-3.txt