feat(outlines): Add support outlines #1269

Aisuko · 2023-11-10T01:28:03Z

Description

This PR fixes #1261

Notes for Reviewers

Please kindly check the demo on Kaggle

Update

This PR has been tested

(outlines) @Aisuko ➜ /workspaces/LocalAI (feat/outlines) $  cd /workspaces/LocalAI ; /usr/bin/env /opt/conda/envs/outlines/bin/python /home/codespace/.vscode-remote/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 59803 -- /workspaces/LocalAI/backend/python/backend_outlines/backend_outlines.py 
Server started. Listening on: localhost:50051
Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 665/665 [00:00<00:00, 943kB/s]
Downloading model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548M/548M [00:16<00:00, 32.5MB/s]
Downloading (…)neration_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 124/124 [00:00<00:00, 603kB/s]
Downloading (…)olve/main/vocab.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.04M/1.04M [00:00<00:00, 5.21MB/s]
Downloading (…)olve/main/merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 2.30MB/s]
Downloading (…)/main/tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 65.7MB/s]
(outlines) @Aisuko ➜ /workspaces/LocalAI (feat/outlines) $  cd /workspaces/LocalAI ; /usr/bin/env /opt/conda/envs/outlines/bin/python /home/codespace/.vscode-remote/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 60939 -- /workspaces/LocalAI/backend/python/backend_outlines/backend_outlines.py 
Server started. Listening on: localhost:50051
(outlines) @Aisuko ➜ /workspaces/LocalAI (feat/outlines) $  cd /workspaces/LocalAI ; /usr/bin/env /opt/conda/envs/outlines/bin/python /home/codespace/.vscode-remote/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 44581 -- /workspaces/LocalAI/backend/python/backend_outlines/test_backend_outlines.py 
...
----------------------------------------------------------------------
Ran 3 tests in 44.882s

OK

Signed commits

Yes, I signed my commits.

mudler · 2023-11-10T07:52:04Z

extra/grpc/backend_outlines/backend_outlines.py

+    def Predict(self, request, context):
+        try:
+            #TODO: stop=["."] is a hard code, we need to make it configurable
+            output=generate.continuation(self.model, stop=["."])(self.prompt)


it would make sense here to reuse the stopwords that are already sent as part of the request

mudler · 2023-11-10T07:52:20Z

great addition @Aisuko !

mudler · 2023-11-10T07:52:56Z

@Aisuko how did you test it? can you give a small example to reproduce it?

Aisuko · 2023-11-10T22:53:22Z

@Aisuko how did you test it? can you give a small example to reproduce it?

Hi, @mudler. I have not tested it. I am working on how to debug it. I will add an example after everything works out.

Signed-off-by: GitHub <noreply@github.com>

Aisuko · 2023-11-15T00:40:36Z

@mudler The PR already rebased from master branch. And I also add the test cases for every functions. The result is in the above comment. Please take a look.

mudler · 2023-11-15T16:30:02Z

Makefile

@@ -395,6 +396,7 @@ prepare-extra-conda-environments:
 	$(MAKE) -C backend/python/huggingface
 	$(MAKE) -C backend/python/vall-e-x
 	$(MAKE) -C backend/python/exllama
+	$(MAKE) -C backend/python/backend_outlines


this needs to be added also in the Dockerfile here:

LocalAI/Dockerfile

Line 153 in 991ecce

## Duplicated from Makefile to avoid having a big layer that's hard to push

mudler · 2023-11-15T16:30:32Z

backend/python/backend_outlines/run.sh

+# get the directory where the bash script is located
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+python $DIR/backend_outlines.py $@


make sure the file has executable bits, and the wrapper needs to be also listed here:

LocalAI/Dockerfile

Line 15 in 991ecce

ENV EXTERNAL_GRPC_BACKENDS="huggingface-embeddings:/build/backend/python/huggingface/run.sh,autogptq:/build/backend/python/autogptq/run.sh,bark:/build/backend/python/bark/run.sh,diffusers:/build/backend/python/diffusers/run.sh,exllama:/build/backend/python/exllama/run.sh,vall-e-x:/build/backend/python/vall-e-x/run.sh,vllm:/build/backend/python/vllm/run.sh"

mudler · 2023-11-15T16:32:39Z

is there maybe a way we can connect more complex features like https://github.com/outlines-dev/outlines#efficient-regex-guided-generation of outlines? I think the real gain would be to have guided output rather then generic output

Signed-off-by: GitHub <noreply@github.com>

Aisuko · 2023-11-15T23:12:28Z

is there maybe a way we can connect more complex features like https://github.com/outlines-dev/outlines#efficient-regex-guided-generation of outlines? I think the real gain would be to have guided output rather then generic output

Agree with you, here the PR only including the basic one. And we can implement this.

However, if we want to connect regex-guided generation, we may need to add a new parameter for regex-guided to PredictOptions. Or there exists any other parameter we can use it.

What's your thought?

rlouf · 2023-11-19T17:27:37Z

Outlines author here, let me know if you need any help!

mudler · 2023-12-17T16:04:51Z

is there maybe a way we can connect more complex features like https://github.com/outlines-dev/outlines#efficient-regex-guided-generation of outlines? I think the real gain would be to have guided output rather then generic output

Agree with you, here the PR only including the basic one. And we can implement this.

However, if we want to connect regex-guided generation, we may need to add a new parameter for regex-guided to PredictOptions. Or there exists any other parameter we can use it.

What's your thought?

maybe we can re-use the grammar field?

But I see that outlines support JSON schema already, and that we have already as we support OpenAI functions, maybe we can re-use that feature and pass down to the backends also the json schema?
https://github.com/outlines-dev/outlines#efficient-json-generation-following-a-pydantic-model

@rlouf is there an example that I can find with JSON schemas? that'd be great so could probably help @Aisuko and me here to speed it up :)

rlouf · 2023-12-17T18:10:44Z

generate.json takes JSON Schemas as well as Pydantic models. I'll add an example in the README soon.

Aisuko changed the title ~~Add support outlines~~ feat(outlines): Add support outlines Nov 10, 2023

Aisuko self-assigned this Nov 10, 2023

Aisuko added the new-backend label Nov 10, 2023

Aisuko requested a review from mudler November 10, 2023 01:30

mudler reviewed Nov 10, 2023

View reviewed changes

Aisuko added the Do-not-merge label Nov 14, 2023

Aisuko force-pushed the feat/outlines branch from b2590a0 to 0423e4c Compare November 15, 2023 00:28

Aisuko removed the Do-not-merge label Nov 15, 2023

Add support outlines

eb80c23

Signed-off-by: GitHub <noreply@github.com>

Aisuko force-pushed the feat/outlines branch from 0423e4c to eb80c23 Compare November 15, 2023 00:40

mudler reviewed Nov 15, 2023

View reviewed changes

Aisuko added 2 commits November 16, 2023 09:51

Merge branch 'mudler:master' into feat/outlines

63049a1

Add build script to Dockerfile

74cf83a

Signed-off-by: GitHub <noreply@github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(outlines): Add support outlines #1269

feat(outlines): Add support outlines #1269

Aisuko commented Nov 10, 2023 •

edited

mudler Nov 10, 2023

Aisuko Nov 15, 2023

mudler commented Nov 10, 2023

mudler commented Nov 10, 2023

Aisuko commented Nov 10, 2023

Aisuko commented Nov 15, 2023

mudler Nov 15, 2023

Aisuko Nov 15, 2023

mudler Nov 15, 2023

Aisuko Nov 15, 2023

mudler commented Nov 15, 2023

Aisuko commented Nov 15, 2023

rlouf commented Nov 19, 2023

mudler commented Dec 17, 2023

rlouf commented Dec 17, 2023

feat(outlines): Add support outlines #1269

Are you sure you want to change the base?

feat(outlines): Add support outlines #1269

Conversation

Aisuko commented Nov 10, 2023 • edited

Update

mudler Nov 10, 2023

Choose a reason for hiding this comment

Aisuko Nov 15, 2023

Choose a reason for hiding this comment

mudler commented Nov 10, 2023

mudler commented Nov 10, 2023

Aisuko commented Nov 10, 2023

Aisuko commented Nov 15, 2023

mudler Nov 15, 2023

Choose a reason for hiding this comment

Aisuko Nov 15, 2023

Choose a reason for hiding this comment

mudler Nov 15, 2023

Choose a reason for hiding this comment

Aisuko Nov 15, 2023

Choose a reason for hiding this comment

mudler commented Nov 15, 2023

Aisuko commented Nov 15, 2023

rlouf commented Nov 19, 2023

mudler commented Dec 17, 2023

rlouf commented Dec 17, 2023

Aisuko commented Nov 10, 2023 •

edited