Spacy-LLM code sample produces no output #13132
Comments
|
Hi @rkatriel, the difference between your sample and the one from the article is the choice of the model. While we try to make our prompts work on as many models as possible, it's hard to guarantee cross-model compatibility. Especially smaller and older models (both of which applies to I recommend using a newer/larger model. |
|
Hi @rmitsch which model in particular do you recommend I try? It can't be bigger than a 7b otherwise it won't fit in memory (or take way too long to run). |
|
Give Mistral a shot. |
|
I tried Mistral with the config parameters from https://spacy.io/api/large-language-models: But I'm getting KeyError: 'mistral'. Below is the traceback. File "/Users/ron/PycharmProjects/AI/OpenAI/spacy-llm-example.py", line 5, in Process finished with exit code 1 |
|
Which |
|
The version of spacy-llm I was using was 0.6.3. I upgraded to the latest (0.6.4) but still got the same error. It looks like the problem was actually with the transformers library. I was using an incompatible version (4.30.0). After upgrading to the latest (4.35.2) Mistral loaded cleanly. But now I'm getting an error encountered earlier while trying to make transformers work with an mps device (#13096):
The offload folder - where the model weights we will offloaded - is an optional parameter when initializing Mistral (https://huggingface.co/docs/transformers/main/model_doc/mistral): Any ideas how to resolve this? |
|
You can pass any parameter to a HF model by including it in your [components.llm.model]
@llm_models = "spacy.Mistral.v1"
name = "Mistral-7B-v0.1"
[components.llm.model.config_init]
offload_folder = "..." |
|
Thanks, Raphael. I modified the code accordingly Now I'm getting a different error when transformers calls the Mac's accelerate package:
This is a known issue with Mistral (see https://docs.mistral.ai/quickstart/). The suggestion is to "pass the parameter --dtype half to the Docker command line." I tried passing --dtype half to the python interpreter but it made no difference. |
Set "model": {
"@llm_models": "spacy.Mistral.v1",
"name": "Mistral-7B-v0.1",
"config_init": {
"offload_folder": "."
"torch_dtype": "half"
}Let me know whether that helps. |
|
Thanks. That did the trick. Now the code runs without errors (albeit slowly, partly due to moderate memory pressure). However, once again no output is produced, perhaps related to the pad token warning. See the console output below. |
|
Log the raw responses by setting config={
"task": {
"@llm_tasks": "spacy.NER.v1",
"labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY"
},
"model": {
"@llm_models": "spacy.Mistral.v1",
"name": "Mistral-7B-v0.1",
"config_init": {
"offload_folder": "."
"dtype": "half,
}
},
"save_io": True
},You can access the response in |
|
I got back {} |
|
When I run this, the response is I recommend to use the latest version of the NER recipe ( "task": {
"@llm_tasks": "spacy.NER.v3",
"labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY",
"label_definitions": {"SAAS_PLATFORM": ..., }
},That will make it easier for the LLM. Unfortunately we can't guarantee that all (OS) models understand all prompts properly. |
|
@rmitsch Thanks, I followed your advice but unfortunately it made no difference. I also tried replacing "spacy.Mistral.v1" with "spacy.OpenLLaMA.v1" and "spacy.StableLM.v1" but the response from the LLM is always empty ({}) so it seems the issue is not specific to a particular OS model. It would be great to have this simple example work for at least one HuggingFace model. |
[components.llm.model] |
|
Hi @kemalcanbora, thanks for the followup! It's been a while since this thread has seen activity. I tried your suggestion and, while it ran cleanly (except for a warning) though very slowly (10 minutes on my MacBook Pro M2), the result is the same as before: an empty response set (see the console trace below). Based on what I've seen to date, I believe the issue is with the Spacy framework itself, not the specific LLM being used. It would be nice to see this simple example work! |
Hi,
The code sample below - which is based on an example in Matthew Honnibal's blog "Against LLM maximalism" (https://explosion.ai/blog/against-llm-maximalism) - fails to produce any output. This is surprising given that the pipeline is configured to find the kind of entities present in the sentence being processed.
Note: This is a continuation of Issue #13096 (Spacy-LLM fails with storage not allocated on MPS device).
How to reproduce the behavior
Here is the code:
Your Environment
Platform: macOS-12.6-arm64-arm-64bit
Python Version: 3.11.4
spaCy Version: 3.6.1
The text was updated successfully, but these errors were encountered: