Spacy-LLM code sample produces no output #13132

rkatriel · 2023-11-17T00:45:51Z

Hi,

The code sample below - which is based on an example in Matthew Honnibal's blog "Against LLM maximalism" (https://explosion.ai/blog/against-llm-maximalism) - fails to produce any output. This is surprising given that the pipeline is configured to find the kind of entities present in the sentence being processed.

Note: This is a continuation of Issue #13096 (Spacy-LLM fails with storage not allocated on MPS device).

How to reproduce the behavior

Here is the code:

import spacy

nlp = spacy.blank("en")
nlp.add_pipe("sentencizer")
nlp.add_pipe(
    "llm",
    config={
        "task": {
            "@llm_tasks": "spacy.NER.v1",
            "labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY"
        },
        "model": {
            "@llm_models": "spacy.OpenLLaMA.v1",
            "name": "open_llama_3b"
        },
    },
)

doc = nlp("There's no PyTorch bindings for Go. We just use Microsoft Cognitive Services.")
for ent in doc.ents:
    print(ent.text, ent.label_, ent.sent)

Your Environment

Platform: macOS-12.6-arm64-arm-64bit
Python Version: 3.11.4
spaCy Version: 3.6.1

The text was updated successfully, but these errors were encountered:

rmitsch · 2023-11-20T08:20:37Z

Hi @rkatriel, the difference between your sample and the one from the article is the choice of the model. While we try to make our prompts work on as many models as possible, it's hard to guarantee cross-model compatibility. Especially smaller and older models (both of which applies to open_llama_3b) may not deliver satisfying results. A particular challenge here is that LLMs need to understand the output format, as we otherwise can't parse the result back.

I recommend using a newer/larger model.

rkatriel · 2023-11-22T19:27:56Z

Hi @rmitsch which model in particular do you recommend I try? It can't be bigger than a 7b otherwise it won't fit in memory (or take way too long to run).

rmitsch · 2023-11-23T08:42:51Z

Give Mistral a shot.

rkatriel · 2023-11-24T03:50:49Z

I tried Mistral with the config parameters from https://spacy.io/api/large-language-models:

        "model": {
            "@llm_models": "spacy.Mistral.v1",
            "name": "Mistral-7B-v0.1"
        },

But I'm getting KeyError: 'mistral'. Below is the traceback.

File "/Users/ron/PycharmProjects/AI/OpenAI/spacy-llm-example.py", line 5, in
nlp.add_pipe(
File "/opt/homebrew/lib/python3.11/site-packages/spacy/language.py", line 814, in add_pipe
pipe_component = self.create_pipe(
^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/spacy/language.py", line 702, in create_pipe
resolved = registry.resolve(cfg, validate=validate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/confection/init.py", line 756, in resolve
resolved, _ = cls._make(
^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/confection/init.py", line 805, in _make
filled, _, resolved = cls._fill(
^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/confection/init.py", line 860, in _fill
filled[key], validation[v_key], final[key] = cls._fill(
^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/confection/init.py", line 877, in _fill
getter_result = getter(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/spacy_llm/models/hf/mistral.py", line 90, in mistral_hf
return Mistral(name=name, config_init=config_init, config_run=config_run)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/spacy_llm/models/hf/mistral.py", line 21, in init
super().init(name=name, config_init=config_init, config_run=config_run)
File "/opt/homebrew/lib/python3.11/site-packages/spacy_llm/models/hf/base.py", line 73, in init
self._model = self.init_model()
^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/spacy_llm/models/hf/mistral.py", line 39, in init_model
model = transformers.AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 456, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 957, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 671, in getitem
raise KeyError(key)
KeyError: 'mistral'

Process finished with exit code 1

rmitsch · 2023-11-24T10:47:19Z

Which spacy-llm version are you using? I can't reproduce this locally.

rkatriel · 2023-11-24T19:03:44Z

The version of spacy-llm I was using was 0.6.3. I upgraded to the latest (0.6.4) but still got the same error.

It looks like the problem was actually with the transformers library. I was using an incompatible version (4.30.0). After upgrading to the latest (4.35.2) Mistral loaded cleanly.

But now I'm getting an error encountered earlier while trying to make transformers work with an mps device (#13096):

ValueError: The current 'device_map' had weights offloaded to the disk. Please provide an 'offload_folder' for them. Alternatively, make sure you have 'safetensors' installed if the model you are using offers the weights in this format.

The offload folder - where the model weights we will offloaded - is an optional parameter when initializing Mistral (https://huggingface.co/docs/transformers/main/model_doc/mistral):

checkpoint = 'mistralai/Mistral-7B-v0.1'
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map='auto', offload_folder=offload_folder)

Any ideas how to resolve this?

rmitsch · 2023-11-27T07:59:59Z

You can pass any parameter to a HF model by including it in your config_init:

[components.llm.model]
@llm_models = "spacy.Mistral.v1"
name = "Mistral-7B-v0.1"

[components.llm.model.config_init]
offload_folder = "..."

rkatriel · 2023-11-27T23:39:00Z

Thanks, Raphael. I modified the code accordingly

    config={
        "task": {
            "@llm_tasks": "spacy.NER.v1",
            "labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY"
        },
        "model": {
            "@llm_models": "spacy.Mistral.v1",
            "name": "Mistral-7B-v0.1",
            "config_init": {
                "offload_folder": "."
            }
        },
    },

Now I'm getting a different error when transformers calls the Mac's accelerate package:

TypeError: BFloat16 is not supported on MPS

This is a known issue with Mistral (see https://docs.mistral.ai/quickstart/). The suggestion is to "pass the parameter --dtype half to the Docker command line."

I tried passing --dtype half to the python interpreter but it made no difference.

rmitsch · 2023-11-28T08:11:49Z

I tried passing --dtype half to the python interpreter but it made no difference.

Set torch_dtype = "half" in your config:

"model": {
    "@llm_models": "spacy.Mistral.v1",
    "name": "Mistral-7B-v0.1",
    "config_init": {
        "offload_folder": "."
        "torch_dtype": "half"
    }

Let me know whether that helps.

rkatriel · 2023-11-28T13:59:03Z

Thanks. That did the trick. Now the code runs without errors (albeit slowly, partly due to moderate memory pressure). However, once again no output is produced, perhaps related to the pad token warning. See the console output below.

/opt/homebrew/bin/python3.11 /Users/ron/PycharmProjects/AI/OpenAI/spacy-llm-example.py 
/opt/homebrew/lib/python3.11/site-packages/spacy_llm/models/hf/base.py:133: UserWarning: Couldn't find a CUDA GPU, so the setting 'device_map:auto' will be used, which may result in the LLM being loaded (partly) on the CPU or even the hard disk, which may be slow. Install cuda to be able to load and run the LLM on the GPU instead.
  warnings.warn(
Loading checkpoint shards: 100%|██████████| 2/2 [00:34<00:00, 17.34s/it]
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

Process finished with exit code 0

rmitsch · 2023-11-28T14:04:40Z

Log the raw responses by setting save_io:

    config={
        "task": {
            "@llm_tasks": "spacy.NER.v1",
            "labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY"
        },
        "model": {
            "@llm_models": "spacy.Mistral.v1",
            "name": "Mistral-7B-v0.1",
            "config_init": {
                "offload_folder": "."
                "dtype": "half,
            }
        },
        "save_io": True
    },

You can access the response in doc.user_data["llm_io"]["response"]. Let me know what the LLM response is.

rkatriel · 2023-11-28T14:56:42Z

I got back {}

rmitsch · 2023-12-06T09:34:36Z

When I run this, the response is "\n\nText:\n'''\nWe use the following open source libraries:\n\n* TensorFlow". I. e. Mistral kinda understands part of the task, but doesn't respond conforming to the output conventions specified in our NER prompt.

I recommend to use the latest version of the NER recipe (spacy.NER.v3, this way "PyTorch" is recognized as OS library) and set label_definitions in your config:

"task": {
    "@llm_tasks": "spacy.NER.v3",
    "labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY",
    "label_definitions": {"SAAS_PLATFORM": ..., }
},

That will make it easier for the LLM. Unfortunately we can't guarantee that all (OS) models understand all prompts properly.

rkatriel · 2023-12-06T14:06:45Z

@rmitsch Thanks, I followed your advice but unfortunately it made no difference. I also tried replacing "spacy.Mistral.v1" with "spacy.OpenLLaMA.v1" and "spacy.StableLM.v1" but the response from the LLM is always empty ({}) so it seems the issue is not specific to a particular OS model. It would be great to have this simple example work for at least one HuggingFace model.

kemalcanbora · 2024-03-17T18:23:41Z

Thanks, Raphael. I modified the code accordingly
    config={
        "task": {
            "@llm_tasks": "spacy.NER.v1",
            "labels": "SAAS_PLATFORM,PROGRAMMING_LANGUAGE,OPEN_SOURCE_LIBRARY"
        },
        "model": {
            "@llm_models": "spacy.Mistral.v1",
            "name": "Mistral-7B-v0.1",
            "config_init": {
                "offload_folder": "."
            }
        },
    },
Now I'm getting a different error when transformers calls the Mac's accelerate package:

TypeError: BFloat16 is not supported on MPS

This is a known issue with Mistral (see https://docs.mistral.ai/quickstart/). The suggestion is to "pass the parameter --dtype half to the Docker command line."

I tried passing --dtype half to the python interpreter but it made no difference.

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-3b"
config_init = {"device": "cpu"}

rkatriel · 2024-03-17T22:20:54Z

Hi @kemalcanbora, thanks for the followup! It's been a while since this thread has seen activity. I tried your suggestion and, while it ran cleanly (except for a warning) though very slowly (10 minutes on my MacBook Pro M2), the result is the same as before: an empty response set (see the console trace below). Based on what I've seen to date, I believe the issue is with the Spacy framework itself, not the specific LLM being used. It would be nice to see this simple example work!

/opt/homebrew/lib/python3.11/site-packages/spacy_llm/models/hf/base.py:133: UserWarning: Couldn't find a CUDA GPU, so the setting 'device_map:auto' will be used, which may result in the LLM being loaded (partly) on the CPU or even the hard disk, which may be slow. Install cuda to be able to load and run the LLM on the GPU instead.
  warnings.warn(
Downloading config.json: 100%|██████████| 819/819 [00:00<00:00, 3.64MB/s]
Downloading instruct_pipeline.py: 100%|██████████| 9.16k/9.16k [00:00<00:00, 65.9MB/s]
A new version of the following files was downloaded from https://huggingface.co/databricks/dolly-v2-3b:
- instruct_pipeline.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Downloading pytorch_model.bin: 100%|██████████| 5.68G/5.68G [02:30<00:00, 37.8MB/s]
Downloading tokenizer_config.json: 100%|██████████| 450/450 [00:00<00:00, 656kB/s]
Downloading tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 6.69MB/s]
Downloading (…)cial_tokens_map.json: 100%|██████████| 228/228 [00:00<00:00, 1.09MB/s]
response = {}

Process finished with exit code 0

rkatriel mentioned this issue Nov 17, 2023

Spacy-LLM fails with storage not allocated on MPS device #13096

Closed

adrianeboyd added the feat/llm Feature: LLMs (incl. spacy-llm) label Nov 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spacy-LLM code sample produces no output #13132

Spacy-LLM code sample produces no output #13132

rkatriel commented Nov 17, 2023

rmitsch commented Nov 20, 2023

rkatriel commented Nov 22, 2023

rmitsch commented Nov 23, 2023

rkatriel commented Nov 24, 2023 •

edited

rmitsch commented Nov 24, 2023

rkatriel commented Nov 24, 2023 •

edited

rmitsch commented Nov 27, 2023 •

edited

rkatriel commented Nov 27, 2023 •

edited

rmitsch commented Nov 28, 2023

rkatriel commented Nov 28, 2023 •

edited

rmitsch commented Nov 28, 2023 •

edited

rkatriel commented Nov 28, 2023

rmitsch commented Dec 6, 2023 •

edited

rkatriel commented Dec 6, 2023 •

edited

kemalcanbora commented Mar 17, 2024

rkatriel commented Mar 17, 2024

Spacy-LLM code sample produces no output #13132

Spacy-LLM code sample produces no output #13132

Comments

rkatriel commented Nov 17, 2023

How to reproduce the behavior

Your Environment

rmitsch commented Nov 20, 2023

rkatriel commented Nov 22, 2023

rmitsch commented Nov 23, 2023

rkatriel commented Nov 24, 2023 • edited

rmitsch commented Nov 24, 2023

rkatriel commented Nov 24, 2023 • edited

rmitsch commented Nov 27, 2023 • edited

rkatriel commented Nov 27, 2023 • edited

rmitsch commented Nov 28, 2023

rkatriel commented Nov 28, 2023 • edited

rmitsch commented Nov 28, 2023 • edited

rkatriel commented Nov 28, 2023

rmitsch commented Dec 6, 2023 • edited

rkatriel commented Dec 6, 2023 • edited

kemalcanbora commented Mar 17, 2024

rkatriel commented Mar 17, 2024

rkatriel commented Nov 24, 2023 •

edited

rkatriel commented Nov 24, 2023 •

edited

rmitsch commented Nov 27, 2023 •

edited

rkatriel commented Nov 27, 2023 •

edited

rkatriel commented Nov 28, 2023 •

edited

rmitsch commented Nov 28, 2023 •

edited

rmitsch commented Dec 6, 2023 •

edited

rkatriel commented Dec 6, 2023 •

edited