Install via requirements.txt documentation doesn't work

[SHxKM]

The docs state I can specify the model like this in requirements.txt:

spacy>=3.0.0,<4.0.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl

This attempts to download spacy 3.4.4.

And a VERY long series of exceptions like this is raised:

45.29             thinc/backends/numpy_ops.cpp:2408:34: error: ‘_PyCFrame’ {aka ‘struct _PyCFrame’} has no member named ‘use_tracing’
45.29              2408 |      (unlikely((tstate)->cframe->use_tracing) &&\
45.29                   |                                  ^~~~~~~~~~~
45.29             thinc/backends/numpy_ops.cpp:1001:43: note: in definition of macro ‘unlikely’
45.29              1001 |   #define unlikely(x) __builtin_expect(!!(x), 0)
45.29                   |                                           ^
45.29             thinc/backends/numpy_ops.cpp:2513:15: note: in expansion of macro ‘__Pyx_IsTracing’
45.29              2513 |           if (__Pyx_IsTracing(tstate, 0, 0)) {\
45.29                   |               ^~~~~~~~~~~~~~~
45.29             thinc/backends/numpy_ops.cpp:46877:3: note: in expansion of macro ‘__Pyx_TraceReturn’
45.29             46877 |   __Pyx_TraceReturn(Py_None, 1);
45.29                   |   ^~~~~~~~~~~~~~~~~

All my attempts so far to cache Spacy's download by specifying the model in requirements.txt have failed (see replies below).

Context: I'm installing this in Docker. Python image is python:3.12.1-slim

[SHxKM]

Another failed attempt:

spacy>=3.7.2,<3.8.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

Returns:

36.32 ERROR: Cannot install en-core-web-sm 3.7.1 (from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl) because these package versions have conflicting dependencies.
36.32
36.32 The conflict is caused by:
36.32     The user requested en-core-web-sm 3.7.1 (from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl)
36.32     The user requested (constraint) en-core-web-sm
36.32
36.32 To fix this you could try to:
36.32 1. loosen the range of package versions you've specified
36.32 2. remove package versions to allow pip attempt to solve the dependency conflict
36.32
36.32 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
36.60

Same for:

spacy>=3.7.2,<3.8.0
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

Output:

37.93 Requirement already satisfied: spacy<3.8.0,>=3.7.2 in /home/python/.local/lib/python3.12/site-packages (from -r requirements.txt (line 18)) (3.7.2)
37.93 ERROR: Cannot install en-core-web-sm 3.7.1 (from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl) because these package versions have conflicting dependencies.
37.93
37.93 The conflict is caused by:
37.93     The user requested en-core-web-sm 3.7.1 (from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl)
37.93     The user requested (constraint) en-core-web-sm
37.93
37.93 To fix this you could try to:
37.93 1. loosen the range of package versions you've specified
37.93 2. remove package versions to allow pip attempt to solve the dependency conflict
37.93
37.93 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
38.23

[remiBoudreau]

@SHxKM Seems Cython does not have a stable release compatible with C Python ABI for v3.12. It still works with v3.11

[SHxKM]

@SHxKM Seems Cython does not have a stable release compatible with C Python ABI for v3.12. It still works with v3.11

Thanks for taking a look. Is there something I can do or should I wait for a new Python version?

[danieldk]

As you said, this tries to install spaCy 3.4.4. spaCy 3.7.0 is the first version that supports Python 3.12.

Could you try to download a recent model release? E.g. https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

[danieldk]

We will update the example in the docs.

[SHxKM]

Why do you want to install a spaCy 3.4.0 model? As you said, this tries to install spaCy 3.4.4. spaCy 3.7.0 is the first version that supports Python 3.12. Could you try to download a recent model release? E.g. https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

@danieldk I don’t. I just showed the example from the docs. I’ve tried Spacy 3.7.2 and the model 3.7.1 - it also fails.

[SHxKM]

@danieldk see my attempt here

[danieldk]

@danieldk see my attempt here

Odd, that works fine here. In a clean 3.12.1 venv:

% cat requirements.txt
spacy>=3.7.2,<3.8.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl
% pip install -r requirements.txt
...
Successfully installed MarkupSafe-2.1.4 annotated-types-0.6.0 blis-0.7.11 catalogue-2.0.10 certifi-2024.2.2 charset-normalizer-3.3.2 click-8.1.7 cloudpathlib-0.16.0 confection-0.1.4 cymem-2.0.8 en_core_web_sm-3.7.1 idna-3.6 jinja2-3.1.3 langcodes-3.3.0 murmurhash-1.0.10 numpy-1.26.3 packaging-23.2 preshed-3.0.9 pydantic-2.6.0 pydantic-core-2.16.1 requests-2.31.0 setuptools-69.0.3 smart-open-6.4.0 spacy-3.7.2 spacy-legacy-3.0.12 spacy-loggers-1.0.5 srsly-2.4.8 thinc-8.2.2 tqdm-4.66.1 typer-0.9.0 typing-extensions-4.9.0 urllib3-2.2.0 wasabi-1.1.2 weasel-0.3.4

[SHxKM]

@danieldk hmm, I’d suspected that this could be related to the fact I’m doing this inside a Docker container. Or maybe I didn’t try this exact combination. Will do in a bit and update.

[SHxKM]

Hmm @danieldk if you say this is working on 3.12.1 then it must be something about running it inside a Dockerfile:

This is what I get:

requirements.txt:

...
spacy>=3.7.2,<3.8.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

Output:

39.66 The conflict is caused by:
39.66     The user requested en-core-web-sm 3.7.1 (from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl)
39.66     The user requested (constraint) en-core-web-sm
39.66
39.66 To fix this you could try to:
39.66 1. loosen the range of package versions you've specified
39.66 2. remove package versions to allow pip attempt to solve the dependency conflict
39.66
39.66 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
39.90

I don't know how it might relate, but this is the base Python image I'm using:

FROM python:3.12.1-slim AS app

[danieldk]

Do you happen to have a small Dockerfile with which we could try to reproduce the issue?

[SHxKM]

Do you happen to have a small Dockerfile with which we could try to reproduce the issue?

It isn’t small by any means. I’m gonna try picking it apart step by step. Maybe start with an older Python base image first.

But the Dockerfile is pretty this.

[SHxKM]

Results so far:

• This is not about the Python base image version
• This is not about conflicting versions, a simple Dockerfile like this:

FROM python:3.12.1-slim-bookworm AS app
WORKDIR .
COPY requirements.txt requirements.txt
RUN pip3 install --no-warn-script-location --no-cache-dir --user -r requirements.txt

Succeeds with all my original requirements (including Spacy).

[SHxKM]

OK @danieldk , I can reproduce it with this Dockerfile:

FROM python:3.12.1-slim-bookworm AS app
WORKDIR .
COPY requirements.txt requirements.txt
RUN pip3 install --no-warn-script-location --no-cache-dir --user -r requirements.txt
# Succeeds if we omit these two lines:
RUN pip3 freeze --user > requirements-lock.txt
RUN pip3 install --no-warn-script-location --no-cache-dir --user \
  -r requirements.txt -c requirements-lock.txt

Basically the "offending line" is trying to produce a requirements-lock.txt file, basically we can't constrain versions.

[SHxKM]

In fact this has nothing to do with Docker, this is purely because a constraint file cannot be generated when this is how the requirements.txt file is laid out:

spacy>=3.7.2,<3.8.0
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

This is because en_core_web_sm does not specify a version, as I understand it.

[danieldk]

Thanks for the example, we'll look into it!

[SHxKM]

Thanks for the example, we'll look into it!

Cheers, if it's possible please update this thread so I can get a ping