Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: ModuleNotFoundError: No module named 'redis' #1979

Closed
1 task done
wannabewayno opened this issue Jun 13, 2023 · 16 comments · Fixed by #2056 · May be fixed by #2024
Closed
1 task done

[Bug]: ModuleNotFoundError: No module named 'redis' #1979

wannabewayno opened this issue Jun 13, 2023 · 16 comments · Fixed by #2056 · May be fixed by #2024
Labels
bug Something isn't working

Comments

@wannabewayno
Copy link

Version

v0.5.2

Platform

Python

What happened?

Hey team!
I love bullmq and found this little guy checking out the python prototype.

This happened installing bullmq inside a docker container:
python v3.10.12
bullmq v0.5.2 (latest)

Error: ModuleNotFoundError: No module named 'redis'

I can confirm that Installing 0.4.3 and then installing 0.5.2 works just fine, so it must be a dependency or requirement issue

I would submit a pull request myself however I am unfamiliar with python and it's package manager, so I feel someone else might be better suited.

How to reproduce.

Just run pip install bullmq on a system that has no prior bullmq installed before.

Relevant log output

------- original error, straight up install of 0.5.2 --------
root@a8b103d9200a:/usr/src/app# pip install bullmq
Collecting bullmq
  Downloading bullmq-0.5.2.tar.gz (34 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-9li_njj4/bullmq_206a44b6f8334b358eecdbf61726f675/setup.py", line 2, in <module>
          from bullmq import __version__
        File "/tmp/pip-install-9li_njj4/bullmq_206a44b6f8334b358eecdbf61726f675/bullmq/__init__.py", line 10, in <module>
          from bullmq.queue import Queue
        File "/tmp/pip-install-9li_njj4/bullmq_206a44b6f8334b358eecdbf61726f675/bullmq/queue.py", line 1, in <module>
          from bullmq.redis_connection import RedisConnection
        File "/tmp/pip-install-9li_njj4/bullmq_206a44b6f8334b358eecdbf61726f675/bullmq/redis_connection.py", line 1, in <module>
          import redis.asyncio as redis
      ModuleNotFoundError: No module named 'redis'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details


------ installing 0.4.3 then 0.5.2 ---------
root@a8b103d9200a:/usr/src/app# pip install bullmq==0.4.3
Collecting bullmq==0.4.3
  Downloading bullmq-0.4.3.tar.gz (34 kB)
  Preparing metadata (setup.py) ... done
Collecting redis (from bullmq==0.4.3)
  Downloading redis-4.5.5-py3-none-any.whl (240 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.3/240.3 kB 4.1 MB/s eta 0:00:00
Collecting msgpack (from bullmq==0.4.3)
  Downloading msgpack-1.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (316 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.8/316.8 kB 939.3 kB/s eta 0:00:00
Collecting semver (from bullmq==0.4.3)
  Downloading semver-3.0.0-py3-none-any.whl (17 kB)
Collecting async-timeout>=4.0.2 (from redis->bullmq==0.4.3)
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Building wheels for collected packages: bullmq
  Building wheel for bullmq (setup.py) ... done
  Created wheel for bullmq: filename=bullmq-0.4.3-py3-none-any.whl size=55945 sha256=ab7b0188c9c19e7e87e931c5a7a30f2832d624c966603c13ed06ba492f33965f
  Stored in directory: /root/.cache/pip/wheels/b5/51/2b/e10d01cc75c551df4dca7a63b5afdd5e05630e69c15e7d7b36
Successfully built bullmq
Installing collected packages: msgpack, semver, async-timeout, redis, bullmq
Successfully installed async-timeout-4.0.2 bullmq-0.4.3 msgpack-1.0.5 redis-4.5.5 semver-3.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@a8b103d9200a:/usr/src/app# pip install bullmq==0.5.2
Collecting bullmq==0.5.2
  Using cached bullmq-0.5.2.tar.gz (34 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: redis in /usr/local/lib/python3.10/dist-packages (from bullmq==0.5.2) (4.5.5)
Requirement already satisfied: msgpack in /usr/local/lib/python3.10/dist-packages (from bullmq==0.5.2) (1.0.5)
Requirement already satisfied: semver in /usr/local/lib/python3.10/dist-packages (from bullmq==0.5.2) (3.0.0)
Requirement already satisfied: async-timeout>=4.0.2 in /usr/local/lib/python3.10/dist-packages (from redis->bullmq==0.5.2) (4.0.2)
Building wheels for collected packages: bullmq
  Building wheel for bullmq (setup.py) ... done
  Created wheel for bullmq: filename=bullmq-0.5.2-py3-none-any.whl size=57620 sha256=7e337be7f32c8414699486bc4c770dc9fb55fd4541aa3cc91b84862e73b3cef9
  Stored in directory: /root/.cache/pip/wheels/94/1e/e4/df756b7ed58eaa4fc25f4ae0f5fcc146b2d04fe7fe1db3c519
Successfully built bullmq
Installing collected packages: bullmq
  Attempting uninstall: bullmq
    Found existing installation: bullmq 0.4.3
    Uninstalling bullmq-0.4.3:
      Successfully uninstalled bullmq-0.4.3
Successfully installed bullmq-0.5.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Code of Conduct

  • I agree to follow this project's Code of Conduct
@wannabewayno wannabewayno added the bug Something isn't working label Jun 13, 2023
@manast
Copy link
Contributor

manast commented Jun 13, 2023

@roggervalf do you know why this error happens?

@roggervalf
Copy link
Collaborator

this is weird, as I can see redis in install_requires https://github.com/taskforcesh/bullmq/blob/master/python/setup.py#L25

@wannabewayno
Copy link
Author

Yeah, I saw that it was listed as a dependency and previous versions install the child dependency just fine. Weird.

@manast
Copy link
Contributor

manast commented Jun 14, 2023

Could it be that semantic_release is not properly copying the requirements file?

@vraj-fyno
Copy link

I am also getting the same error while installing it in docker
Is there any resolution to this @manast ?

@manast
Copy link
Contributor

manast commented Jun 23, 2023

@vraj-fyno does it only fail when building the docker image?

@wannabewayno
Copy link
Author

wannabewayno commented Jun 23, 2023

Apologies for closing and re-opening, I fat fingered the close button on mobile whilst on public transport.

The issue is still persists for me.
It fails for me:

  • Building the docker image
  • Running an interactive container and installing with ‘pip install bullmq’
  • Installing in a venv via ‘pip install bullmq’

the issue is still present in v1.0.0

installing redis via pip then installing bullmq via pip doesn’t work.

Installing bullmq v0.4.3 via pip then installing bullmq v1.0.0 via pip works as expected.

@vraj-fyno
Copy link

@manast As mentioned by @wannabewayno it's correct

Although one correction, if you do pip install bullmq --upgrade, it fails again

(venv) vraj@Vraj-s-Macbook OptimizingNetwork % pip install bullmq --upgrade
Requirement already satisfied: bullmq in ./venv/lib/python3.11/site-packages (0.4.3)
Collecting bullmq
  Using cached bullmq-1.0.0.tar.gz (36 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/kh/zv_9jsbj34dcggyz7_zx0l5r0000gn/T/pip-install-sqwo9k_1/bullmq_ce4f157779e549f5bd7aeef0541a5ed4/setup.py", line 2, in <module>
          from bullmq import __version__
        File "/private/var/folders/kh/zv_9jsbj34dcggyz7_zx0l5r0000gn/T/pip-install-sqwo9k_1/bullmq_ce4f157779e549f5bd7aeef0541a5ed4/bullmq/__init__.py", line 10, in <module>
          from bullmq.queue import Queue
        File "/private/var/folders/kh/zv_9jsbj34dcggyz7_zx0l5r0000gn/T/pip-install-sqwo9k_1/bullmq_ce4f157779e549f5bd7aeef0541a5ed4/bullmq/queue.py", line 1, in <module>
          from bullmq.redis_connection import RedisConnection
        File "/private/var/folders/kh/zv_9jsbj34dcggyz7_zx0l5r0000gn/T/pip-install-sqwo9k_1/bullmq_ce4f157779e549f5bd7aeef0541a5ed4/bullmq/redis_connection.py", line 1, in <module>
          import redis.asyncio as redis
      ModuleNotFoundError: No module named 'redis.asyncio'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Let me know if you need more details, are you not able to reproduce the issue?

@stephenhuh
Copy link

having the same issue here

@stephenhuh
Copy link

stephenhuh commented Jun 25, 2023

interestingly enough, when i run pip install redis before running pip install -r requirements.txt the error seems to go away, and starts complaining about other errors.

so it starts complaining about semver with the same error, then msgpack but ultimately when i set my build command to pip install redis; pip install msgpack; pip install semver; pip install -r requirements.txt it works.

so most likely those requirements are nto actualyl being loaded into bullmqs own modules although pip seems to be recognizing it.

--

worth saying i only ran into these issues in a deploy environment.

@manast
Copy link
Contributor

manast commented Jun 25, 2023

It would be nice if somebody more knowledgable than me on how pip handles dependencies could shed some light on why this happens 🤔

@stephenhuh
Copy link

stephenhuh commented Jun 25, 2023

this might be the culprit @manast haha literally just got these logs (though i could be wrong, may be helpful)

Jun 25 06:23:03 PM    Attempting uninstall: redis
Jun 25 06:23:03 PM      Found existing installation: redis 4.6.0
Jun 25 06:23:03 PM      Uninstalling redis-4.6.0:
Jun 25 06:23:03 PM        Successfully uninstalled redis-4.6.0
Jun 25 06:23:08 PM    DEPRECATION: bullmq is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Jun 25 06:23:08 PM    Running setup.py install for bullmq: started
Jun 25 06:23:08 PM    Running setup.py install for bullmq: finished with status 'done'
Jun 25 06:23:21 PM  Successfully installed Cython-0.29.35 bpy-3.5.0 bullmq-1.2.0 certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 numpy-1.25.0 redis-4.5.5 requests-2.31.0 urllib3-2.0.3 zstandard-0.21.0

@vraj-fyno
Copy link

@manast Can we fix this and update pip?

@roggervalf
Copy link
Collaborator

hey guys, could you please try to upgrade your pip:
pip3 install --upgrade pip
and install last version 1.3.1

@vraj-fyno
Copy link

vraj-fyno commented Jun 29, 2023

@roggervalf I was able to make it work

Here are few observations

  • It doesn't work in Python 3.8 and lower, I had to upgrade the docker image to
python:latest
  • In my requirements.txt I had

  • redis==3.5.3

  • bullmq

  • I think bullmq was not compatible with the above redis. So BullMQ ran it the below error

 => [internal] load build context                                                                                                                                                                     0.2s
 => => transferring context: 706.00kB                                                                                                                                                                 0.2s
 => CACHED [2/9] RUN mkdir -p /usr/src/app                                                                                                                                                            0.0s
 => CACHED [3/9] WORKDIR /usr/src/app                                                                                                                                                                 0.0s
 => CACHED [4/9] RUN pip3 install python-dotenv                                                                                                                                                       0.0s
 => [5/9] COPY ./requirements.txt ./                                                                                                                                                                  0.0s
 => ERROR [6/9] RUN pip3 install -r requirements.txt                                                                                                                                                  6.4s
------
 > [6/9] RUN pip3 install -r requirements.txt:
#14 0.430 Collecting redis==3.5.3 (from -r requirements.txt (line 1))
#14 0.588   Downloading redis-3.5.3-py2.py3-none-any.whl (72 kB)
#14 0.632      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.1/72.1 kB 1.6 MB/s eta 0:00:00
#14 0.875 Collecting pymongo==4.3.3 (from -r requirements.txt (line 2))
#14 0.918   Downloading pymongo-4.3.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (495 kB)
#14 0.993      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 495.1/495.1 kB 6.7 MB/s eta 0:00:00
#14 0.996 Requirement already satisfied: pip==23.1.2 in /usr/local/lib/python3.11/site-packages (from -r requirements.txt (line 3)) (23.1.2)
#14 0.996 Requirement already satisfied: python-dotenv in /usr/local/lib/python3.11/site-packages (from -r requirements.txt (line 4)) (1.0.0)
#14 1.054 Collecting mysql-connector-python (from -r requirements.txt (line 5))
#14 1.087   Downloading mysql_connector_python-8.0.33-py2.py3-none-any.whl (390 kB)
#14 1.101      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 390.1/390.1 kB 30.3 MB/s eta 0:00:00
#14 1.138 Collecting requests (from -r requirements.txt (line 6))
#14 1.169   Downloading requests-2.31.0-py3-none-any.whl (62 kB)
#14 1.173      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 29.0 MB/s eta 0:00:00
#14 1.353 Collecting boto3 (from -r requirements.txt (line 7))
#14 1.388   Downloading boto3-1.26.163-py3-none-any.whl (135 kB)
#14 1.394      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.9/135.9 kB 26.6 MB/s eta 0:00:00
#14 1.417 Collecting pybars3 (from -r requirements.txt (line 8))
#14 1.454   Downloading pybars3-0.9.7.tar.gz (29 kB)
#14 1.459   Preparing metadata (setup.py): started
#14 1.707   Preparing metadata (setup.py): finished with status 'done'
#14 1.729 Collecting bullmq (from -r requirements.txt (line 9))
#14 1.764   Downloading bullmq-1.3.1.tar.gz (37 kB)
#14 1.774   Installing build dependencies: started
#14 2.709   Installing build dependencies: finished with status 'done'
#14 2.710   Getting requirements to build wheel: started
#14 2.824   Getting requirements to build wheel: finished with status 'done'
#14 2.825   Installing backend dependencies: started
#14 3.319   Installing backend dependencies: finished with status 'done'
#14 3.319   Preparing metadata (pyproject.toml): started
#14 3.431   Preparing metadata (pyproject.toml): finished with status 'done'
#14 3.462 Collecting dnspython<3.0.0,>=1.16.0 (from pymongo==4.3.3->-r requirements.txt (line 2))
#14 3.493   Downloading dnspython-2.3.0-py3-none-any.whl (283 kB)
#14 3.500      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 283.7/283.7 kB 47.1 MB/s eta 0:00:00
#14 3.640 Collecting protobuf<=3.20.3,>=3.11.0 (from mysql-connector-python->-r requirements.txt (line 5))
#14 3.674   Downloading protobuf-3.20.3-py2.py3-none-any.whl (162 kB)
#14 3.679      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 35.7 MB/s eta 0:00:00
#14 3.735 Collecting charset-normalizer<4,>=2 (from requests->-r requirements.txt (line 6))
#14 3.768   Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (193 kB)
#14 3.776      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.0/194.0 kB 35.7 MB/s eta 0:00:00
#14 3.808 Collecting idna<4,>=2.5 (from requests->-r requirements.txt (line 6))
#14 3.838   Downloading idna-3.4-py3-none-any.whl (61 kB)
#14 3.841      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 31.7 MB/s eta 0:00:00
#14 3.880 Collecting urllib3<3,>=1.21.1 (from requests->-r requirements.txt (line 6))
#14 3.910   Downloading urllib3-2.0.3-py3-none-any.whl (123 kB)
#14 3.915      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB 31.4 MB/s eta 0:00:00
#14 3.951 Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 6))
#14 3.987   Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
#14 3.991      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB 42.4 MB/s eta 0:00:00
#14 4.202 Collecting botocore<1.30.0,>=1.29.163 (from boto3->-r requirements.txt (line 7))
#14 4.234   Downloading botocore-1.29.163-py3-none-any.whl (11.0 MB)
#14 4.606      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.0/11.0 MB 28.7 MB/s eta 0:00:00
#14 4.640 Collecting jmespath<2.0.0,>=0.7.1 (from boto3->-r requirements.txt (line 7))
#14 4.673   Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
#14 4.702 Collecting s3transfer<0.7.0,>=0.6.0 (from boto3->-r requirements.txt (line 7))
#14 4.734   Downloading s3transfer-0.6.1-py3-none-any.whl (79 kB)
#14 4.742      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 kB 12.9 MB/s eta 0:00:00
#14 4.772 Collecting PyMeta3>=0.5.1 (from pybars3->-r requirements.txt (line 8))
#14 4.812   Downloading PyMeta3-0.5.1.tar.gz (29 kB)
#14 4.818   Preparing metadata (setup.py): started
#14 4.920   Preparing metadata (setup.py): finished with status 'done'
#14 4.927 INFO: pip is looking at multiple versions of bullmq to determine which version is compatible with other requirements. This could take a while.
#14 4.928 Collecting bullmq (from -r requirements.txt (line 9))
#14 4.965   Downloading bullmq-1.3.0.tar.gz (4.1 kB)
#14 4.970   Installing build dependencies: started
#14 5.577   Installing build dependencies: finished with status 'done'
#14 5.578   Getting requirements to build wheel: started
#14 5.681   Getting requirements to build wheel: finished with status 'done'
#14 5.682   Installing backend dependencies: started
#14 5.992   Installing backend dependencies: finished with status 'done'
#14 5.992   Preparing metadata (pyproject.toml): started
#14 6.096   Preparing metadata (pyproject.toml): finished with status 'done'
#14 6.139   Downloading bullmq-1.2.0.tar.gz (36 kB)
#14 6.150   Preparing metadata (setup.py): started
#14 6.250   Preparing metadata (setup.py): finished with status 'error'
#14 6.252   error: subprocess-exited-with-error
#14 6.252   
#14 6.252   × python setup.py egg_info did not run successfully.
#14 6.252   │ exit code: 1
#14 6.252   ╰─> [12 lines of output]
#14 6.252       Traceback (most recent call last):
#14 6.252         File "<string>", line 2, in <module>
#14 6.252         File "<pip-setuptools-caller>", line 34, in <module>
#14 6.252         File "/tmp/pip-install-hrycjqqh/bullmq_798bdde93fec42bfa045c9cf59f31ba0/setup.py", line 2, in <module>
#14 6.252           from bullmq import __version__
#14 6.252         File "/tmp/pip-install-hrycjqqh/bullmq_798bdde93fec42bfa045c9cf59f31ba0/bullmq/__init__.py", line 10, in <module>
#14 6.252           from bullmq.queue import Queue
#14 6.252         File "/tmp/pip-install-hrycjqqh/bullmq_798bdde93fec42bfa045c9cf59f31ba0/bullmq/queue.py", line 2, in <module>
#14 6.252           from bullmq.redis_connection import RedisConnection
#14 6.252         File "/tmp/pip-install-hrycjqqh/bullmq_798bdde93fec42bfa045c9cf59f31ba0/bullmq/redis_connection.py", line 1, in <module>
#14 6.252           import redis.asyncio as redis
#14 6.252       ModuleNotFoundError: No module named 'redis'
#14 6.252       [end of output]
#14 6.252   
#14 6.252   note: This error originates from a subprocess, and is likely not a problem with pip.
#14 6.253 error: metadata-generation-failed
#14 6.253 
#14 6.253 × Encountered error while generating package metadata.
#14 6.253 ╰─> See above for output.
#14 6.253 
#14 6.253 note: This is an issue with the package mentioned above, not pip.
#14 6.253 hint: See above for details.

The above error could have been handled more gracefully by forcing a redis update.

For now the resolution is installing the latest redis. Thanks for fixing the issue though!
Can we know the resolution?

@roggervalf
Copy link
Collaborator

hi @vraj-fyno, sorry for the delay. I'm working on a fix for setting the minimum required version of python

github-actions bot pushed a commit that referenced this issue Jul 10, 2023
## [4.2.1](v4.2.0...v4.2.1) (2023-07-10)

### Bug Fixes

* **flow:** emit delayed event when parent is moved to delayed ([#2055](#2055)) ([f419ff1](f419ff1))
* **python:** add requires-python config ([#2056](#2056)) fixes [#1979](#1979) ([a557970](a557970))
* **python:** fix isPaused method when custom prefix is present ([#2047](#2047)) ([7ec1c5b](7ec1c5b))

### Features

* **python:** add moveToWaitingChildren job method ([#2049](#2049)) ([6d0e224](6d0e224))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
5 participants