Task works (as function) without .delay() in views, does not work as a task with celery

[tvanzo]

Could not find a dublitcate issue regarding this.

I have celery setup fine in that it works for simple tasks like a hello world (calling hello_world.delay() in django views there is no problem).

I created another more complicated function that involves making many calls to a an api to retrieve data to update the database. Just calling this function without celery (task_a()) there is no problem, and the database gets updated(takes around 15 seconds or so...that is why I want to use celery).

My settings are as follow..I am not sure if I am supposed to change anything with regards to the backend to get it working as well..I am using django default sqlite..not really sure what 'django-db' means:
WSGI_APPLICATION = 'trydjango.wsgi.application'
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
BROKER_TRANSPORT = 'redis'
CELERY_RESULT_BACKEND = 'django-db'
CELERY_TIMEZONE = 'Asia/Makassar'

But when I add .delay() to this to make it a celery task I get the following error. I can't figure out why there would be this disparity, any thoughts? :

INFO/MainProcess] Received task: todo.tasks.update_comments[ae9bebfe-4b3c-46ec-b197-0ffa56847f30]

[2019-10-31 05:33:52,652: ERROR/ForkPoolWorker-2] Task todo.tasks.update_comments[ae9bebfe-4b3c-46ec-b197-0ffa56847f30] raised unexpected: SSLError(MaxRetryError('HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /youtube/v3/commentThreads?part=replies%2Csnippet&allThreadsRelatedToChannelId=UC_EyncGJh2QuQHhcfuDWL6g&searchTerms=%23vi&maxResults=100&order=time&key=xxxmyapikey (Caused by SSLError(SSLError("bad handshake: SysCallError(54, 'ECONNRESET')")))'))
Traceback (most recent call last):
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
cnx.do_handshake()
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (54, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in validate_conn
conn.connect()
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/connection.py", line 394, in connect
ssl_context=context,
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/util/ssl.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Users/tylervanzo/Desktop/trydjango/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: SysCallError(54, 'ECONNRESET')",)

[auvipy]

please post on celery-users list or IRC #celery first before you aare sure it's a celery bug.

[tvanzo]

@auvipy sorry about that, just posted to both, thanks for the headsup. although the IRC thing seems weird to me..just started using it, but anytime I go off my computer it logs out and there is no way to see the messages sent when offline.

[tvanzo]

I believe this could be a bug with celery due to the fact that the function works fine as a non celery task (aka before I put .delay()). It makes the api calls and updates the db smoothly. When I add it as a task I get ssl errors and max retries exceeded. Can't figure out why this would be. I have celery configured correctly it seems (as other simple tasks work fine like hello_world) unless there is specific settings for api call functions.

[auvipy]

please come with a test case

[tvanzo]

Test Case:
The reason I think it could be a bug with Celery:
My task works with celery with an api call outside of the call I am making to google.

[r1=requests.get('https://jsonplaceholder.typicode.com/todos/1')
        rj1=r1.json()
        print(rj1))]

When I make it the get request the google api (with my api key), I get the ssl error, but only when it is a task through celery. The actually url there is no problem with running a curl to get the data/when its not a task and run by itself it works fine. So it seems like there is some kind of issue with celery and the google api, I have no idea where/what it could be.

r1=requests.get('https://www.googleapis.com/youtube/v3/commentThreads?part=snippet%2Creplies&allThreadsRelatedToChannelId=UCQMZZMfz0D1RueGzcYLd87Q&searchTerms=love&key=MYKEY')

[tvanzo]

btw you can test out the google api/get the url for the get request here (just unselect the api and oauth:
https://developers.google.com/youtube/v3/docs/commentThreads/list?apix_params=%7B%22part%22%3A%22snippet%2Creplies%22%2C%22allThreadsRelatedToChannelId%22%3A%22UCQMZZMfz0D1RueGzcYLd87Q%22%2C%22searchTerms%22%3A%22love%22%7D

[auvipy]

update on master?

[auvipy]

were you able to figure out a resolution to this issue?

[thedrow]

I believe this could be a bug with celery due to the fact that the function works fine as a non celery task (aka before I put .delay()). It makes the api calls and updates the db smoothly. When I add it as a task I get ssl errors and max retries exceeded. Can't figure out why this would be. I have celery configured correctly it seems (as other simple tasks work fine like hello_world) unless there is specific settings for api call functions.

You might have a different Python interpreter running Celery which wasn't built with OpenSSL support.
Can you import the ssl module?

[Roughmoyez]

am going through the same issue when i call my celery tasks with .delay