Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requests not working in a Docker container #3948

Closed
javixeneize opened this issue Mar 31, 2017 · 42 comments
Closed

Requests not working in a Docker container #3948

javixeneize opened this issue Mar 31, 2017 · 42 comments

Comments

@javixeneize
Copy link

Hi

Thats the old story about SSL not working with requests, but one step further.... Docker containers

I have an application that uses requests, and it works fine in my local machine, but, when deploying it in a Docker container, i am having an error with requests module (SSL error)

[2017-03-31 11:32:29,863] ERROR in app: Exception on /send [POST]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "app.py", line 62, in sendrequest
response=sess.post(url,params, headers=h,verify=False)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 535, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

I have heard it might be related to openSSL. Any idea on how this can be resolved? Should i include any dependency?

@sigmavirus24
Copy link
Contributor

Can you run openssl version in your container?

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

This can also very definitely be related to the Python version, we'd like to know that as well.

@javixeneize
Copy link
Author

Hi

Python 2.7 (12)
Openssl 1.0.2g

Thanks

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

Are you running the same version of Requests in the container and outside? What are the two versions?

@javixeneize
Copy link
Author

2.11.1 when working, and 2..12.5 inside the container. Does it make any difference?

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

Yeah, there are some decent code changes between those two versions. Want to try downgrading to 2.11 quickly in the container to see what happens?

@javixeneize
Copy link
Author

Still failing, but with a different error message
...
...
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 491, in send
raise SSLError(e, request=request)
SSLError: EOF occurred in violation of protocol (_ssl.c:590)

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

Hrm. That different error is pretty much the same error. Can you reveal to us what web server you're trying to contact?

@javixeneize
Copy link
Author

Thats my own server, which has a self signed certificate. I am sending verify=False to ignore SSL but doesnt seem to like it

response=sess.post(url,params, headers=h,verify=False)

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

Can you show the TLS configuration for your server, and what OpenSSL version it is linked against?

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

So, "wrong" here is defined only in whether there is a mismatch between what the client and server expect. Out of interest, does your server expect SNI? Are you reaching your server via hostname or IP?

@sigmavirus24
Copy link
Contributor

In that container, can you curl your server with the same URL? Or even just telnet to it? I wonder if requests can even reach the server from where you've deployed the container.

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

Ok, what's the result of running openssl s_client -connect host:port to your server from inside the container?

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

Sorry, the whole result. I'm interested in what the result of the negotiation is.

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@sigmavirus24
Copy link
Contributor

That is not the whole result. Please copy and paste everything from that command.

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

No, the whole result. All the output. Everything it prints.

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

That information is exactly what I am interested in. Something in our TLS Client Hello is making your server mad, and so I'm interested in seeing what your server is negotiating.

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@Lukasa
Copy link
Member

Lukasa commented Mar 31, 2017

It's not your certificate. The error message in question (Unexpected EOF) means that during the TLS handshake the server has sent us a TCP FIN or RST packet. That means the server has chosen to close the connection, not us. That means the server has decided we are not doing something it likes. As a result, this cannot be the fault of your certificate: we haven't gotten to the point of validating it yet.

@sigmavirus24
Copy link
Contributor

@javixeneize without the information we've asked you for, I'm not sure what else we can do to help.

@javixeneize
Copy link
Author

javixeneize commented Mar 31, 2017 via email

@harry1064
Copy link

harry1064 commented May 31, 2017

root@4f66ccbaef3c:/# openssl s_client -connect api.quinto.ai:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = fbbot.quintoapp.com
verify return:1
---
Certificate chain
 0 s:/CN=fbbot.quintoapp.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFCjCCA/KgAwIBAgISA1Bg18LrjA3qyyrEDmzE+5YSMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzAzMTQwNzMyMDBaFw0x
NzA2MTIwNzMyMDBaMB4xHDAaBgNVBAMTE2ZiYm90LnF1aW50b2FwcC5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQD0xbEa4W4k4BlPrIxeVX+ekPl1
Od4OuwepY5Ha2BQd6YMiphh+we5H6JVu2XDuPbQnmQMtEwGa2T2Adhic4bGPPC7+
0j+utJuqGBRIbYJ09A5EQOhB4HhOSI82l1ZpPkHpvOiC4UoEgG4KOLnqBX0JydI3
8vhiV4EgbLr77wARsEeinK+Zj+7bpsEK8q+B7mR5km6f6tKT/i++Wd4Fx3Pz7iuK
aCulKzG4IMxopE/9DDf608H/3cFcSHvg/4IguPoOCx2ArNKE7QCNFGYAx9HhnV2y
AYVbd2WGWeJKuNWEwCF+nvxGDo4cHdT6kq9HHB6kPTYrZb7PrKtkq1C3MNB/AgMB
AAGjggIUMIICEDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFENiqFyUkXGaxd/woyxi
6SqQz2WqMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMHAGCCsGAQUF
BwEBBGQwYjAvBggrBgEFBQcwAYYjaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNy
eXB0Lm9yZy8wLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5j
cnlwdC5vcmcvMB4GA1UdEQQXMBWCE2ZiYm90LnF1aW50b2FwcC5jb20wgf4GA1Ud
IASB9jCB8zAIBgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIB
FhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtU
aGlzIENlcnRpZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlp
bmcgUGFydGllcyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRp
ZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9y
ZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEAf80at95dsb9WsSMaChtKNEJR
FfuPd/5MZaqFxWM6+AtEGZt2qbeOExIShEHFehUSWQnrCBTiPY6ildK1E5nhduap
4K0O7FrnMVaNBhnoBT7jsZMs7ivLpaKCT6imR71hQTUv07xw1kQJMu/jrHHtVjNi
9iI+VryZeETIVBtCXbirwKxT0JYLicdS/9M9m9wC7/H8xWVkcRR5dMI2Im+4klX4
eGmgi+XCJPkDZZEpfQHmIqQQ9ccCpP0BFs0JqfwLich71NdPihVnJDhVZrEVMcgs
+412WdWCOTIXrEzsL6xddypVETY2k5Z3S8sZTInWA9nYOGHW82xwh6/tRU+yiQ==
-----END CERTIFICATE-----
subject=/CN=fbbot.quintoapp.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3157 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 78B1ABFE5A7BF1E698FB5D43D1A75D6F874DD9D2E12816E3276B349FC0C4B96B
    Session-ID-ctx:
    Master-Key: B6EE0F224CB1A93379B86524E9F01D618A018E2F1D68F5AB29F7570611F0D9CF4210F9946335A9FAAEEA143B0BC98D26
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 00 39 b1 81 4d f9 90 0a-b2 dd a8 e7 b5 6b 74 7c   .9..M........kt|
    0010 - ba 4e 8b 51 75 df c4 ae-e1 6c dc 3e 05 a1 0e fc   .N.Qu....l.>....
    0020 - 4e 61 83 34 f4 ea 06 b7-8d 54 82 01 a8 b2 fa 2a   Na.4.....T.....*
    0030 - 48 69 01 b5 06 6b ee 18-3d 93 f5 d7 31 d8 66 8f   Hi...k..=...1.f.
    0040 - a4 6f f4 6c 2d 48 37 9f-33 b7 36 49 39 1f 2f 31   .o.l-H7.3.6I9./1
    0050 - 1a 0d 8f 8e 34 36 3d d1-09 fb 6b 5b 5d 68 80 3e   ....46=...k[]h.>
    0060 - 66 d9 44 11 4d 12 d5 cc-df eb c3 45 ae 04 10 56   f.D.M......E...V
    0070 - 34 ad 98 8f f9 1b f2 33-00 a4 b3 3c a5 40 80 8e   4......3...<.@..
    0080 - 9b f1 b5 40 e5 2b 29 86-7e 2b f6 95 03 4b e3 b4   ...@.+).~+...K..
    0090 - ab 16 25 bc 47 bf fb 87-dc 13 0e 10 a8 1b 18 fb   ..%.G...........
    00a0 - 3b 65 07 96 05 ce 1a c2-9a d4 d8 73 fd 38 40 8b   ;e.........s.8@.
    00b0 - 0e 52 df 26 19 fc 9f 04-06 28 b3 25 5c e2 64 51   .R.&.....(.%\.dQ

    Start Time: 1496212705
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
read:errno=0

@Lukasa
Copy link
Member

Lukasa commented May 31, 2017

What's your actual problem @harry1064?

@harry1064
Copy link

@Lukasa if you see the above result of openssl s_client -connect api.quinto.ai:443 command i ran inside docker container running on my server, You will see CN to be fbbot.quintoapp.com which is pointing to host server where my docker container is running.
But the same command i ran on host server, CN was api.quinto.ai.
So basically, i have a python server running on docker container and i want to make request to another server api.quinto.ai but I am not not able to do that using requests. Same python code i ran on interpreter on host machine, that is working fine but when i log in to docker container and ran same code from inside the python interpreter, its not working

@Lukasa
Copy link
Member

Lukasa commented May 31, 2017

By default OpenSSL s_client does not present the server name indication field, which means that the remote server will present whatever certificate it chooses. Do you get the same output in both cases if you change your command to:

openssl s_client -connect api.quinto.ai:443 -servername api.quinto.ai

Even if you don't, this sounds like a problem with docker, not with Requests: you appear to be reproducing a problem using the openssl command line, which doesn't use Requests in any way. So I'm not sure how you want us to solve your problem.

@harry1064
Copy link

I got same response with both the cases. So i also think this is the problem with docker as they are maybe using mapping between iptables.

@michi88
Copy link

michi88 commented Jul 20, 2017

Maybe this can help someone. For me I needed to install this:

OpenSSL 1.0.2g 1 Mar 2016

requests[security]==2.7.0  # not 2.18.x
cryptography==1.9   # not 2.0

@sigmavirus24
Copy link
Contributor

This discussion went stale months ago. I'm closing this. We can reopen it if necessary.

@gabrielfgularte
Copy link

@harry1064 @javixeneize do you guys have found a solution for this? I'm with the same problem and maybe this can be a Docker problem. But I really don't know how to overcome this...

Thanks!

@javixeneize
Copy link
Author

javixeneize commented Jan 11, 2018 via email

@michi88
Copy link

michi88 commented Jan 12, 2018

try installing these versions:

requests[security]==2.7.0
cryptography==1.9
pyOpenSSL==17.4.0

@Fizmath
Copy link

Fizmath commented May 13, 2019

Hello ,
I had similar problem , after degrading to the above mentioned versions everything went smoothly ( python 3.6 )

@yankee14
Copy link

yankee14 commented Aug 2, 2021

I am having a similar issue with requests in python 3.9.2. My script works correctly outside of docker, but does not run correctly in a container. I tried to rule out networking as the issue with docker run --network host. I tried installing the module versions listed here #3948 (comment) but pip throws a bunch of errors and can't install them.

# pip list
Package            Version
------------------ ---------
APScheduler        3.7.0
certifi            2021.5.30
cffi               1.14.6
charset-normalizer 2.0.4
cryptography       3.4.7
idna               3.2
lxml               4.6.3
numpy              1.21.1
pandas             1.3.1
pip                21.2.2
pycparser          2.20
pyOpenSSL          20.0.1
python-dateutil    2.8.2
pytz               2021.1
requests           2.26.0
setuptools         54.2.0
six                1.16.0
timeago            1.0.15
tzlocal            2.1
urllib3            1.26.6
wheel              0.36.2

@sharpvik
Copy link

sharpvik commented Aug 5, 2021

Same as above here... This is frustrating. Me and my senior poked around with the packets, monitored some traffic but nothing helps, even downgrade to requests[security]=2.7.0. Have you got any idea what this is? Because it is really stalling my project rn.

@yankee14
Copy link

yankee14 commented Aug 5, 2021

@javixeneize @sigmavirus24 can we reopen this?

@sigmavirus24
Copy link
Contributor

The comments above explain exactly what the problem is. There is no way to write code or release to fix your problem and we're not currently capable of hand-holding everyone's custom problem for free.

As has been explained above there's something that should be doscoverably different between your environments that's causing requests to upset your server during the TLS handshake. That's only discoverable by you. Re-opening this isn't appropriate as a result

@sharpvik
Copy link

sharpvik commented Aug 5, 2021

As has been explained above there's something that should be doscoverably different between your environments that's causing requests to upset your server during the TLS handshake.

In my case, I don't think that my server is upset with the handshake. I can't reach even google.com from container with a basic requests.get('https://www.google.com').

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants