How to disable SSL certificate verification in Python? #5394

synchronizing · 2019-06-29T01:30:29Z

I have read the docs up and down and I can't seem to find a reference for disabling SSL certificate verification. As of right now, I currently have a project where I am doing an intentional man-in-the-middle attack to switch proxies on need-bases.

client <-> Proxy Switcher (server acting as proxy)
Proxy Switcher (emulated client) <-> Exchanges

On the return of anything but a 200, the proxy switcher automatically switches proxy and try again. A man-in-the-middle is needed to verify that the HTTPS requests are coming back with the proper headers, and this part seems to work fine. However, communicating between the client and proxy switcher seems to be returning back (expected) SSL certificate issues, as proxy switcher automatically generates its own local certificate. This, though, I would like to disable.

Without adding the certificate to my local trust env, I receive the error:

(Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Which is warranted, but would be best if could be disabled by some exchange flag. When adding the certificate to my local trust env, I receive the error

(Caused by SSLError(SSLCertVerificationError("hostname 'pro.coinbase.com' doesn't match 'Felipes-MacBook-Pro.local'")))

Which is also warranted, but a check I would much rather disable. Any help would be appreciated. My idea would be something simple as:

ex = getattr(ccxt, exchange)(
    {
        "session": cfscrape.create_scraper(),
        "enableRateLimit": False,
        "verify_ssl_certificates": False,
    }
)

Note: I have tried the verify flag with no success.

The text was updated successfully, but these errors were encountered:

kroitor · 2019-06-29T02:08:10Z

@synchronizing

What's your version of Python?
Are you using the sync or the async version of the lib?

kroitor · 2019-06-29T02:11:47Z

@synchronizing ↓ does this help?

session = cfscrape.create_scraper()
session.verify = False
ex = getattr(ccxt, exchange)(
    {
        "session": session,
        "enableRateLimit": False,
    }
)

synchronizing · 2019-06-29T02:12:02Z

@synchronizing

What's your version of Python?

Are you using the sync or the async version of the lib?

Python 3.7.2 and using sync (tested with cfscrape, and without). Will give it a try on async as well, just to be sure, as I know aiohttp has fewer SSL verifications.

kroitor · 2019-06-29T02:12:30Z

@synchronizing let us know if the comment above does not resolve the issue for you: #5394 (comment)

synchronizing · 2019-06-29T02:13:23Z

@synchronizing ↓ does this help?

session = cfscrape.create_scraper()
session.verify = False
ex = getattr(ccxt, exchange)(
    {
        "session": session,
        "enableRateLimit": False,
    }
)

Tested with the above with the same SSL verification error, unfortunately.

synchronizing · 2019-06-29T02:16:13Z

Tested with ccxt_async, and it seems to work fine -- passes through the proxy finder as well, without throwing back any issue. SSL certificated added in env (not sure if it would work without.)

However, cfscrape is still desired for coinbasepro.

kroitor · 2019-06-29T02:17:09Z

This page says it should have worked with the sync version as well: https://2.python-requests.org/en/master/user/advanced/#ssl-cert-verification

@synchronizing can you post a complete short snippet of your code to reproduce it, say, 10-20 lines? We need to make sure that there's no other interference, therefore we need a complete snippet, including the instantiation.

synchronizing · 2019-06-29T02:18:48Z

Sure thing @kroitor -- as of now the code is wrapped in an API server, so give me a few to extract the relevant lines to a separate text file for easy testing on your end.

Also: is the session.verify a boolean, or a certificate location string? From previous issues I saw here, I thought it was simply a bool flag.

kroitor · 2019-06-29T02:22:34Z

Also: is the session.verify a boolean, or a certificate location string? From previous issues I saw here, I thought it was simply a bool flag.

It should work either way, as a boolean or as a string-path, if the docs are correct.

synchronizing · 2019-06-29T02:23:47Z

Also: is the session.verify a boolean, or a certificate location string? From previous issues I saw here, I thought it was simply a bool flag.

It should work either way, as a boolean or as a string-path, if the docs are correct.

Sounds good. Give me a few to compile the problem down to a few lines of code.

synchronizing · 2019-06-29T02:41:29Z

Utilizing the following code:

import ccxt

exchange = ccxt.binance()
exchange.session.verify = False # With, or without line.
fetch = exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
print(fetch)

With export to a man-in-the-middle proxy:

export http_proxy=http://127.0.0.1:8888
export https_proxy=http://127.0.0.1:8888

I still receive error on the sync version of ccxt. If you would like to test it out with the man-in-the-middle, you can find it on my repo here. Just run the example/example_server.py file.

I just came to realize the reason it might have worked with aiohttp is that even with setting exchange.session.trust_env and exchange.session.trust_env_aiohttp, neither flags respect the set http_proxy and https_proxy env variable.

synchronizing · 2019-06-29T02:45:56Z

Update: aiohttp does not work either.

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch


print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

With implicit aiohttp_proxy set (since http_proxy and https_proxy is does not seem to be respected), the SSL checks still return error. I know for a fact aiohttp uses the flag ssl (rather than verify, like the requests library, to enable/disable SSL checks) but I assume there is no existing flag for ccxt.

kroitor · 2019-06-29T02:49:26Z

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

↑ This is not a correct way of configuring it. You should add an async session and set verify = False on it, before passing it to the binance constructor. The derived exchange class does not support the verify option.

More about it here:

https://github.com/ccxt/ccxt/blob/master/python/ccxt/async_support/base/exchange.py#L75

kroitor · 2019-06-29T02:51:44Z

It sync-misbehavior might be a bug in the MITM proxy: https://www.google.com/search?q=python+https+proxy+ssl+verify+requests. Looks like you're not the only person having difficulties when using proxies + ssl verify.

synchronizing · 2019-06-29T02:53:02Z

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch
↑ This is not a correct way of configuring it. You should add an async session and set verify = False on it, before passing it to the binance constructor. The derived exchange class does not support the verify option.

When you say this, would this be the correct format?

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888"})
    exchange.session.verify = False
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

synchronizing · 2019-06-29T02:56:25Z

It might be a bug in the MITM proxy: https://www.google.com/search?q=python+https+proxy+ssl+verify+requests. Looks like you're not the only person having difficulties when using proxies + ssl verify.

Tell me about it -- SLL + proxies is a nightmare, as I've come to find out 😩. However, mitm is acting as the destination server for the client, so it's not actually communicating the request forward to the destination server as a normal proxy would. Instead, it initiates an emulated client that does that, and then returns the request of the emulated client back to the actual client, reading the request in the middle. The issue is the initial communication between mitm and the client with ccxt, mainly due to a self-signed certificate in the middle. Even with verify flag set to false, the error seems to persist.

kroitor · 2019-06-29T02:58:39Z

would this be the correct format?

Nope. This would be the correct format:

import aiohttp
import asyncio

event_loop = asyncio.get_event_loop()

async def fetch_stuff():
    connector = aiohttp.TCPConnector(ssl=False, loop=event_loop)
    session = aiohttp.ClientSession(loop=event_loop, connector=connector, trust_env=True)
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", 'session': session})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(event_loop.run_until_complete(asyncio.gather(fetch_stuff())))

Does that help?

synchronizing · 2019-06-29T03:00:26Z

would this be the correct format?

Nope. This would be the correct format:

import aiohttp
import asyncio

event_loop = asyncio.get_event_loop()

async def fetch_stuff():
    trust_environment_variables = False
    connector = aiohttp.TCPConnector(ssl=False, loop=event_loop)
    session = aiohttp.ClientSession(loop=event_loop, connector=connector, trust_env=trust_environment_variables)
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", 'session': session})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(event_loop.run_until_complete(asyncio.gather(fetch_stuff())))

Does that help?

Yes, it did! I received a request in the mitm which got processed but didn't get properly returned (which is fault from mitm). Much appreciated man! Let me give it a try with sync ccxt.

kroitor · 2019-06-29T03:01:27Z

@synchronizing something like the above should work for the sync version as well. Just need to dig the internets on the proper way to configure it. However, this is beyond CCXT, unfortunately.

synchronizing · 2019-06-29T03:02:55Z

@synchronizing something like the above should work for the sync version as well. Just need to dig the internets on the proper way to configure it. However, this is beyond CCXT, unfortunately.

Perfectly understandable -- all I was hoping for was a proper CONNECT and GET with mitm, as from there on out I knew ccxt had completed the proper ssl steps.

kroitor · 2019-06-29T03:03:43Z

@synchronizing are you ok if we close this for now?

synchronizing · 2019-06-29T03:04:03Z

Forgive me for the repetitive questioning: but with cfscraper and standard sync functionality, can we set a session? While I have you on the wire.

If this is beyond the project due to cfscraper, no worries. The help above has already been fantastical and I truly do appreciate it.

Also, completely irrelevant: Don't fall for AdBlocker Pro -- they were bought out by some company a while ago, and still display advertisements. Open source solution is uBlock Origin which very few people seem to be aware is out there and is also a superb blocker in comparison (including YouTube advertisement, thank the Lord.)

kroitor · 2019-06-29T03:10:47Z

Forgive me for the repetitive questioning: but with cfscraper and standard sync functionality, can we set a session?

Yes, that should be possible. However, there may be bugs outside of CCXT:

https://github.com/kennethreitz/requests/issues/3829

I will add the exchange.verify option for the sync version shortly and will need your help to test it against your proxy.

synchronizing · 2019-06-29T03:11:49Z

Forgive me for the repetitive questioning: but with cfscraper and standard sync functionality, can we set a session?

Yes, that should be possible. However, there may be bugs outside of CCXT:

kennethreitz/requests#3829

I will add the exchange.verify option for the sync version shortly and will need your help to test it against your proxy.

Sounds like a plan! Again, I appreciate the help. Feel free to close this issue and re-open when I am needed for testing.

kroitor · 2019-06-29T03:16:22Z

@synchronizing should not take too long, will let you know when it's there, 30-60 minutes.

kroitor · 2019-06-29T03:32:59Z

@synchronizing found a couple of minor issues along the way, added the fixes, so, it will arrive shortly (5-10 minutes). Standing by.

synchronizing · 2019-06-29T03:35:03Z

@synchronizing found a couple of minor issues along the way, added the fixes, so, it will arrive shortly (5-10 minutes). Standing by.

What version should I be on the look out for?

kroitor · 2019-06-29T03:36:14Z

@synchronizing 1.18.844 (the one upcoming).

kroitor · 2019-06-29T03:47:24Z

Ok, it has arrived, looking forward to hearing from you.

synchronizing · 2019-06-29T03:53:33Z

sync version is working as expected on 1.18.844:

import ccxt as ccxt

exchange = ccxt.binance(
    {
        "proxies": {"http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888"},
        "verify": False,
    }
)
fetch = exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
print(fetch)

async version does not work with the following:

import ccxt.async_support as ccxt
import asyncio

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

SSL error is thrown for the above code. However, it does work when loading session as shown previously here. Assumingly, verify might not be setting the ssl flag in the TCPConnector to False internally within ccxt.

kroitor · 2019-06-29T03:59:15Z

Assumingly, verify might not be setting the ssl flag in the TCPConnector to False internally within ccxt.

Ok, I've added one more edit to it, let us know if 1.18.845 does not resolve the issue on the async side. Closing this for now. Feel free to reopen it or just ask further questions, if any. We will be happy if you report back anyways.

synchronizing · 2019-06-29T04:01:02Z

Sounds good -- will give it a try, thank you again.

synchronizing · 2019-06-29T04:50:31Z

async is still not working with 1.18.845 with "verify" : False set.

import ccxt.async_support as ccxt
import asyncio

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    print(exchange.verify)
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

kroitor · 2019-06-29T04:52:22Z

@synchronizing doesn't work with verify: False, but works with #5394 (comment) ?

synchronizing · 2019-06-29T04:53:30Z

@synchronizing doesn't work with verify: False, but works with #5394 (comment) ?

Correct, unfortunately.

kroitor · 2019-06-29T04:57:15Z

@synchronizing are you sure about that? This is strange, because they should be technically equal...

kroitor · 2019-06-29T04:58:37Z

@synchronizing ah, nvm, found another bug there with the ordering of the calls, will fix it in a moment.

synchronizing · 2019-06-29T04:59:44Z

@synchronizing are you sure about that? This is strange, because they should be technically equal...

They seem to be different on further testing:

import ccxt.async_support as ccxt
import asyncio
import aiohttp

async def fetch_stuff():
    connector = aiohttp.TCPConnector(ssl=False, loop=asyncio.get_event_loop())
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    print(exchange.session._connector._ssl)
    print(connector._ssl)
    await exchange.close()

asyncio.get_event_loop().run_until_complete(fetch_stuff())

Outputs:

<ssl.SSLContext object at 0x10f55c480>
False

@synchronizing ah, nvm, found another bug there with the ordering of the calls, will fix it in a moment.

Awesome, lmk!

kroitor · 2019-06-29T05:05:25Z

@synchronizing ok, let's try again with 1.18.846. Unfortunately, I can't really test it on my side atm, your help with debugging it is very much appreciated! The autobuild takes 10-15 minutes.

synchronizing · 2019-06-29T05:06:16Z

@synchronizing ok, let's try again with 1.18.846. Unfortunately, I can't really test it on my side atm, your help with debugging it is very much appreciated! The autobuild takes 10-15 minutes.

My pleasure man -- I appreciate all the work on your end. I'll be on the lookout for the new release to give it a try.

kroitor · 2019-06-29T05:11:32Z

https://travis-ci.org/ccxt/ccxt/builds

synchronizing · 2019-06-29T05:14:25Z

https://travis-ci.org/ccxt/ccxt/builds

Working as expected! Thank you again, cheers.

) * [bleutrade] started adding v3 endpoints * [bleutrade] ooops * [bleutrade] ledger fixes * [bleutrade] added example order * parse transaction fee correctly for okex3 In `parse_transaction` of okex3, [Line](https://github.com/ccxt/ccxt/blob/master/python/ccxt/okex3.py#L2099) ``` fetchWithdrawals { amount: "4.72100000", withdrawal_id: "1729116", fee: "0.01000000eth", ## here the fee is endwith 'eth' currency: "ETH", .... } ``` the value of fee is end with 'currency'. So the [code](https://github.com/ccxt/ccxt/blob/master/python/ccxt/okex3.py#L2140) cannot parse the real fee, and the feeCost is None. ``` feeCost = self.safe_float(transaction, 'fee') ``` * okex3 parseTransaction fee currency id vs code refix * okex3 typo leftover * okex3 eslint trailing space * fix #5383 * 1.18.839 [ci skip] * 1.18.840 [ci skip] * coingi referral url * 1.18.841 [ci skip] * minor edits in python/setup.py * Huobi Pro: Add back private get order/orders The order/history endpoint is only 48hrs. * 1.18.842 [ci skip] * huobipro fetchOpenOrders is now configurable fix #5376 fix #5392 fix #5388 * 1.18.843 [ci skip] * added self.verify for Python 2/3 fix #5394 * exchange.py async minor edit #5394 * exchange.py async minor edit #5394 * async_support/base/exchange.py minor simplifications #5394 * 1.18.844 [ci skip] * 1.18.845 [ci skip] * async_support/base/exchange.py __init__ ssl verify fix #5394 * 1.18.846 [ci skip] * add info the issue template * 1.18.847 [ci skip] * [coinbase] minor fix to honour limit param in fetchTransactionsWithMethod; and small tidy up in sign method * bleutrade fetchOrders edit * bleutrade parseLedgerEntry minor edits in comments * bleutrade parseLedgerEntry new unified safeCurrencyCode standard * bleutrade parseLedgerEntry edits * bleutrade old doc url is now 404, switched to v3 * bleutrade has['fetchLedger'] = true * bleutrade parseOrder minor edits in comments * bleutrade parseLedgerEntry deduplication rework startsWith → indexOf safe-methods everywhere removed duplication of description parsing for fees, and other data removed fields that do not belong to the ledger * bleutrade fetchLedger respect and propagate params

brandsimon · 2019-07-08T06:56:23Z

@kroitor
Should we log a warning if SSL Certificates are disabled or overwritten, since this can result in stolen keys? MITM Attack and there are some exchanges using login-data like keys (for example dx.exchange).

synchronizing · 2019-07-08T12:52:53Z

@kroitor
Should we log a warning if SSL Certificates are disabled or overwritten, since this can result in stolen keys? MITM Attack and there are some exchanges using login-data like keys (for example dx.exchange).

Let me just add:

aiohttp (ccxt.async_support) does not throw out SSL certificate warning.
requests (ccxt) does throw SSL certificate warning through urllib3.

kroitor · 2019-07-08T15:15:47Z

@brandsimon in this particular case, the MITM attack is done deliberately by the owner. But in general, yes, I think it would be great to have a warning so that we keep people aware! )

@synchronizing thx for the hints!

kroitor assigned frosty00 Jun 29, 2019

kroitor changed the title ~~Disable SSL certificate verification.~~ How to disable SSL certificate verification? Jun 29, 2019

kroitor added the question label Jun 29, 2019

kroitor changed the title ~~How to disable SSL certificate verification?~~ How to disable SSL certificate verification in Python? Jun 29, 2019

synchronizing closed this as completed Jun 29, 2019

kroitor added a commit that referenced this issue Jun 29, 2019

async_support/base/exchange.py minor simplifications #5394

5f0e741

kroitor closed this as completed Jun 29, 2019

kroitor added a commit that referenced this issue Jun 29, 2019

async_support/base/exchange.py __init__ ssl verify fix #5394

212872a

npomfret pushed a commit to npomfret/ccxt that referenced this issue Jun 29, 2019

added self.verify for Python 2/3 fix ccxt#5394

fd0479a

npomfret pushed a commit to npomfret/ccxt that referenced this issue Jun 29, 2019

exchange.py async minor edit ccxt#5394

1f78ae0

npomfret pushed a commit to npomfret/ccxt that referenced this issue Jun 29, 2019

exchange.py async minor edit ccxt#5394

0e8305c

npomfret pushed a commit to npomfret/ccxt that referenced this issue Jun 29, 2019

async_support/base/exchange.py minor simplifications ccxt#5394

b0d0512

npomfret pushed a commit to npomfret/ccxt that referenced this issue Jun 29, 2019

async_support/base/exchange.py __init__ ssl verify fix ccxt#5394

d07a9dc

kroitor mentioned this issue Jul 16, 2020

Adding possibility to use http proxies via http settings #599

Merged

How to disable SSL certificate verification in Python? #5394

How to disable SSL certificate verification in Python? #5394

Comments

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019 • edited

synchronizing commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019 • edited

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019

kroitor commented Jun 29, 2019 • edited

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019 • edited

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019 • edited

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

kroitor commented Jun 29, 2019

synchronizing commented Jun 29, 2019

brandsimon commented Jul 8, 2019

synchronizing commented Jul 8, 2019 • edited

kroitor commented Jul 8, 2019

synchronizing commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

kroitor commented Jun 29, 2019 •

edited

synchronizing commented Jun 29, 2019 •

edited

synchronizing commented Jul 8, 2019 •

edited