option to be be able to use datasets behind a proxy

[tarrade]

Is your feature request related to a problem? Please describe.
Right now behing a proxy, it is not working:

ds_train = tfds.load(name="cats_vs_dogs", split=tfds.Split.TRAIN)

C:\Program Files\Anaconda3\envs\env_gcp_dl_2_0_ds\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    514                 raise SSLError(e, request=request)
    515
--> 516             raise ConnectionError(e, request=request)
    517
    518         except ClosedPoolError as e:
ConnectionError: HTTPConnectionPool(host='storage.googleapis.com', port=80): Max retries exceeded with url: /tfds-data/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000192F3D06668>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))

I don't think this is supported for now (I didn't see it in the documentation):
https://www.tensorflow.org/datasets/api_docs/python/tfds/load

This will impact quite a lot of people working in company and university

Describe the solution you'd like
I am not an expert but using request seems to be the standard way. Below on example from a Google GCP tool:

from google.cloud import storage
import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS']=xxx
os.environ['HTTPS_PROXY']=xxx
os.environ['REQUESTS_CA_BUNDLE']=/xxx/xxx
client = storage.Client()

ignore the GOOGLE_APPLICATION_CREDENTIALS' whihc is specific to GCP. The user need to setup one or 2 env variables and everything is done in the backgroud (I guess this is using requests)

http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

[captain-pool]

@rsepassi @Conchylicultor @cyfra Is this issue fixed? if not, can you assign this to me?

[Conchylicultor]

It should be fixed with @captain-pool contribution #488

[tarrade]

Thanks @captain-pool

"To Configure the Proxy Settings, The User needs to set the Proxies for HTTP, HTTPS and FTP in the Environment Variables
TFDS_HTTP_PROXY, TFDS_HTTPS_PROXY, TFDS_FTP_PROXY respectively."

Do you also have an option to pass a CA certificate for SSL ?

Right now it is crahsing with :

requests.exceptions.SSLError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/53169/files/Kather_texture_2016_image_tiles_5000.zip (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))

This is typical of SSL interception and you need to put SSL verify false (if possible) or simply pass the CA certificate. Did you implement a REQUESTS_CA_BUNDLE environment variable as well ? Wich lib is used in your implementation ? Request ?

[captain-pool]

Hey @tarrade the downloader uses both requests and urllib. And Sorry, I totally missed the feature request for CA file. I just made it flexible for Proxies. Will add the support for CA Certificates ASAP.

[captain-pool]

@Conchylicultor should I skip the certificate verification by passing CERT_NONE from ssl, or should I put an option for adding certificate file?

[tarrade]

Hi @captain-pool , no problem. I know it is only compny that are using proxy and CA certificate and we are suffering from that everyday. I will be happy to test it when you have it ready. Just tell me in which nithly build it was collected. Thanks

[captain-pool]

Can you re open the issue?

…

On Mon, 10 Jun 2019, 9:19 pm Dr. Fabien Tarrade, ***@***.***> wrote: Hi @captain-pool <https://github.com/captain-pool> , no problem. I know it is only compny that are using proxy and CA certificate and we are suffering from that everyday. I will be happy to test it when you have it ready. Just tell me in which nithly build it was collected. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#275?email_source=notifications&email_token=ADKYRWJRBRNKJYOM7NUBFLLPZZZZBA5CNFSM4G7G2XT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKIWOQ#issuecomment-500468538>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKYRWMNOKF4RRXCHTPAFC3PZZZZBANCNFSM4G7G2XTQ> .

[tarrade]

On my side I cannot reopen this ticket. By the way I forgot thank you for the implementation of this request. You will help a lot of people using company laptop

[captain-pool]

@tarrade I think #663 should fix this. Give a Check