pycurl 7.45.3 wheel not working for https in debian/ubuntu systems #834

ap-wtioit · 2024-02-26T09:08:14Z

What did you try to do?

Using pycurl 7.45.3 to get the content of an https page.

What happened?

pycurl 7.45.3 does not work in debian bullseye with python 3.8 in its default configuration. it is looking for ca-certificate bundle in /etc/pki/tls/certs/ca-bundle.crt instead of /etc/ssl/certs/ca-certificates.crt.

Note: dependencies installes as needed with 7.45.2 to have a more comparable setup.
docker run --rm python:3.8-slim-bullseye bash -c 'apt-get update && apt-get install -y libcurl4-openssl-dev libssl-dev gcc && pip install pycurl==7.45.3 && python -c "import pycurl; curl = pycurl.Curl(); curl.setopt(pycurl.URL, \"https://www.google.com\"); curl.perform()"'

<trimmed logs>
Collecting pycurl==7.45.3
  Downloading pycurl-7.45.3-cp38-cp38-manylinux_2_28_x86_64.whl (4.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 21.6 MB/s eta 0:00:00
Installing collected packages: pycurl
Successfully installed pycurl-7.45.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
Traceback (most recent call last):
  File "<string>", line 1, in <module>
pycurl.error: (77, 'error setting certificate verify locations:\n  CAfile: /etc/pki/tls/certs/ca-bundle.crt\n  CApath: none')

What did you expect to happen?

pycurl working on debian bullseye with python 3.8 as it did in 7.45.2
docker run --rm python:3.8-slim-bullseye bash -c 'apt-get update && apt-get install -y libcurl4-openssl-dev libssl-dev gcc && pip install pycurl==7.45.2 && python -c "import pycurl; curl = pycurl.Curl(); curl.setopt(pycurl.URL, \"https://www.google.com\"); curl.perform()"'

<trimmed logs>
Collecting pycurl==7.45.2
  Downloading pycurl-7.45.2.tar.gz (234 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 234.2/234.2 kB 3.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: pycurl
  Building wheel for pycurl (setup.py): started
  Building wheel for pycurl (setup.py): finished with status 'done'
  Created wheel for pycurl: filename=pycurl-7.45.2-cp38-cp38-linux_x86_64.whl size=306820 sha256=ead76500187afc94725e57481901bf7d592cdf262858ece79f35024aaa928f11
  Stored in directory: /root/.cache/pip/wheels/07/65/79/9497b682fcb877281b1600e6580f5a9627bed14d29c361ecaa
Successfully built pycurl
Installing collected packages: pycurl
Successfully installed pycurl-7.45.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="...

Workaround

Symlinking /etc/ssl/certs/ca-certificates.crt to /etc/pki/tls/certs/ca-bundle.crt makes pycurl work:
docker run --rm python:3.8-slim-bullseye bash -c 'mkdir -p /etc/pki/tls/certs/ && ln -s /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt && pip install pycurl==7.45.3 && python -c "import pycurl; curl = pycurl.Curl(); curl.setopt(pycurl.URL, \"https://www.google.com\"); curl.perform()"'

Collecting pycurl==7.45.3
  Downloading pycurl-7.45.3-cp38-cp38-manylinux_2_28_x86_64.whl (4.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 21.3 MB/s eta 0:00:00
Installing collected packages: pycurl
Successfully installed pycurl-7.45.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="...

What is the PycURL version?

PycURL/7.45.3 libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0

What is your Python version?

Python 3.8.18

What is your operating system and its version?

Debian Bullseye

Other versions

If your report references any software besides PycURL, for example pip,
what are the versions of this software?
pip --version

pip 23.0.1 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)

Is this the most recent PycURL release?

Yes

Did you check libcurl behavior?

Yes, docker run --rm python:3.8-slim-bullseye bash -c 'apt-get update && apt-get install curl && curl https://www.google.com' is working as expected.

The text was updated successfully, but these errors were encountered:

swt2c · 2024-02-26T14:09:39Z

It looks like it's probably a problem with the wheels we started producing with 7.45.3. As a workaround, you could probably do:
pip install pycurl --no-binary :all:

pycurl/pycurl#834

kurtseifried · 2024-03-01T18:46:09Z

In case anyone ends up here as well:

QUICK WAY TO FIX IN YOUR PYTHON CODE (basically recreate the function that implements CURL_CA_BUNDLE_AUTODETECT as per https://github.com/curl/curl/blob/9e2ee704940e4e8ba38ba770b86a79ec091990bf/CMakeLists.txt#L999):

        self.ca_bundle_path = self.find_ca_bundle([
            '/etc/ssl/certs/ca-certificates.crt',
            '/etc/pki/tls/certs/ca-bundle.crt',
            '/etc/ssl/ca-bundle.pem',
            '/etc/pki/tls/cacert.pem',
            '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem',
            '/etc/ssl/cert.pem',
        ])
        if not self.ca_bundle_path:
            sys.exit("No CA bundle found. SSL certificate verification might fail. Exiting.")

and then later on:

        c.setopt(c.CAINFO, self.ca_bundle_path)

Is a quick way to replicate the functionality of CURL_CA_BUNDLE_AUTODETECT and get your code working.

pycurl/pycurl#834 TYPE: hotfix

pycurl 7.45.3 returns error code 77 (CURLE_SSL_CACERT_BADFILE) for TLS connections on Debian-based systems: pycurl/pycurl#834

1951FDG · 2024-04-12T12:20:50Z

Also affects Google Colab. As a workaround, set:

CURLOPT_CAINFO
CURLOPT_CAPATH

c = pycurl.Curl()
c.setopt(pycurl.CAINFO, "/etc/ssl/certs/ca-certificates.crt");
c.setopt(pycurl.CAPATH, "/etc/ssl/certs/");

lilydjwg added a commit to lilydjwg/nvchecker that referenced this issue Feb 27, 2024

workaround a pycurl wheel issue to fix ci

e5751d2

pycurl/pycurl#834

swt2c changed the title ~~pycurl 7.45.3 not working in debian bullseye with python3.8~~ pycurl 7.45.3 wheel not working for https in debian/ubuntu systems Mar 1, 2024

swt2c mentioned this issue Mar 1, 2024

Add support for CURL_CA_BUNDLE_AUTODETECT #835

Closed

Tschuppi81 mentioned this issue Mar 4, 2024

Pin pycurl OneGov/onegov-cloud#1211

Merged

Tschuppi81 added a commit to OneGov/onegov-cloud that referenced this issue Mar 4, 2024

Org: Pinning pycurl to 7.45.2

4266881

pycurl/pycurl#834 TYPE: hotfix

manics mentioned this issue Mar 6, 2024

binderhub image: refreeze requirements.txt jupyterhub/binderhub#1827

Merged

cuu508 added a commit to healthchecks/healthchecks that referenced this issue Mar 12, 2024

Downgrade pycurl to 7.45.2

035989d

pycurl 7.45.3 returns error code 77 (CURLE_SSL_CACERT_BADFILE) for TLS connections on Debian-based systems: pycurl/pycurl#834

This was referenced Apr 4, 2024

[GRDM-42361] binderhubでビルドエラーが発生する RCOSDP/CS-binderhub#22

Closed

[GRDM-42361] binderhubでビルドエラーが発生する lifematics/CS-binderhub#7

Open

cat-bro mentioned this issue Apr 4, 2024

pin pycurl for pulsar usegalaxy-au/infrastructure#1883

Merged

zagrim mentioned this issue Apr 19, 2024

Telegram notification test: error setting certificate verify locations:\n CAfile: /etc/pki/tls/certs/ca-bundle.crt motioneye-project/motioneye#2962

Open

lilydjwg mentioned this issue Apr 23, 2024

HTTP 599 Error lilydjwg/nvchecker#266

Open

swt2c mentioned this issue May 15, 2024

error setting certificate verify locations #841

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pycurl 7.45.3 wheel not working for https in debian/ubuntu systems #834

pycurl 7.45.3 wheel not working for https in debian/ubuntu systems #834

ap-wtioit commented Feb 26, 2024 •

edited by swt2c

swt2c commented Feb 26, 2024

kurtseifried commented Mar 1, 2024

1951FDG commented Apr 12, 2024 •

edited

pycurl 7.45.3 wheel not working for https in debian/ubuntu systems #834

pycurl 7.45.3 wheel not working for https in debian/ubuntu systems #834

Comments

ap-wtioit commented Feb 26, 2024 • edited by swt2c

What did you try to do?

What happened?

What did you expect to happen?

Workaround

What is the PycURL version?

What is your Python version?

What is your operating system and its version?

Other versions

Is this the most recent PycURL release?

Did you check libcurl behavior?

swt2c commented Feb 26, 2024

kurtseifried commented Mar 1, 2024

1951FDG commented Apr 12, 2024 • edited

ap-wtioit commented Feb 26, 2024 •

edited by swt2c

1951FDG commented Apr 12, 2024 •

edited