api_ca does not default to system cert bundle

[nlvw]

What is the issue ?

When using ARA 1.6.0 with a HTTPS API Server the callback plugin will fail due to a certificate error. It seems like it is not checking the default cert bundle on they system (Fedora 37 in my case) as the following fixes the error

[ara]
api_client = http
api_server = https://ara.example.com
api_ca = /etc/pki/tls/cert.pem

Without api_ca the connection fails with

[WARNING]: Failure using method (v2_playbook_on_start) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
HTTPSConnectionPool(host='ara.nmsu.edu', port=443): Max retries exceeded with url:
/api/v1/playbooks (Caused by SSLError(SSLCertVerificationError(1, '[SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer
certificate (_ssl.c:992)')))

PLAY [Test Ansible Connection] ************************************************************
[WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
'NoneType' object is not subscriptable

TASK [connection test] ********************************************************************
[WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
'NoneType' object is not subscriptable
ok: [hpc-wiki-p]

TASK [stop ssh client persistant connection] **********************************************

PLAY RECAP ********************************************************************************
hpc-wiki-p                 : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin
(<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f4b47a812d0>):
'NoneType' object is not subscriptable

What should be happening ?

The default ca bundle trusted by the system should be used by default when api_ca is not specified

Software Versions

Ansible = 7.1.0 (Core = 2.14)
OS = Fedora 37
Python = 3.11
ARA = 1.6.0
Install Source = pip

[dmsimard]

Hi @nlvw and thanks for the issue.

Unless mistaken, the client doesn't specify a path to a CA bundle at all (even a default one) when doing requests unless there is one provided via the configuration so I would expect that to come from python or the requests library.

Out of curiosity, could you use pure python requests to see if it also gets a SSL error?

Something like this (replacing the demo URL with your own) should be sufficient:

> python3 -c 'import requests; print(requests.get("https://demo.recordsansible.org/api/v1/").json())'
{'labels': 'https://demo.recordsansible.org/api/v1/labels', 'playbooks': 'https://demo.recordsansible.org/api/v1/playbooks', 'plays': 'https://demo.recordsansible.org/api/v1/plays', 'tasks': 'https://demo.recordsansible.org/api/v1/tasks', 'hosts': 'https://demo.recordsansible.org/api/v1/hosts', 'latesthosts': 'https://demo.recordsansible.org/api/v1/latesthosts', 'results': 'https://demo.recordsansible.org/api/v1/results', 'files': 'https://demo.recordsansible.org/api/v1/files', 'records': 'https://demo.recordsansible.org/api/v1/records'}

For what it's worth, I am also running on Fedora and do not need to specify ``api_ca` when using the demo and it works with a letsencrypt cert.

I know there's an env variable to set different bundle paths but that shouldn't be necessary.

[hille721]

@dmsimard I guess in your case it's not a self signed certificate. As soon as you are using a self signed certificate you have to give the path to the system ca bundle to the Python requests library, e.g. by simple setting REQUESTS_CA_BUNDLE.
Have to do the same for my Ara installations.

[hille721]

And the path differ between e.g. Ubuntu and Fedora. Thus hard to set a default there.

[dmsimard]

@hille721 I don't have bandwidth to try and reproduce right now but we have a working integration test to validate that it works for self signed certificates here: https://github.com/ansible-community/ara-collection/pull/57/files and it looks sufficient to specify api_ca.

It hasn't run in a while but I will trigger it to get some fresh results.

[dmsimard]

I can try and investigate later but in the meantime the job returned successfully: https://ansible.softwarefactory-project.io/zuul/build/50e09cec38da4fafbc5dc57136d8f1fe

[hille721]

Don't get me wrong. Don't see this as a bug or problem. We need to set REQUEST_CA_BUNDLE not only for Ara but for all our Python tools in which the requests library is used and there we have a lot. Thus not a problem for me and that's why I won't spend time for further investigation.

[hille721]

And just to clarify, we don't set any certificate / ca-bundle in the Ara config, but ONLY the REQUEST_CA_BUNDLE environment variable.

So either you have to set api_ca in Ara config which will pass this to requests as verify argument OR you can directly specify the REQUEST_CA_BUNDLE env variable without special ara config. Which makes sense if you are having more tools which are using the requests library

[hille721]

Using requests with not self signed url:

(.venv) $  python3 -c 'import requests; print(requests.get("https://demo.recordsansible.org/api/v1/").json())'
{'labels': 'https://demo.recordsansible.org/api/v1/labels', 'playbooks': 'https://demo.recordsansible.org/api/v1/playbooks', 'plays': 'https://demo.recordsansible.org/api/v1/plays', 'tasks': 'https://demo.recordsansible.org/api/v1/tasks', 'hosts': 'https://demo.recordsansible.org/api/v1/hosts', 'latesthosts': 'https://demo.recordsansible.org/api/v1/latesthosts', 'results': 'https://demo.recordsansible.org/api/v1/results', 'files': 'https://demo.recordsansible.org/api/v1/files', 'records': 'https://demo.recordsansible.org/api/v1/records'}

Using requests with self signed certificate, wont't work

(.venv) $  python3 -c 'import requests; print(requests.get("https://raspi4.fritz.box:9090/"))'
Traceback (most recent call last):
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/lib64/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hille/.venv/lib64/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/hille/.venv/lib64/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raspi4.fritz.box', port=9090): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/hille/.venv/lib64/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/hille/.venv/lib64/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/hille/.venv/lib64/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/hille/.venv/lib64/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/hille/.venv/lib64/python3.10/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raspi4.fritz.box', port=9090): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

Simple set REQUESTS_CA_BUNDLE and it will work:

(.venv) $  REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt python3 -c 'import requests; print(requests.get("https://raspi4.fritz.box:9090/"))'
<Response [200]>

tested with Fedora 36 and a Virtual Environment.

Fun fact: If I'm using not a virtual environment, but the system Python it will work without setting REQUESTS_CA_BUNDLE.

$ echo $REQUESTS_CA_BUNDLE

$ python3 -c 'import requests; print(requests.get("https://raspi4.fritz.box:9090/"))'
<Response [200]>
$ which python3
/usr/bin/python3

[dmsimard]

Yes, it's a common enough issue that I have seen with python things before but I had not noticed it with ara (yet?).

I suppose since this is a known issue we should add a little something to the troubleshooting docs: https://github.com/ansible-community/ara/blob/master/doc/source/troubleshooting.rst

I'll leave this issue opened in the meantime.