Update librdkafka version

[gsebil08]

Hello,

Thanks again for this library, we use it in production.
Recently, we wanted to update our dependencies, and we are now facing an ssl.ca.location error while trying to consume Kafka messages (seems to be macOS related, and fixed by this).

{"at":"2021-10-15T15:06:35.627932Z","severity":"Warning","app":["application"],"markers":{},"msg":"Librdkafka log: KafkaLogErr FAIL: [thrd:ssl://<uri>]: ssl://<uri>/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 33ms in state SSL_HANDSHAKE)","thread":"38"}
{"at":"2021-10-15T15:06:35.627932Z","severity":"Warning","app":["application"],"markers":{},"msg":"Librdkafka log: KafkaLogErr ERROR: [thrd:ssl://<uri>]: 1/1 brokers are down","thread":"39"}
{"at":"2021-10-15T15:06:35.627932Z","severity":"Warning","app":["application"],"markers":{},"msg":"Librdkafka log: KafkaLogErr ERROR: [thrd:app]: rdkafka#consumer-3: ssl://<uri>/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 33ms in state SSL_HANDSHAKE)","thread":"40"}

However, it looks like you are still using a pretty old version of librdkafka: confluentinc/librdkafka@849c066 which had been released in 2018.

Do you plan soon or later to update your librdkafka version dependency?

Regards,

Gauthier.

[gsebil08]

I just tried naively to update on my side, by updating the file build-librdkafka

RDKAFKA_VER="063a9ae7a65cebdf1cc128da9815c05f91a2a996"

And all the tests passed:

Finished in 50.6026 seconds
51 examples, 0 failures
Test suite integration-tests: PASS
Test suite logged to:
/hw-kafka-client/dist-newstyle/build/x86_64-osx/ghc-8.10.7/hw-kafka-client-4.0.3/t/integration-tests/test/hw-kafka-client-4.0.3-integration-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

Then I referenced my local hw-kafka-client in the stack.yaml of my project, to try this new version.
Unfortunately, it does not solve my certificate issue:

{"at":"2021-10-18T09:43:46.656207Z","severity":"Warning","app":["product-aggregator"],"markers":{},"msg":"Librdkafka log: KafkaLogErr FAIL: [thrd:ssl://<uri>]: ssl://<uri>/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 108ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)","thread":"41"}
{"at":"2021-10-18T09:43:46.656207Z","severity":"Warning","app":["product-aggregator"],"markers":{},"msg":"Librdkafka log: KafkaLogErr ERROR: [thrd:app]: rdkafka#consumer-3: ssl://<uri>/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 108ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)","thread":"42"}

So I guess it was not linked to this bug.

Do you want me to open a pull request with the RDKAFKA_VER to the latest commit?

Do you have any other clue for my trouble?

[gsebil08]

Hello,

Any news regarding my question/trouble?

[gsebil08]

I forgot to update the status here.
Trouble was coming from rdkafka provided via my own shell.nix. Version 1.8.0 introduced a bug.

This issue is now only about upgrading RDKAFKA_VER="063a9ae7a65cebdf1cc128da9815c05f91a2a996".