[QUESTION] Does aiokafka can replace the unmaintained kafka-python project?

[LiorA1]

Hi,

I'm new to Kafka and trying to get an initial sense of the available libraries & theirs status.

Does 'aiokafka' still based on top of the unmaintained kafka-python project?

P.S-
Maybe its worth to open a "Discussions" tab ?

[Lameorc]

Seconding the above, I'll add some more context:
The kafka-python library has several discussions as to whether it's maintained or when new release is happening:

• Is this library maintained? Is there a fork that is more maintained? dpkp/kafka-python#2349
• is kafka-python deprecated? dpkp/kafka-python#2290
• Release plan for kafka-python dpkp/kafka-python#2353

The main reason this is of interest to me is that the kafka python doesn't support the latest kafka versions and I don't expect it will based on the above discussions. I've not managed to find any sort of compatibility matrix but we're trying to switch to upgrade kafka v3+ which seems impossible with this library as of now?

[vmaurin]

I am not 100% sure as I haven't checked actively kafka release notes since v2.8.x, but Kafka protocol is known to be forward and backward compatible since v0.8.x

Part of client connection, there is a phase of protocol supports, so a client lib with the 0.8.x protocol should be able to talk to a broker with 3+. Then it might not be always the optimal way, and you will lack new protocol features

[Lameorc]

Huh, shame on me for not double checking things I've been told... I'll try and integrate my app with newer kafka and see how that goes. Thanks for the hint!

[jainal09]

So, is there a roadmap ahead for aio-kafka. What is the plan for supporting future kafka releases if kafka-python is not maintaining there source. Also, are there plans to replace kafka-python with confluent-kafka-python library which is well maintained and up-to date

[marcosschroh]

Hi all,

We have migrated our kafka cluster to the latest kafka versión and everything kept working. We run some tests against kafka-python /aiokafka and we did not find any issues. It is true that the new kafka version indicates a breaking change but the API remains the same. Anyway, it is a bit worrying that aiokafka depends on kafka-python (unmaintained)

[wbarnha]

Which Kafka version includes the breaking change out of curiosity? I'll make note of this in kafka-python.

[ods]

There is a plan to remove dependency on kafka-python. But the main problem is that we need more maintainers for aiokafka project.

[wbarnha]

I've had enough with Confluent's Python implementation and I'm fully divorcing their codebase. Its documentation is quite lacking, compared to aiokafka. I volunteer to assist in maintaining this project. This is the only project that actively maintains and releases a high-quality Kafka client library for Python with proper documentation, and is critical to the stability of many production infrastructures all over.

Unfortunately, despite my best efforts to revive kafka-python, I do not have access to PyPi. So I can't release packages unless it's under a new name, such as kafka-python-ng, and then I'd propagate that change here if we wanted to add additional Kafka Protocol message structures.

[LiorA1]

in favor of confluent-kafka-python ? Or to be a stand-alone ?

to open a tab of required tickets somehow cant motivate people pick them up?

[wbarnha]

Hi folks,

As of recent events, I have received push access for the existing kafka-python project. I'm aware of aiokafka's current dependency of kafka-python, and I have received access from the original maintainers to revive the kafka-python project. The current plan is to migrate the CI/CD pipeline for kakfa-python to GitHub, and then immediately create a release of the current state of the project. After that, I'll review existing PRs for kafka-python, and hopefully eliminate the need for us to worry about our kafka-python dependency.

Re: confluent-kafka-python

Sadly, this project is an entirely different beast when it comes to asynchronous functionality. There's still some work missing from the project and it's not suitable to completely replace aiokafka. confluentinc/confluent-kafka-python#1448 for example has been pending merge for quite some time and most of their library is used in synchronous contexts. I'm not sure what their intended direction for that project is, but until there's obvious asynchronous support, I'm trying to avoid it. I've (unsuccessfully) performed various experiments with confluent-kafka-python in asynchronous contexts inside faust-streaming/faust#418 and I'm not particularly fond of Python's Kafka projects to be entirely dependent on Confluent's library. The librdkafka bindings built for the project are another beast that can be difficult to troubleshoot. I'd go as far as to argue that the current aiokafka + kafka-python implementation in pure Python/Cython has some benefits offered in typing and compatibility with other Python implementations such as PyPy.

I understand the perils of relying on FOSS maintainers, many people such as myself are trying to balance their jobs and personal lives and are inevitably fated to depart FOSS support at some point due to circumstances beyohd their control. I do think that hat we should look into moving towards a community that has more say in voting on the direction of aiokafka and handling pending PRs.

I would be willing to step up to the plate to support aiokafka, but I'm already burdened by Faust and Kafka-Python, I don't want to neglect other projects.

[wbarnha]

Wrapping up a workflow for kafka-python in dpkp/kafka-python#2378, hopefully this will be the change that brings the project back to life!

[ods]

in favor of confluent-kafka-python ? Or to be a stand-alone ?

Standalone. The plan is to incorporate the code aiokafka depends on.

[aiven-anton]

@sivaNbalusu5 Sorry the process is taking some time but making good progress, I'm hopeful I can show something no later than end of next week, but cannot make any concrete promise. I'm personally pretty excited about it, and doing what I can :)

[aiven-anton]

Alright, I can finally share this: https://github.com/Aiven-Open/kio

[aiven-anton]

I have some follow-up tasks to do regarding publishing docs, and adding installation instructions, but the fundamentals are there. Please ask if I can shed light on something.

[aiven-anton]

@wbarnha I'm curious whether you've had any time to review this and if so whether you see some value in it. Note that there is a docs/ directory with basic usage examples here.

[wbarnha]

Sadly, I've been very short on time as of recently for open source development, so I have not reviewed the documentation yet. Thank you for providing me a link! I'll review this further when I have more time but the examples so far are very helpful.

[sdy72]

Hi, I am new here. Since we are moving to use Python based Kafka clients this thread is nice reading.
Keeping up with Kafka versions, would that also address this issue when creating a topic?

kafka.errors.IncompatibleBrokerVersion: IncompatibleBrokerVersion: Kafka broker does not support the 'MetadataRequest_v0' Kafka protocol.

Using AIOKafkaAdminClient against a fairly new Kafka server confluentinc/cp-kafka (on Docker for testing)

[aiven-anton]

@sdy72 It's at least definitely related, I'm not sure when that version was dropped though. The library mentioned here implements all versions of all protocol messages, so could potentially be used to fix this as well. Hoping to have some news about publishing soon.

[DavidOsparks]

I'm also new here, and I'd like to know what's the goal of aiokafka 1.0 and approximately when can we see it? Is there any document for this?

[vmaurin]

Since version 0.9, it is a standalone lib, so not depending on kafka-python anymore (but use 0.10 as it fixes some dependency issues). In my company, we use it in production for a while now without any issues, so I don't think you need to wait for a 1.x version to move forward with it

[DavidOsparks]

Yeah, I will. Just be curious about what's the milestone of 1.0. Will there be any exciting new feature we can expect from version 1.0?

[ods]

@DavidOsparks There is not many people willing to contribute to the project. My priorities are to fix known bugs, add type annotations and make some internal changes to simplify maintenance.