Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow .subscribe() to accept array of topics #640

Closed
WaleedAshraf opened this issue Feb 13, 2020 · 4 comments
Closed

Allow .subscribe() to accept array of topics #640

WaleedAshraf opened this issue Feb 13, 2020 · 4 comments

Comments

@WaleedAshraf
Copy link
Contributor

As discussed in these comments,
#289 (comment)
#289 (comment)
by @ThisIsMissEm and @tulios.

The limitation on KafaJS is that it doesn't let you subscribe to multiple topics at once with an array of topics. That fixes the issues of having multiple consumers for multiple topics.

It's be something like

await consumer.subscribe([
	{ topic: 'topic-a', fromBeginning: true },
	{ topic: 'topic-b', fromBeginning: true }
])

Implementation
It can be possible to pass the array of topics to .subscribe() method, and it will be used as const topicsToSubscribe = [] here https://github.com/tulios/kafkajs/blob/master/src/consumer/index.js#L163

@tulios
Copy link
Owner

tulios commented Feb 13, 2020

@WaleedAshraf it's not possible to pass an array but you can call subscribe multiple times or use subscribe with a regexp.

@WaleedAshraf
Copy link
Contributor Author

@tulios, sorry, maybe I misunderstood the comment in other issues.
You mean it is possible to do this:

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
})

const consumer1 = kafka.consumer({ groupId: 'test-group' })

await consumer1.connect()
await consumer1.subscribe({ topic: 'topic-a', fromBeginning: true })
await consumer1.subscribe({ topic: 'topic-b', fromBeginning: true })
await consumer1.subscribe({ topic: 'topic-c', fromBeginning: true })

The groupId is the same, and the consumer is subscribed to multiple topics.

@tulios
Copy link
Owner

tulios commented Feb 13, 2020

Yes, this is possible and used in production. You can also do:
await consumer1.subscribe({ topic: /topic-*/, fromBeginning: true })

@WaleedAshraf
Copy link
Contributor Author

WaleedAshraf commented Feb 13, 2020

Thanks, 👍 That solves my problem. I'll close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants