Replies: 4 comments 2 replies
-
@karls thanks for taking the time to set up an example. I'm assuming that when you state "long running publisher" you mean "slow publisher that blocks internally" which, yep, would block Pika's I/O loop. What isn't clear in the docs with regard to Feel free to submit a PR for adding your code as well as expanding the documentation to make it clearer how to use |
Beta Was this translation helpful? Give feedback.
-
Hey @lukebakken.
Pretty much yep. Long-running, waiting for data on a queue/webhook/whatever. I suppose if it publishes very slowly there isn't really a need to keep the connection open. Just in my case the traffic was very variable.
I guess if I had been more intimately familiar with how AMQP works I'd have spotted this earlier. From the docstring it's not clear why should I care about data events (what are they even?) being processed. Again, it's very likely down to my own ignorance, but I suspect there are many people like me who read some docs, see that a connection should be opened and kept open, develop a mental model of how it should work, and are then puzzled that it doesn't quite do that. I'll put together a PR with an example and maybe a few tweaks for the docstring and we can collaborate in the PR from there. |
Beta Was this translation helpful? Give feedback.
-
Hey @lukebakken Apologies for the delay, couldn't find the time to sit down and flesh this out. I've created #1384 with an example and an updated docstring for |
Beta Was this translation helpful? Give feedback.
-
Hi @karls, This example is really useful and completely
Thanks so much for sharing! Kind regards |
Beta Was this translation helpful? Give feedback.
-
Hi,
I recently had to work through how to create a consumer-publisher pair for the platform I'm helping build. After setting it all up it seemed to work OK when there was lots of traffic on the upstream queue, but during periods of lower traffic the publisher would get its connection closed and the RabbitMQ (AWS MQ) logs would show a missed heartbeat failure.
I was super puzzled by that and spent several days going through the docs, examples, Googling, etc. It seems like there are quite a few people hitting the same or a similar issue.
What ended up being the issue was not realising that the publisher cannot just wait on an internal queue or some other mechanism for messages, as that would block the IO loop for the Pika connection, causing it to miss heartbeats. However, the publish example doesn't in any way indicate how one would set up a long-running publisher — as part of a web-server/crawler/etc. I suspect many people take that example and modify it enough to get it to work for their purposes, often working around the closed connection issues with a
try...except
hack.Do you think it'd be a good idea to include an example in
examples/
of how one would set up a long-running publisher? And maybe include relevant information in the docs forConnection.process_data_events
?I've created a repo, which I included in my SO answer, which demonstrates — hopefully — the correct way to run a long-running publisher (feedback/tips very welcome). The code in the repo covers a bit more ground than is strictly needed for a self-contained example, but it could be adapted pretty easily.
Beta Was this translation helpful? Give feedback.
All reactions