HTTP publishing body size limitation

[grignards]

Hello.
We are using pushpin for real-time updates in the last version of our product Tracim.

During our tests we got unpublished messages when their size is big. After investigation the problem is linked to be HTTP publishing which limits the content's body size. If I'm not wrong, it is this line https://github.com/fanout/pushpin/blob/e1312a54c0db0a4e381583de6b5e8d3708c67204/src/handler/simplehttpserver.cpp#L38
Publishing the same content through 0mq is fine.

Is the HTTP limit a "hard" one, meaning that pushpin is not meant to handle bigger contents? Or could it be made configurable?

[jkarneges]

I suppose this ought to be made configurable, and the size limits between HTTP and ZeroMQ inputs should be more consistent. However, large messages are not recommended as they can slow down queues. For comparison, Fanout's cloud service has a 64kB size limit.

Thought experiment: if you had to pick a different size limit and couldn't change it later (meaning it ought to suit your current needs and any future ones), what would you choose?

Also, what transport is being used by the subscribers?

[grignards]

OK, it was not our primary intent to publish large messages, more a side-effect of our time constraints.
Currently our biggest messages are around 4MB but we'll likely change their structure to not include all data which would make their size drop around 1kB. We are using Server-Sent Events as transport.
In the mean time we can use 0mq although I would have preferred to stick with HTTP publishing to avoid 0mq's persistent connections between our backend and pushpin. If you're willing to accept a PR making the HTTP body size configurable I can have a look at it.
Is the main effect of "slowing down queues" a delay in message publishing?

[jkarneges]

With the HTTP streaming transport (which you are using if you're doing SSE), it is possible to use reliable streaming to set a URL to fetch data from and then publish a hint instead of the actual data. This will cause Pushpin to fetch the data from the URL and send it to the client, in which case the data has no size limit.

However, it might be easier to just increase the publishing size limit. A PR for that would be great.

By "slowing down queues" I mean if you add a broker layer between your publishing code and Pushpin. Not a problem now but something to be aware of as you grow.

[grignards]

Thank you for the additional information.
I just created a PR to add configuration for http max body size and max headers size.