Add chunking (streaming) to broker.broadcast

[jeffrson]

Is your feature request related to a problem? Please describe.

I'm using a couple of services with moleculer and NATS. NATS has a config for max payload - if that's exceeded I would get a MAX_PAYLOAD_EXCEEDED exception. This is why I use streams in broker.call, which works fine.
However, when I notify other services via broker.broadcast, streams are apparently not supported, so I indeed get that MAX_PAYLOAD_EXCEEDED error sometimes.

Describe the solution you'd like

Add chunking/streaming to broker.broadcast.

Describe alternatives you've considered

Well, I could increase the config setting. But this is probably just a temporary "workaround", since it is not recommended to set it larger than 8MB (while having a max value of 64MB).

Additional context

Many thanks for moleculer :-)

[AndreMaz]

Don't know your use-case but have you considered reversing the logic?

Instead of adding chunking/streaming to broker.broadcast(<my-event>) why don't broadcast all the info that's required to get the desired data? This way each consumer of the <my-event> will receive the info about the data location (e.g, URL, ID, blob, etc.) and fetch it via ctx.call()

[jeffrson]

Yes, I did. And yes, it would not be completely impossible. But:

For the use case: our applications that subscribe to events (by moleculer decorators) handle data rather generic (via JSON and "dictionaries"). Such we would need case distinctions which do not fit quite well into our current infrastructure.

Also I think (IMHO :-) ) moleculer would benefit from this feature, because it feels like an omission to support streaming in one direction of data transport, but not in the other.