You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
important channel A, slow values but decision making depend on it (e.g. trade size)
fast channel B, e.g. ticks
then we calculate C = Zip(Repeat(A), Signal(B)). But we cannot guarantee total order of messages. Item from A could arrive later but have earlier time stamp than the last item from B. In sync case nothing could be done.
But if we MNA in a loop, then the item from A will call TryComplete in Zip. It will have smaller TS than the current Zip value, and now it will be just ignored (actually, Zip will try to MN and won't). So we could have a newer value in A, but the last result in C does not reflect it until a new value from B.
But MN could detect that there is a new value in A and recalculate current or even several values from A key. Zip already had/has some code for this, but it is either switched off or just throws OutOfOrderKey.
A problem could be that we have too many exceptions if we write streams in batches. One solution is to try hard to keep order. If a source has total order then we do not break that stream in independent streams but write the entire fat stream and then create projections, that could be cached on disk. Disadvantage is that there will be too much cached duplicate data or we lose random access and be forward-only.
If MN/MNA could return the same key with updated value, final consumers could deal with this but how to propagate this?
TBD...
The text was updated successfully, but these errors were encountered:
Case:
then we calculate
C = Zip(Repeat(A), Signal(B))
. But we cannot guarantee total order of messages. Item from A could arrive later but have earlier time stamp than the last item from B. In sync case nothing could be done.But if we MNA in a loop, then the item from A will call
TryComplete
in Zip. It will have smaller TS than the current Zip value, and now it will be just ignored (actually, Zip will try to MN and won't). So we could have a newer value in A, but the last result in C does not reflect it until a new value from B.But MN could detect that there is a new value in A and recalculate current or even several values from A key. Zip already had/has some code for this, but it is either switched off or just throws OutOfOrderKey.
A problem could be that we have too many exceptions if we write streams in batches. One solution is to try hard to keep order. If a source has total order then we do not break that stream in independent streams but write the entire fat stream and then create projections, that could be cached on disk. Disadvantage is that there will be too much cached duplicate data or we lose random access and be forward-only.
If MN/MNA could return the same key with updated value, final consumers could deal with this but how to propagate this?
TBD...
The text was updated successfully, but these errors were encountered: