Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MoveNextAsync could return the same key vs OutOfOrderKey #142

Open
buybackoff opened this issue Sep 22, 2018 · 0 comments
Open

MoveNextAsync could return the same key vs OutOfOrderKey #142

buybackoff opened this issue Sep 22, 2018 · 0 comments
Labels

Comments

@buybackoff
Copy link
Member

Case:

  • important channel A, slow values but decision making depend on it (e.g. trade size)
  • fast channel B, e.g. ticks

then we calculate C = Zip(Repeat(A), Signal(B)). But we cannot guarantee total order of messages. Item from A could arrive later but have earlier time stamp than the last item from B. In sync case nothing could be done.

But if we MNA in a loop, then the item from A will call TryComplete in Zip. It will have smaller TS than the current Zip value, and now it will be just ignored (actually, Zip will try to MN and won't). So we could have a newer value in A, but the last result in C does not reflect it until a new value from B.

But MN could detect that there is a new value in A and recalculate current or even several values from A key. Zip already had/has some code for this, but it is either switched off or just throws OutOfOrderKey.

A problem could be that we have too many exceptions if we write streams in batches. One solution is to try hard to keep order. If a source has total order then we do not break that stream in independent streams but write the entire fat stream and then create projections, that could be cached on disk. Disadvantage is that there will be too much cached duplicate data or we lose random access and be forward-only.

If MN/MNA could return the same key with updated value, final consumers could deal with this but how to propagate this?

TBD...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant