Stream utilities on top of keystone

[PEZO19]

Coming from a CycleJs and (its engine) xstream background, I am wondering how to get best from both keystone's amazing feature set (live tree, references, transactions, UndoManager, snapshots, patches, async flow, middlewares, Class/Data Models) with something like xstream's feature set (hot streams, sync update semantics, reactive programming instead passive, higher order streams (streams of streams, .flatten()/.flattenConcurrently()/.flattenSequentelly())).

"Action" (/method call) based imperative/passive programming feels weak to me after using xstream and expressing the program through reactive streams. As I understand, MST and keystone are reactive "on the surface of (Root)State(s)", but not necessarily reactive regarding to the "substeps"/"subprocedures", "under the hood". I am wondering if it is possible to overcome that and I am asking for your help to understand the (theoretical) limitations.

First, I was wondering about something like mst-effect, but even that (being able to use standalone-like actions driven by rxjs code(?)) feels a bit limited, so now I am wondering if I could implement a very simple stream lib, like xstream on top of keystone and wondering about the gains and limitations.

On basic level, that would mean, that 1. I could express the code through streams 2. which streams have all their state in keystone, so "undoing" / middlewares / patches, etc. could be used with it together. Async stream operators could be implemented via flow, this does not seem to be a problem. However, maybe huge part of the possible gains could be earned without such a stream lib, using only the existing onPatch/onSnapshot/others, but it is not clear to me due lack of keystone/MST xp. Furthermore, it is not really clear to me what gotchas this "dual" approach would introduce.

@xaviergonz Can you please tell me your opinion on such an approach? How would you write the "most reactive" (stream based?) code with the existing features? What do you think the worst parts of such a "dual" approach would be?

[xaviergonz]

Do you have a code example of how you'd want / expect it to work? It is easier for me to visualize when I see code rather than theory.

[PEZO19]

@xaviergonz Sure. I think multiple approaches are valid from quite different angles, but here is a simpler problem/idea I test against in theory:

Given something like this:

@model('MyModel')
class MyModel extends Model {

@modelAction
myModelAction1(arg11: Arg11, arg12: Arg12){
   do_stuff_with_imperative_or_passive_procedure_call1(arg11, arg12)
  // later when requirements change, 
  // if you want to "do more" with information available here, in `myModelAction1`,
  // you have to modify it (and the whole procedure/call stack cascade)
  // eg. if `Arg11` changes, you have to change it everywhere 
  //(here in 2 places: myModelAction1, do_stuff_with_imparative_or_passive_procedure_call1))
}

@modelAction
myModelAction2(arg21: Arg21, arg22: Arg22){
   do_stuff_with_imperative_or_passive_procedure_call2(arg21, arg22)
}
}

// I assume this Stream object must be implemented in the living tree (like rootRefs...?) to support keystone features
const myModelAction1$ = createStreamFromAction(MyModel, 'myModelAction1') // types can be inferred!
const myModelAction2$ = createStreamFromAction(MyModel, 'myModelAction2') // types can be inferred! 

// I am aware that onAction exists, but not sure about this level of type support and
// onAction feels "too high level" / "too observing-only(?)" and
// not sure how it'd play with other features - and middleware - (how to undo something happened in onAction, etc...)
// onAction ??? How that'd work? 
// "createStreamFromAction" feels like a pleasant `onAction` tailored for that purpose

// createStreamFromAction is not the only useful stream, we could have
// createStreamFromComputed or something similar/others

// The point is
// 1. easy referring to actions as streams while keeping type safety (action = typed event)
// 2. using stream operators and while doing that, keeping mobx-keystone transaction/undo/patch features/semantics
// (3. with streams, types are automatically inferred, if myModelAction1$ changes, first_5_combined$ changes accordingly automatically)

const first_5_combined$ = 
    CombineStreamOperator(myModelAction1$, myModelAction2$)
    .MapStreamOperator(([myModelAction1, myModelAction2]) => { // .MapStreamOperator = .map in rxjs, just wanted to be explicit here
        const {name, params: {arg11, arg12}} = myModelAction1 // types can be inferred!
        const {name, params: {arg21, arg22}} = myModelAction2 // types can be inferred!

        // being able to do something with the (combined) actions (which are like "typed events")
        // using neat Stream operators (CombineStreamOperator, MapStreamOperator)
        return "something"
        // todo: here we should be able to do any state mutation in living tree too... 
        // which would belong to the same transaction... 
        // (transaction started by (last) myModelAction1 or myModelAction2, the sole inputs of first_5_combined$)
    })
    // being able to do use other "standard"/"well-known"/"easy to use" Stream Operators: eg.: "take"
    // this would mean: we are only interested in first 5 "events"/"actions" coming **out** from CombineStreamOperator,
    // everything after that is ignored
    .take(5)
    // if something above it would throw, then the whole (keystone) transaction would be "rolled back",
    // but with replaceError (which is a StreamOperator) we can just eat that error
    .replaceError(() => 'OK__ignored_error') 

Note, that logic can be written from top to bottom instead from bottom to top.
With passive programming we have to modify myModelAction1 and myModelAction2 if we want to make them "more", eg. when more computation is based on them while our system/requirements evolve.

With (streams/)reactive (which "reactive" keystone supports from many angles, I understand that) we could once create/declare an abstraction (myModelAction1$, myModelAction2$) and use it multiple times without changing its implementation/meaning itself.

I also understand, that this is maybe not the way for heavy state mutating parts of the app (not sure, maybe very useful for that too once thought out!, but maybe limited for that...), but for ephemeral state and "Operator like" mini state machines (like CombineOperator, TakeOperator, ReplaceErrorOperator) it would be really powerful sometimes I think.

Creating / destroying related objects is a question though to me, but this was in the original question too: I myself am not really sure what can even make sense theoretically: eg. beyond: createStreamFromAction and createStreamFromComputed would it make sense to createStreamFromNewData<T> which new data is created by this function itself (completely new objects in keystone, in a registry/via rootRef, etc.)?

Please let me know if that was clear enough or not, I'd be really happy to have a conversation about it. Also, if these are "solved issues" / there are workarounds possible, then I am interested too, I do not want to push my thoughts too much, just looking for answers to problems. :)

Another, a bit related question: how familiar are you with rxjs/xstream/other stream libs? The better I know your background on these, the better examples I can give. Is there something you liked/disliked about these techs?

@sisp May I ask for your input as well? I have seen that you are a veteran too here in the issue section :)

[PEZO19]

@xaviergonz One more addition:

Reusing onActionMiddleware would be really nice like
myAction$ = createStreamFromAction(onActionMiddleware, subtreeRoot, ‘myActionName’, 'onFinish')
with inferred name + args type, however I need access to the "inner" (not just topmost level) actions as well which are being called. Is it possible to expose these here (via onActionMiddleware) or elsewhere (that'd be better I think, if possible, I am a bit afraid of middlewares for that, but maybe it is only the true canonical way)? The main reason it would be better to expose elsewhere (not in middleweres only) is that this way I could write business logic via "normal" actions, depending on actions and its "business state" could live outside of middlewares. I do not really want to put business state in middlewares for multiple reasons, beyond not being as clean/clear conceptually, it is not clear how it would behave together with other middlewares/ other keystone aspects/features.