project update - ryvencore

[leon-thomm]

Hey! This is an attempt to try out the discussions feature, I think this could be great, and to give interested people an update on the project. If you have any ideas or comments/feedback, please use this to mention them, I think you can also open your own threads for new topics. Here I'll just give a little insight into what I've been working on and what's in the pipeline, cuz I think it's pretty cool.

So, some of you already know, based on the Ryven project I've been working on abstracting away a framework for building these flow-based visual scripting editors. I think Ryven's basic concept is great, however Ryven is supposed to be rather "general purpose" and the true power for this paradigm lies in the optimization for specific domains. And I've made a lot of progress during the last weeks, so the first (presumably buggy) alpha will probably be released very soon. This new thingy, called ryvencore, will be an easy to use python library to create Qt based Ryven-like editors for python, so they can be optimized towards specific domains, no matter whether it regards workflow automization, machine learning, a web API, realtime data processing or whatever.

Future Ryven versions, i.e. from Ryven 3 onwards, will be based on ryvencore too. I also made a lot of progress there but this might take a bit more time because there are some features like code generation that don't really work yet and I won't be able to work on it so much during the next months, as my study is going to be quite demanding. If you want to see how Ryven will look like, take a look at the dev branch. But it's worth to mention that quite a few internal systems, such as the nodes system, were completely reimplemented and work very different in ryvencore now. Of course, ryvencore will also experience mayor changes in the future, but compared to the last Ryven version, the code and implementations have been improved a lot, to support scalability. ryvencore will provide you with the backend structure for all the abstract components (nodes, flows, scripts, session, vars manager, logger, etc.) as well as the whole GUI for the flows.

There really have been tons of improvements, but listing all the technical details would be boring, I'll probably summarize this in the release notes of Ryven 3, so here I'll just show two big additions that strongly affect the usage.

The first big thing is a prototype for function scripts, basically scripts that represent a function and are registered as node. A function script's flow has an input node and an output node and you can add and remove parameters and returns easily.

A function script's flow is just a normal flow, so you can add any node, even the function node itself. Here I made this flow of a function script foo:

And then you can just use the function script's node anywhere. When you modify the properties, like adding/removing parameters and returns, all instances will be updated.

Oh, right, and there is a completely new design system with new themes and they can now be modified using a json config.

Another thing that I worked on for months now is threading compatibility, and I think this is a really really big deal. Whenever you want to do some realtime analysis and/or modification on data, you want to thread your application. Since ryvencore also provides the GUI for the flows and everything contained, if you want to keep your abstract components in a separate thread while your GUI runs in the main thread, all communication between those components must be implemented accordingly. While this is not planned as a feature for the Ryven editor since it does complicate programming nodes, i.e. implementing communication between custom widgets and the node, and this can be a bit hard if you don't have experience in parallel programming, I feel like this feature could be of crucial importance in the future as it opens the door to the world of reatime data processing.

You can see a short demo editor I made below, where an expensive OpenCV effect is applied on a large image, but the whole computation happens in a separate thread. Just as proof of concept.

open_cv.mp4

Other than that, I added a bunch of things that make it more intuitive to use like collapsing nodes

hiding unconnected ports of nodes, node icons, html descriptions and so on.

A lot of work went into this and I think it's pretty neat. But what do you think?

Cheers.

[dumblob]

I'm following Ryven for some some time without actually trying it out. But about 11 years ago I did a master thesis which focused on BPMN (workflow stuff) and was looking for a similar tool as is being described here under the name "ryvencore". For the UI (user interface) I in the end stayed with https://demo.bpmn.io/new (it's open source), but that's just "sketcher" without all the interesting stuff under the hood (though admittedly I like the UI so much, that I'd now welcome something like that in Ryven/ryvencore).

I'm really glad you're working on it and will follow the development. Feel free to ask any questions as I did some thorough analyses back then and can probably answer even some advanced questions (among other things I'm a lot into "stream/live/reactive/..." concepts in UI, databases, etc. and parallelism).

[leon-thomm]

Thank you very much this is really helpful. I will follow your suggestion of specifying this behavior in the docs. I see your point of making the "backend" of ryvencore completely Qt independent, I know that this would be the professional (because theoretically more scalable) approach. I just don't have the skills and knowhow myself to do that in a clean and sustainable way, so I did my best (the only functional dependency of the abstract components to Qt is signals) and maybe people who have more experience with this can help in the future.

I took a very brief look at nng (with bindings like pynng it looks pretty neat). Are you suggesting using these probably rather network-based protocols to communicate between the threads? In which ways would this enable much more scalable deployment?

[dumblob]

Sorry for the late reply.

I will follow your suggestion of specifying this behavior in the docs.

That'd do 😉.

I took a very brief look at nng (with bindings like pynng it looks pretty neat). Are you suggesting using these probably rather network-based protocols to communicate between the threads? In which ways would this enable much more scalable deployment?

Looking at "passing data" not as separate messages/signals but as "continuous" channels/streams/flows of samples is advantageous in a similar way as is "structured programming" to "writing assembly". So my answer to your first question is yes (though nng under the hood can use zero-copy passing if on the same machine, so it's not worse to what you already have performance-wise).

My answer to the second question is simply, that it's much easier to integrate with channels/streams/flows (which have strictly defined semantics and behavior) than with callbacks/signals (which vary a lot - even if ryvencore would be Qt-agnostic but someone wanted to integrate with framework having slightly different signal/callback semantics). So it seems better to stick with such "middle layer" structured abstraction (channels/streams/flows) than no abstraction (signals/callbacks).

Last but not least, if ryvencore will already have some built-in high-performance data passing mechanism, then anyone integrating/using ryvencore doesn't need to think about it and just use it as well instead of trial-and-error with some other mechanisms. In other words "(small) batteries included" 😉.

And of course, you don't have to use nng or any such technology and provide your own channel/stream/flow mechanism though I'd strongly recommend to not reinvent the wheel.

[leon-thomm]

Thanks for the clarifications!

Looking at "passing data" not as separate messages/signals but as "continuous" channels/streams/flows of samples is advantageous in a similar way as is "structured programming" to "writing assembly"

That, however, also implies that all internal usages of the mechanisms for passing the data (the signals) have to adapt significantly, right?
The signals represent discrete events, to notify everyone who's connected of actions like instantiation of a node in a flow. In which way would such a "continuous flow of data"-approach differ from that?

If the nng-like approach is simple and doesn't complicate the implementations, it would be quite possible to abstract away a pure backend, which even suggests implementing different front ends. Though, right now 3/4 of the whole code regards only front end, so it would still be a lot of work (but just imagine having a JS-based front end too 😮). I already worked on this during the last weeks, and made the first successful tests of creating and saving a flow using the whole thing (my current ryvencore-based Ryven version), and then running it only using the backend, in just a few lines of code, like this

    # loading project str from project file
    [...]
    project_dict = json.loads(project_str)

    # creating session and loading the contents
    session = RCSession(no_gui=True)
    session.register_nodes([Button_Node, Print_Node, Val_Node])
    function_scripts, scripts = session.load(project_dict)
    
    # manually accessing contents
    flow = scripts[0].flow
    print_node, button_node, val_node = flow.nodes
    button_node.update()  # prints 'hi'

Why would you go for nng instead of more popular implementations like ZeroMQ, and do you have suggestions for other implementations that I should take a look at?

As a first start, I might release ryvencore-qt (I will never run out of names lol) with built-in backend and signaling support, but the sooner the backend provides Qt-agnostic communication the better. Assuming the used editor didnt enhance the nodes in ways that make them useless without the front end, flows could then run independently only using the backend... What do you think about this apprach? It still raises a lot of questions regarding implementation details for me, so as long as there won't be a few people with experience with this laying the foundation, it would be a rather long term process I think

[leon-thomm]

In which way would such a "continuous flow of data"-approach differ from that?

More specifically: as far as I understand, data must be explicitly received/expected/awaited (socket.recv()). How is an application that often needs to be actively notified by incoming, not specifically expected data usually wrapped into this?

[dumblob]

Hi Leon, I'm really sorry for the long delay. I wrote an answer few days after you posted your questions, but my browser crashed (Murphy, Murphy, Murphy 😢) and I lost the answer and since then didn't find any time to write again.

Looking at "passing data" not as separate messages/signals but as "continuous" channels/streams/flows of samples is advantageous in a similar way as is "structured programming" to "writing assembly"

That, however, also implies that all internal usages of the mechanisms for passing the data (the signals) have to adapt significantly, right?
The signals represent discrete events, to notify everyone who's connected of actions like instantiation of a node in a flow. In which way would such a "continuous flow of data"-approach differ from that?

This is my mistake - I should have mentioned it right up front, so bear with me. Technically it's the same. The difference is that "continuous" channels/streams/flow/... push the programmer to write more synchronous code. Whereas pub-sub (aka signals-slots) encourage the programmer to write asynchronous code. The problem with the latter is, that it becomes sooner or later spaghetti (not visually but semantically). That's not as bad on its own. The bad thing is that developing spaghetti gets every day more demanding and thus costly (both money-wise as well as psychologically 😉). But in case of ryvencore-qt this is also not a big issue as the app doesn't have the potential to "grow exponentially in complexity" over the years.

What might become a problem is the separation of the backend and front end. Again, not because it's a nice & clean architectural pattern, but in your case because you're trying to run GUI in one thread and the rest of the app in other threads.

For that signal-slots are also fine, but I'd recommend imagining each thread as one periodically (in an infinite loop) called function as a reaction to any change in any signal/channel. This function must then get all signals it potentially can react to as an argument. This way the body of the function becomes synchronous because already at the beginning of the function all values of arguments will be known and only later somewhere in the body of this function there'll be if( slot S1 received a new signal ){ react on it; }. This makes the relation of the signal event consequences clear and allows for changing signals-slots to channels (from a messaging library of your choice) by simply prepending the thread function (which is in the infinite loop) with a blocking select over all such channels (each corresponding to one signal).

The more synchronous each thread's event loop looks like, the significantly easier it is to decouple the app in the future and the slower the process of "spaghettying" 😉. Feel free to look at ECS as that's my go-to app decoupling approach which scales over many threads and gains traction during the last 10 years.

I originally even misunderstood some of the goals of Ryven, but now after reading the web page again I still think it's currently not necessary to abandon Qt's signals (as I said in #50 (reply in thread) ). It's sufficient to provide app binaries for Windows, Android (for tablets), macOS, iOS (again tablets), flatpak (Linux etc.). With Qt this should be much easier than decoupling the app and writing a new web front end from scratch.

If the nng-like approach is simple and doesn't complicate the implementations, it would be quite possible to abstract away a pure backend, which even suggests implementing different front ends. Though, right now 3/4 of the whole code regards only front end, so it would still be a lot of work (but just imagine having a JS-based front end too ). I already worked on this during the last weeks, and made the first successful tests of creating and saving a flow using the whole thing (my current ryvencore-based Ryven version), and then running it only using the backend, in just a few lines of code, like this

    # loading project str from project file
    [...]
    project_dict = json.loads(project_str)

    # creating session and loading the contents
    session = RCSession(no_gui=True)
    session.register_nodes([Button_Node, Print_Node, Val_Node])
    function_scripts, scripts = session.load(project_dict)
    
    # manually accessing contents
    flow = scripts[0].flow
    print_node, button_node, val_node = flow.nodes
    button_node.update()  # prints 'hi'

This is awesome! I guess this'll cover vast majority of use cases in practice and thus you'll probably never need to decouple the editor app into business logic & front end (I've deliberately not used the term "backend" because that's already used for the unattended mode app which executes in-editor-app-assembled-graph).

Why would you go for nng instead of more popular implementations like ZeroMQ, and do you have suggestions for other implementations that I should take a look at?

ZeroMQ is historically first one of that kind. And it has different downsides (except for being a larger dependency - both for development as well as when deployed, having performance problems - especially latency and poorer scaling over many threads, its use is less straightforward and connected with error prone API etc.). But the community is large and generally welcoming, so it's not as bad in practice 😉.

For more messaging libraries, take a look at Redis, RabbitMQ, AMQP, MQTT (this one is only pub-sub and thus smallest out of all), etc. Most need a broker instance (unlike Scalable Protocols libraries like nng, ZeroMQ, nanomsg, mangos, scaporust, ...).

Generally I think the winners here are all broker-less libraries which fully (natively) support websockets as transport layer and support x86 and ARM along with all of Windows, macOS, Android, iOS, Linux. I'd guess only nng passes this test though (ZeroMQ is close except for websockets - there are some unofficial extensions for websockets, but I don't have any experience with them, but wouldn't trust them as much as I do nng 😉 - but this is very subjective and biased).

As a first start, I might release ryvencore-qt (I will never run out of names lol) with built-in backend and signaling support, but the sooner the backend provides Qt-agnostic communication the better. Assuming the used editor didnt enhance the nodes in ways that make them useless without the front end, flows could then run independently only using the backend... What do you think about this apprach?

Yes, from the example above and the current description on the Ryven website, I understood that such an "unattended" mode already exists. Then the editor app could provide a menu item "Check executability in unattended mode" which would detect any nodes using stuff specific to the drawing app.

It still raises a lot of questions regarding implementation details for me, so as long as there won't be a few people with experience with this laying the foundation, it would be a rather long term process I think

I'll try to be more responsive. Don't hesitate and ask (even per email if you prefer).

In which way would such a "continuous flow of data"-approach differ from that?

More specifically: as far as I understand, data must be explicitly received/expected/awaited (socket.recv()). How is an application that often needs to be actively notified by incoming, not specifically expected data usually wrapped into this?

As noted above, the assumption is to structure the code in synchronous manner (it's safer, much more understandable, better extensible) per thread and then select over all the channels in an infinite loop (actually polling is being used often due to performance reasons, but that's implementation detail). In other words, transform the signal-slot asynchronicity into a synchronous system so that you can expect the events at one place. Asynchronicity shall happen only between truly concurrent pieces (os-level threads, os-processes, network nodes, ...) and shall be buffered on input if possible (without buffering it blocks the writer which would defeat the purpose of concurrency 😉).

[mmorys]

Hey, great work on this project!

I had been looking for something like this to implement in a project I'm working on, namely interactive data processing focused on plot visualization. I'm extending glueviz at the moment for the plotting part, and implementing Ryven for the function execution. I had implemented a crude tool like this simply using an interactive plot of circles and lines, and have some ideas to share from that experience.
I am happy to contribute modifications/features on some source development here as I make modifications to utilize Ryven. Here are some ideas that may benefit the project.

• Refactoring the codebase to a Python package structure. This is a big one for making the work more accessible to the community, by uploading the project to PyPi. Then people can just pip install ryven and have it. I forked Ryven master and began making these structure changes (only the main Ryven gui folder working within package so far), before realizing that you had an active dev branch and these would be soon obsolete 🤦‍♂️ . I am happy to help in structuring Ryven to be packaged. To see how this simplifies life, you can install my branch with :

$ pip install git+https://github.com/mmorys/Ryven.git@refactor-to-package

and run the ryven GUI from the command line with:

$ ryven

• There are some cool packages out there for processing function graphs, with built in distributed processing. This could provide an onramp for your multithreading work. Dask provides the ability to create custom graphs. A derivative work that may be more applicable to your use case is Streamz. Check out the PYData Conference talk about it. I had implemented both Dask and Streamz in my crude flow tool to allow asynchronous stream processing through a function graph/flow.

• I love your idea of a function script. It would be awesome if function scripts could be implemented the way PyFlow handles what they call subgraphs, so you can collapse nodes into a script and reopen the in a new tab to view/edit.

• While the NodeManager is a great tool for creating Nodes, GUIs can get tedious to use for creating simple nodes from a function. I began creating a way to generate a Node from a function simply using the decorator to specify inputs/outputs. It would look something like this... For your custom Python function you want to Node-ify, you'd write:

@ryvennode(a=ryven.Integer(), b=ryven.Choices(['x','y','z'], widget=True), outputs='c')
def myfunction(a, b):
    # do something...
    return c

In the Ryven software, we then have an importer, not unlike the current Node importer. The user selects a folder/Python file, and the importer loads it in, and automatically creates a Ryven Package with all the decorated functions it finds. This would likely be limited to functions with simple input/output specifications without special main widgets, but that's probably a majority of use cases.

The ryven.<Class>() stuff would be a way of specifying the type of port, and how it's displayed. I based this idea off Param, a package that does just this and is used by Panel for creating GUI dashboards declaratively. You could potentially even use Param instead of making your own type definitions.

Sorry for the wall of text. I look forward to helping out where I can.

-Marcin

[mmorys]

• One other suggestion I have is to replace imports from pyside2 directly, to imports from qtpy. PySide2 is one of several Python bindings for QT, the two widely used ones being PySide2 and PtQt5. qtpy is simply an interface that allows a user to easily switch between the two, simply by setting an environment variable.
  For Ryven, import statements would change from from PySide2.QtCore import Qt to from qtpy.QtCore import Qt. In the main run script, you can then have a line os.environ['QT_API'] = 'pyside2', and add qtpy to your package requirements.
  The benefit here is that if a user needs to use PyQt5 for licensing or other reasons, they can do so easily, while you can still use PySide2 with the same code.

[leon-thomm]

[there's an even bigger wall incoming] Hey Marcin, thank you so much for your interest. It's really the best part of this, to see other people bring new ideas for making this even more versatile. Now to your points:

Package Structure

Yes, the framework is going to be like that, Ryven itself will be quite small after the framework has its own public repo and package, but I see that making a PyPi package for ryven itself as well wouldn't hurt, so I'll do that, thanks.

Multithreading

The multithreading part so far only regards internal communication between frontend and backend to increase performance. I think what you are suggesting is a parallelization system according to the actual flow. I have thought about that and will do some tests with the framework at some point, but this would rather be something that's not part of the Ryven editor but a specialized one made with the framework. So, thanks for the suggestions, I'll definitely take a look at that.

Functions/Subgraphs

In ryvencore you can (re)use functions everywhere, a FunctionScript technically is just a Script with some extra stuff, so it has its own flow and in Ryven it has its own tab with its own logs and variables, just like the other scripts as well. This keeps it simple and still enables infinite abstraction (theoretically).

NodeManager and Function Parsing

I'm personally very reluctant to implement such auto parse mechanisms. I see that this speeds up the process and it totally makes sense to think about it, but I believe that you will find yourself being less productive than with source code if you just use the diagram structure and not the possibility of adding to the UI. It's really the UI that does it for me, so for any approach to auto parse functions into nodes, I strongly suggest at the very least enabling easily adding GUI, including custom widgets. From ryvencore release onwards, all information about a Node will be included in the code like this:

class NOTGateNode(LogicGateNodeBase):
    title = 'NOT'
    description_html = [...]
    init_inputs = [
        rc.NodeInputBP('data'),
    ]
    init_outputs = [
        rc.NodeOutputBP('data'),
    ]
    icon = './assets/NOT2.svg'

    def update_event(self, input_called=-1):
        self.set_output_val(0, int(not self.input(0)))

    def gen_formula(self, out_index=-1):
        return f'|NOT|({self.input_formula(0)})'

    [...]

If you would manage to make a decorator for data nodes that generates the working classes, so the user can then go ahead and enhance those if needed, this could be a really nice addition to the framework actually!

In ryven(core) there are no builtin type definitions. You can pass any object beween nodes you like (of course you can program them in a way that they only exchange abstracted and structured data). This is much simpler and more scalable than introducing a type system, and because you have stuff like value preview on mouse hover (improved in the framework), I don't see a need to add it, I am really focused on simplicity there. Or what exactly are you imagining?

Using QtPy as the wrapper was already on my list, so I'll change this.

Thanks again for all the input!

[mmorys]

Thanks for the quick reply. Your first three points make sense. The threading issue is a tricky one with more than one solution, for sure.

As far as the function parsing, I think your class definition method makes sense, and would accomplish the same end as what I was referring to. Essentially, I envision a situation where the user can define a minimal specification of node inputs and outputs in text format, from which a node is created, whether through a class definition or decorator that creates the class.

So this is my understanding of what you have going, and how a user may use it. If I have a Python package with say 2 functions I want to use in ryven I could now do this? (changing the inputs/outputs of course):

from my_package import fn1, fn2

class Fn1Node(SomeNodeBase):
    title = 'fn1'
    description_html = [...]
    init_inputs = [
        rc.NodeInputBP('data'),
    ]
    init_outputs = [
        rc.NodeOutputBP('data'),
    ]
    def update_event(self, input_called=-1):
        self.set_output_val(0, fn1(self.input(0)))

    [...]

class Fn2Node(SomeNodeBase):
    title = 'fn2'
    description_html = [...]
    init_inputs = [
        rc.NodeInputBP('data'),
    ]
    init_outputs = [
        rc.NodeOutputBP('data'),
    ]
    def update_event(self, input_called=-1):
        self.set_output_val(0, fn2(self.input(0)))

    [...]

If that is the case, it accomplishes the same goal I had in mind. Unless I missed something in Ryven, before in order to create a Node you went through the NodeManager GUI, which is very helpful for initially creating Nodes, but once you get to wrapping dozens of functions it's a lot of clicking. It would be easier to copy and paste such a node template as you presented here and edit the fields. If this template file is then read in as a Node Package, then that's basically exactly what I was looking for. Implementing it as a decorator to generate the class isn't a big step from there, but also wouldn't necessarily make life that much easier, just change the syntax.

To your last point, I don't propose a typing system necessarily, more of an introspection system for automating widget creation. If the user can specify in the node class that input 0 is a float in the range 0 to 1 (In Param this would be param.Number(0.5, bounds=(0, 1))), then the Node could generate an appropriate slider widget at the input for the user to control.
This would be especially useful for cases where a function has a strict set of strings as inputs. If the list of acceptable options for input 0 of my function is known to be ['auto', 'enabled', 'disabled'], than the class definition for the input should take in something like param.ObjectSelector(default="auto", objects=["auto", "enabled", "disabled"]) which would automatically create the appropriate dropdown selection widget on the node with the objects, instead of making the user look up the function documentation to find what should go in a text box. The implementation of Panel/Param is a good example of fundamental input types/widgets. You don't need to patrol the user's connections to enforce typing, but many functions in the data processing realm consist of one input for the data, and many inputs for primitive parameters that would be most easily handled by widgets.

I'm looking forward to seeing what Ryven-core really is, because I don't think I fully understand what it is and how it relates to what you currently have in Ryven.

Thanks!

[leon-thomm]

Ah yes, I see now. Providing a set of fixed classes, maybe using Param, for declaratively creating the widgets totally makes sense. This could really get much easier by following an already existing system like Param and just adding some basic Qt widgets for the options. I will think about how this could be integrated.

To not further increase confusion: the node packages system is part of Ryven only, not part of ryvencore. In the framework (ryvencore or ryvencore-qt or whatever exact name it will have), where almost everything will be happening, nodes get registered in a Session object. The Session basically represents a project. So, say you want to build your own little editor at some point, you are then totally free in setting up any structure of your nodes' sources, at the end of the day you just need to call my_session.register_nodes([FirstNode, SecondNode, ThirdNode, ...]) and then they can be used in flows. This is also what Ryven will do internally after importing the node package. So, your code snippet shows exactly how you would define the nodes for your functions you want to add. All this will also be described in detail in the documentation for ryvencore which I've been working on for quite a while now.

And I will also think about ways to enable including nodes directly from sources for Ryven, so you don't have to use the NodeManager anymore, as from now on all node information will be part of its class. Due to this fact, I will probably change it so that the package translator generates the full node class code, so that technically the actual package file doesn't contain any information about the individual nodes anymore. It will probably just store a list of the names of the nodes included in the package so Ryven can then automatically collect all those from the directory. But as I mentioned, all that Ryven stuff might take a little while, as I am currently mainly focused on getting the framework right.

Cheers

[leon-thomm]

I just made a few tests, and it really seems quite easy to automatically generate the classes now. I explained my worries about this feature, but there's no doubt that it could be quite useful for getting started and for built-in python libraries. I wrote a small script that inspects a built-in module, extracts global routines and creates the nodes (files). I tried this with Python's math module by creating a math folder and running python autogen.py 'math' '#88aadd' 'math', and then I imported them in Ryven

This could be nicely automized. I had only a few problems with reading the functions. My generated nodes exclude a few whose signatures couldn't be parsed by inspect. I can't fix that right now, but it should be possible.

[Ahleroy]

Hi,
Macros were removed in Ryven 3.1.0. My understanding is that they were the recommended approach to dealing with subgraphs/groups/modules. In the release note of the latest version, it is explained that a node approach could lead to the same result. It is not completely clear however how one can achieve such a system.
Would it be possible to provide an example?