DataStation has three modes: browser, desktop, and server. Browser panel eval happens in-browser. Desktop and server panel eval happen on the desktop or server. Each panel's eval has logic that decides whether to evaluate the panel in-memory or to pass over RPC to the backend (the desktop or server process).
Desktop RPC happens through Node's builtin RPC and Server RPC happens over HTTP. RPC logic from the UI is defined in ui/asyncRPC.ts and also desktop/preload.ts (for some Electron RPC helpers on desktop only).
The results of panel evals are kept only on disk. Only metadata (preview, shape, size, etc.) from panel evals are returned in RPC.
This directory contains the Electron app and code that runs RPC calls. It loads the entire UI bundle and responds to RPC requests from it.
All state in the desktop app is stored in JSON files on disk at the moment. The result of panel evals is stored on disk.
It might be ideal to store state in a database (especially for the server use case) but that complicates the install for all supported languages that would then depend not just on a JSON library but also on a database library. Still, this may be the right way to go at the same time as making language setup easier by providing a button or something to install all missing dependencies.
Evaluation happens by spawning a runner (see ./runner below) subprocess where all the eval handlers are actually run. This allows easy resource cleanup and easy "kill" panel eval support.
./runner/eval.go is the main eval entrypoint but the command line entrypoint is at ./runner/cmd/main.go.
This directory contains the server (Express) app and code that proxies RPC calls over HTTP to "desktop" implementations. Really, the code is just located in the ./desktop/panel/ directory but it is run in/by the server process at the moment.
All state in the server app is stored in PostgreSQL. The result of panel evals is still stored on disk however. So it isn't currently possible to run more than one instance of the server without something like sticky sessions.
This contains the in-browser UI that makes RPC calls to handle project state and panel evals.
This contains the code the client uses to make RPC calls to either the desktop process or the server process.
This is where all logic for each panel is contained. If you wanted to add a new panel type, you could copy an existing panel and register it in ./ui/panels/index.ts. You would also need to declare it in ./shared/state.ts.
This is where all data connectors are defined. If you wanted to add a new connector, you could copy an existing connector and register it in ./ui/connectors/index.ts. You would also need to declare it in ./shared/state.ts.
This contains helper utilities used all over the place.
This is where types for all entities in state are described. Any time you want to add a new panel or connector, etc. you'll need to add it here.
If the change needs to be read by the runner you'll also need to add the change to ./runner/state.go. There is nothing at the moment keeping these two files in sync other than tests that may exist.
This is where all supported languages are defined and where their libraries for interacting with DataStation are defined.
The reason it's shared is because Python and JavaScript run in the browser in browser mode but on a machine in desktop/server mode.
Language wrappers are defined in Jsonnet so that they can include
comments and newlines. That is translated to JSON and embedded in Go
with yarn build-language-definitions. The CI will complain if these
definitions ever go out of date but it may be confusing locally not to
see your changes until you run this and recompile the Go runner.