Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In-chain overlays #240

Open
martin-traverse opened this issue Dec 15, 2022 · 0 comments
Open

In-chain overlays #240

martin-traverse opened this issue Dec 15, 2022 · 0 comments
Milestone

Comments

@martin-traverse
Copy link
Contributor

This allows flows to be overlaid without creating a new flow definition or splitting the flow into separate jobs.

Overlays are attached to flow sockets as a list, multiple overlays can be applied to each socket and are executed in order.

Overlays can be applied to any socket, including flow inputs and outputs as well as input and output sockets for models. Overlays applied to a flow input or model output affect all consumers of that dataset. Flows applied to a model input or flow output only affect a single dataset.

Implemented by running a graph transform that expands the original graph by inserting the overlays, then executing the expanded graph in the regular executor. Implementation effort is surprisingly small! Overlay nodes are tagged to provide more meaningful logging and error reporting.

Originally done at the platform level in the orchestrator, but implementing in the runtime would make overlays available during model build and via the TRAC shell.

@martin-traverse martin-traverse added this to the 0.8 milestone Dec 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant