Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Pipeline serialization #198

Open
eddiebergman opened this issue Dec 9, 2023 · 0 comments
Open

[Feature] Pipeline serialization #198

eddiebergman opened this issue Dec 9, 2023 · 0 comments
Labels
feature A new feature

Comments

@eddiebergman
Copy link
Contributor

It could be useful to be able to export or import pipelines from a static file definition like .yaml for the sake of running larger experiments. One such example is running things with Hydra or when you have to run many experiments parallelized over a cluster.


This could be fairly straightforward with most pipeline setups that rely on libraries for their components and involve simple config and spaces.

Some clear problems with this of course are functional elements of a pipeline, for example the config_transform, which typically invoke user based transforms during the configuration of a pipeline, or custom components.

These problems could be possible to solve as hydra seems to do some import magic to make it happen, at the cost of less transparency. This also means exported pipelines will not be transferable from one environment to the next.

@eddiebergman eddiebergman added the feature A new feature label Dec 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A new feature
Projects
None yet
Development

No branches or pull requests

1 participant