You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A current shady spot in SciPipe is when needing to generate many sets of parameter values or file names to feed a downstream pipeline. This can be done to some extent using e.g. a globber component, but that is very limited to a very specific use case.
Below is a sketch on an idea on how to implement a type of component that can generate this based on shell scripts.
The idea is that you are supposed to write a shell script that produces a set of JSON-objects, one per line, with the parameter and output filename fields populated.
The API could look like this:
looper:=wf.NewLooper("looper", "for f in data/*.csv; do echo \"{ 'outfile': '{o:outfile:$f}' }\"; done;")
otherProc:=wf.NewProc("other-proc", "some-command -in {i:infile} ... Etc etc")
outerProc.In("infile").From(looper.Out("outfile"))
// ... etc etc ...
The example above is basically just a globber, but the same method could be used for populating parameters as well. I will update the example shortly to illustrate the combined generation of filenames and parameters.
The text was updated successfully, but these errors were encountered:
A current shady spot in SciPipe is when needing to generate many sets of parameter values or file names to feed a downstream pipeline. This can be done to some extent using e.g. a globber component, but that is very limited to a very specific use case.
Below is a sketch on an idea on how to implement a type of component that can generate this based on shell scripts.
The idea is that you are supposed to write a shell script that produces a set of JSON-objects, one per line, with the parameter and output filename fields populated.
The API could look like this:
The example above is basically just a globber, but the same method could be used for populating parameters as well. I will update the example shortly to illustrate the combined generation of filenames and parameters.
The text was updated successfully, but these errors were encountered: