Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make WDL notice when a file can no longer be used and delete it #4872

Open
unito-bot opened this issue Apr 17, 2024 · 0 comments
Open

Make WDL notice when a file can no longer be used and delete it #4872

unito-bot opened this issue Apr 17, 2024 · 0 comments
Labels

Comments

@unito-bot
Copy link

It would be good if files in WDL workflows were deleted from the job store when they can no longer be accessed by any remaining jobs.

We could add nodes to the workflow graph to delete File values, and make them depend on the completion of all the WDL nodes that actually use those File values, and when they run they delete the files.

I think MiniWDL’s analysis tracks what we would need for this, but we might also want to think about passing sub-environments around the top-level workflow graph that only have the values that actually are referenced, so we don’t feed all the File variables to a bunch of nodes that don’t reference them.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1540

@unito-bot unito-bot added the wdl label Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant