Files from file_cache are persisting

[luiztauffer]

The files stored at file_cache are persisting on apptainer mode:

[magland]

I added the file cache in order to enable jobs to use the outputs of previous jobs without needing to re-download. It also enables pipelines where the intermediate files are not uploaded to the cloud.

I suppose we should make some configuration option to either not use this cache, or to automatically clear old files.

[luiztauffer]

yeah that makes sense, probably at the job config level, options like these?

• use cache from previous jobs
• clean cache from this job