compressed output

[nsheff]

maybe there's a compress_add() function. then, the clean step will compress these (using some provided function, maybe pigz by default).

if a target is given that is in the compress list, and the compressed file exists, instead of repeating, it will just decompress the file.

running in 'dirty' mode would not compress the files.
or maybe there's a new mode for 'compress' but don't delete.

ok here we go:

--clean and --compress both default to 1.
You can set: --clean 0 to NOT clean. you can set --compress 0 to NOT compress.
You can also set --clean 2 to clean extra; same for compressing.

If the pipeline author would use:

pm.clean_add(file, 1) that puts it at the default clean level. for level 2, it would be pm.clean_add(file, 2). The default is 1 so it's backwards compatible and you can still do clean.add(file).`
The same idioms would be used for compress.

question: what about cleaning a compressed file? like, it's compress level 1, and clean level 2? So you may want it compressed, or maybe deleted?

in that case, maybe clean runs first so we don't compress something we're going to just delete anyway.