Trying snakemake ?

[Nitnelav]

So how deep into trying snakemake are you ?

[sebhoerl]

Let's discuss this in a couple of weeks during the workshop :) I always played a bit around with it, and now I'm applying it properly in another project to see how it works.

The biggest disadvantage is that everything is handled via files that are created by one step and read by the next, so we lose some efficiency from holding things in memory. But on the other hand, snakemake has a lot of built-in functionality to submit pipelines to clusters, clouds, etc.

The pipeline definition might be a bit more complex, because everything is managed in a configuration file and dependencies are rather handled through file names. But on the other hand, I'm not happy with having all dependencies hardcoded currently in the pipeline.

So the big advantage for me would be that we can have a much more modular system. My idea would be to have a repository that acts as a kind of library for implementations and/or wrappers of algorithms for travel demand synthesis in a central spot, and then we have pipeline definitions in snakemake that make use of this code. This would make it also much easier to contribute things and build custom pipelines. Ideally we would then just have a pipeline configuration file for France, one for California, etc. but they make use of the same code blocks.