Post-merge snakemake.params

[virio-andreyana]

This is a re-do for the pull request in #296.

Changes proposed in this Pull Request

Use the params: section in rule definition to keep track of changed settings in config.yaml.

https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules

The goal is that rules for which parameters have changed rerun automatically.

This can currently be done with:

https://snakemake.readthedocs.io/en/stable/project_info/faq.html#how-do-i-trigger-re-runs-for-rules-with-updated-code-or-parameters

snakemake -n -R `snakemake --list-params-changes`

or

snakemake -R $(snakemake --list-params-changes)

This will be even more convenient, once this snakemake PR is released:

snakemake/snakemake#1107

snakemake --rerun-params-changed

or

snakemake --rerun-changed

(which would also include code changes)

It seems to work very nicely to pass lists/dicts as params, which would help us keep config "blocks".

This PR is not complete yet, but a proof-of-concept for a selection of config settings. We can discuss from here and complete later.

Rules Checklist

• Snakefile
• build_electricity.smk
• build_sector.smk
• collect.smk
• common.smk
• postprocess.smk
• retrieve.smk
• solve_electricity.smk
• solve_myopic.smk
• solve overnight.smk

Script Checklist

• _helpers.py
• add_brownfield.py
• add_electricity.py
• add_existing_baseyear.py
• add_extra_components.py
• base_network.py
• build_ammonia_production.py
• build_biomass_potentials.py
• build_biomass_transport_costs.py
• build_bus_regions.py
• build_clustered_population_layouts.py
• build_cop_profiles.py
• build_cutout.py
• build_electricity_demand.py
• build_energy_totals.py
• build_gas_input_locations.py
• build_gas_network.py
• build_heat_demand.py
• build_hydro_profile.py
• build_industrial_distribution_key.py
• build_industrial_energy_demand_per_country_today.py
• build_industrial_energy_demand_per_node_today.py
• build_industrial_energy_demand_per_node.py
• build_industrial_production_per_country_tomorrow.py
• build_industrial_production_per_country.py
• build_industrial_production_per_node.py
• build_industry_sector_ratios.py
• build_natura_raster.py
• build_population_layouts.py
• build_population_weighted_energy_totals.py
• build_powerplants.py
• build_renewable_profiles.py
• build_retro_cost.py
• build_salt_cavern_potentials.py
• build_sequestration_potentials.py
• build_shapes.py
• build_ship_raster.py
• build_shipping_demand.py
• build_solar_thermal_profiles.py
• build_temperature_profiles.py
• build_transport_demand.py
• cluster_gas_network.py
• cluster_network.py
• copy_config.py
• make_summary.py
• plot_network.py
• plot_summary.py
• prepare_links_p_nom.py
• prepare_network.py
• prepare_sector_network.py
• retrieve_databundle.py
• retrieve_gas_infrastructure_data.py
• retrieve_sector_databundle.py
• simplify_network.py
• solve_network.py
• solve_operations_network.py

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in all of config.default.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv.
• A release note doc/release_notes.rst is added.

[virio-andreyana]

The majority of the param conversion is done by a python algorithm. Some changes are simple, only changing the word "config" into "param". The complicated part is in simplify_network.py and solve_network.py. While the simplify_network.py has been amended, solve_network.py is the hardest. It ask for almost every variable variable there is in config.

I believe its better to just add in param config=config since any changes that happens in config require us to rerun the solve_network.py anyway. But what do you think?

[FabianHofmann]

Hey @virio-andreyana, I would not pass the whole config but use the sections which are really needed (even if it is a lot). Imagine you just want to alter some colours for the visualization of results (which happens after the solving), then the whole solving process would be retriggered.