Update configuration system design

[euronion]

Closes #292 .

Changes proposed in this Pull Request

New configuration system:

• Allow overwrites in config.yaml of config.default.yaml (instead of fully substituting the configuration file.
• Move configs to own subfolder config/

TODOs

• Check if migration works. Maybe @FabianHofmann can you help with that?
• Check if references in documentation have been missed

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
• Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

[euronion]

What do you think? @FabianHofmann @fneum @martacki @lisazeyen @koen-vg ?

[euronion]

I'm trying to get the CI running. Turns out --configfile does not overwrite the configfile: ... specified in the Snakefile, but merges both of them. :/

[koen-vg]

I think this looks good! The implementation looks sensible. There seems to still be some default configuration in the renewable: section (apart from the cutout name) in config.tutorial.yaml? I would try if it works to only include:

renewable:
  onwind:
    cutout: europe-2013-era5-tutorial
  offwind-ac:
    cutout: europe-2013-era5-tutorial
  offwind-dc:
    cutout: europe-2013-era5-tutorial
  solar:
    cutout: europe-2013-era5-tutorial

And on that note, maybe we can also take this opportunity to update config.tutorial.yaml to use

atlite:
  cutouts:
    # Automatically determine geographical bounds and time span.
    base:
      module: era5

That should be the preferred way nowadays, instead of messing around with coordinates, right?

[euronion]

@FabianHofmann CI is failing due to an atlite error. Can you easily pinpoint it?

 [Wed Dec  8 16:15:06 2021]
rule build_renewable_profiles:
    input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_offshore.geojson, cutouts/europe-2013-era5-tutorial.nc
    output: resources/profile_offwind-ac.nc
    log: logs/build_renewable_profile_offwind-ac.log
    jobid: 14
    benchmark: benchmarks/build_renewable_profiles_offwind-ac
    wildcards: technology=offwind-ac
    threads: 2
    resources: tmpdir=C:\Users\RUNNER~1\AppData\Local\Temp, mem=20000

INFO:__main__:correction_factor is set as 0.8855
Traceback (most recent call last):
  File "D:\a\pypsa-eur\pypsa-eur\.snakemake\scripts\tmp5p25ngee.build_renewable_profiles.py", line 265, in <module>
    availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
  File "C:\Miniconda\envs\test\lib\site-packages\atlite\gis.py", line 555, in compute_availabilitymatrix
    assert (
AssertionError: For parallelization all raster files in excluder must be closed
[Wed Dec  8 16:15:11 2021]
Error in rule build_renewable_profiles:
    jobid: 14
    output: resources/profile_offwind-ac.nc
    log: logs/build_renewable_profile_offwind-ac.log (check log file(s) for error message)

Traceback (most recent call last):
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\executors\__init__.py", line 593, in _callback
    raise ex
  File "C:\Miniconda\envs\test\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\executors\__init__.py", line 579, in cached_or_run
    run_func(*args)
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\executors\__init__.py", line 2461, in run_wrapper
    raise ex
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\executors\__init__.py", line 2358, in run_wrapper
    run(
  File "D:\a\pypsa-eur\pypsa-eur\Snakefile", line 527, in __rule_build_renewable_profiles
    ll=ll,
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\script.py", line 1369, in script
    executor.evaluate()
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\script.py", line 381, in evaluate
    self.execute_script(fd.name, edit=edit)
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\script.py", line 580, in execute_script
    self._execute_cmd(
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\script.py", line 412, in _execute_cmd
    return shell(
  File "C:\Miniconda\envs\test\lib\site-packages\snakemake\shell.py", line 287, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'C:/Miniconda/envs/test/python.exe "D:\a\pypsa-eur\pypsa-eur\.snakemake\scripts\tmp5p25ngee.build_renewable_profiles.py"' returned non-zero exit status 1.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: D:\a\pypsa-eur\pypsa-eur\.snakemake\log\2021-12-08T155358.836795.snakemake.log
Error: Process completed with exit code 1.

[FabianHofmann]

yes, i will have a look

[euronion]

Benching this PR for now. Might be of interest to re-open later. Right now no concrete need or benefit seen.