Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reorganizing and adding docs #35

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

reorganizing and adding docs #35

wants to merge 11 commits into from

Conversation

gidden
Copy link
Member

@gidden gidden commented Mar 14, 2024

Hey @coroa this PR does a few things

  1. it updates the config.yaml and workflow.py to streamline data locations, specifically requiring paths for history, scenarios (I see now it has proxies and gridding, maybe we want to further reduce this..)
  2. removes config.yaml from git tracking
  3. moves regionmapping information into the config.yaml and start to think about how to support running the workflows for multiple models/scenarios

I did a poor man's version of your path expansion to support configured paths in the region mapping, feel free to rip that apart.

Here's the version of my config I have on disk.

# Setting `base_path` redirects everything else from there
out_path: "../results"
data_path: "../data/rescue"

history_path: "$data_path/historical"
scenario_path: "$data_path/scenarios"
gridding_path: "$data_path/gridding_process_files"
proxy_path: "$data_path/gridding_process_files/proxy_rasters"
postprocess_path: "$data_path/postprocessing"

encoding:
  zlib: true
  complevel: 2

region_mappings:
  REMIND-MAgPIE 3.2.0-4.7.0: 
    path: "$data_path/regionmapping_remindmagpie_h12.csv"
    country_column: "CountryCode"
    region_column: "RegionCode"
    sep: ";"

country_combinations:
  sdn_ssd: ["ssd", "sdn"]
  isr_pse: ["isr", "pse"]
  srb_ksv: ["srb", "srb (kosovo)"]

variable_template: "CEDS+|9+ Sectors|Emissions|{gas}|{sector}"
base_year: 2020

luc_sectors:
  - "Agricultural Waste Burning"
  - "Grassland Burning"
  - "Forest Burning"
  - "Peat Burning"
  - "Agriculture"
  - "Aggregate - Agriculture and LUC"

Everything now lives in a single directory (data_path), which I can host on sharepoint (it's just copied over from our previous folder): ...\RESCUE - Documents\WP 1\data_2024_03_14

A next step would be to set up the gridding proxy and masks to write into these locations.

Let me know what you think!

@gidden gidden marked this pull request as ready for review March 19, 2024 08:01
@gidden gidden requested a review from coroa March 19, 2024 08:01
@gidden
Copy link
Member Author

gidden commented Mar 20, 2024

@coroa 44d0911 implements file skipping. I noticed by the way that we not longer use aneris.grid()?

@coroa
Copy link
Collaborator

coroa commented Mar 21, 2024

I noticed by the way that we not longer use aneris.grid()?

Yes. See also: iiasa/aneris#64 (comment)

@gidden
Copy link
Member Author

gidden commented Mar 21, 2024

Got it, thanks!

@gidden
Copy link
Member Author

gidden commented Mar 21, 2024

This is good to go now, noting existing conflicts..

@coroa coroa changed the base branch from review/round-2 to main March 27, 2024 10:35
@coroa
Copy link
Collaborator

coroa commented Apr 2, 2024

I rebased this on the latest changes on main and then added to fixup commits to stuff i'd have done differently

(for the path resolution for regionmappings and how to check for a single model)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants