PFT/landuse tools: availability, documentation, training

[slevis-lmwg]

TODOs as of Feb 2024.

• @lawrencepj1 updates/completes the documentation, currently in prep: This is for clm5 and is needed by end of 2024 for clm52, as well.
• Reformat documentation from .doc to .rst (@mvdebolskiy , June-July)
• Peter shows a few of us how the tools work in a half-day training: Individual tools, as well as complete workflows.
• Update tools to the newest version.
• @ekluzek and @lawrencepj1 move tools to ESCOMP
• Make tools part of ctsm/tools and use the infrastructure of cime/ctsmtools to make them work on any machine
• Run mksurfdata_esmf with the new vegetation raw data Run mksurfdata_esmf with the new vegetation raw data #2452

Upcoming meeting in Mar 2024.

@lawrencepj1 @mvdebolskiy @ekluzek

[slevis-lmwg]

@lawrencepj1 shared with us by email the latest pdf of his documentation. I'm open to uploading here if you all think it makes sense.

[lawrencepj1]

Thanks @slevis-lmwg yes please upload and make available as a work in progress

[slevis-lmwg]

CLM50_Land_Data_Tech_Note_v20.pdf

[slevis-lmwg]

Meeting 2023/11/16 @lawrencepj1 @ekluzek @mvdebolskiy @slevis-lmwg

Peter points out:

• There are 3 overall steps in the big picture of preparing CTSM raw datasets.
• Step 1 is most volatile in its process because datasets change in formats, projections, grids, etc.

So Peter proposes standardizing steps 2 and 3 initially.

Peter is organizing step 2 on the cgd machines here:
/project/tss/lawrence/ctsm52landusedataprep/ctsm52landusedata025
This directory has a /tools subdirectory and a README that Peter is in the middle of completing.

Peter is organizing step 3 on cheyenne and needs to move stuff to /campaign before cheyenne goes away.
Peter gave us a quick tour of these directories:

/glade/p/cesm/sdwg_dev/thesis/collections/cesm_tools/ctsm52landusedatatool
/glade/p/cesm/sdwg_dev/thesis/data/cesm_tools/lumipfinallcc/historicLUH2_181217
/glade/p/cesm/sdwg_dev/thesis/data/cesm_tools/surfacedata/ctsm52finalrawdata

This is a sample namelist used by the C-program (each directory here has a diff. namelist):
/glade/p/cesm/sdwg_dev/thesis/data/cesm_tools/surfacedata/ctsm52finalrawdata/globalctsm52histLUH2deg025_220305/globalctsm52histLUH2namelist.txt

For /glade/p data we decided (Erik, Peter, and Sam L will meet Dec 6 to do the following):

• @lawrencepj1 will move output (i.e. ctsm rawdata) to /glade/campaign/cesm/cesmdata/cseg/inputdata
• @lawrencepj1 will move inputs to /glade/campaign/cesm/development/lmwg

[mvdebolskiy]

@lawrencepj1 can you add the path to where READMEs and shell-scripts are for step 2?

[slevis-lmwg]

@mvdebolskiy I think you mean the path that I listed under step 2 above. The README is there and the scripts are in /tools. Do you mean something else?

[mvdebolskiy]

Yeah, step two. I thought it was also on thesis. nvm

[mvdebolskiy]

Updates:
Peter finished building all the pre-proc tools on derecho.
Potential vegetation #2320 added.
New directory: /glade/campaign/cesm/development/lmwg/ with raw and surface data for CTSM5.2. They include namelists for ctsm52landusetool
Ctsm5.2 release will include only re-processed old rawdata (with new variables etc).
List of datasets for CESM3
Project board for CTSM5.2 datasets

[slevis-lmwg]

@mvdebolskiy thank you for tracking today's conversation with these notes.

[lawrencepj1]

README from Land Use Data Prep

CTSM52 Land Use Data Prep Workflow

This is step two in the bigger picture of generating CTSM raw PFT/CFT data.
The prior stage is to take the 1km MODIS, AVHRR, ICESAT Satellite and EarthSTAT
cropping products processed into a consistent format including grids and land masks.

1. Generate Base LUH2 surface datasets

   Use tools/runcreateluh2statesfile LUH2SeriesName startyear endyear
   Use tools/runcreateluh2transitionsfile LUH2SeriesName startyear endyear
   Use tools/runcreateluh2managementfile LUH2SeriesName startyear endyear

2. Generate CTSM52 MODIS EarthStat and Current raw and surface data files

   Use tools/runcreatemodisearthstatsurffile CTSM52_MODISEarthStat 2005
   Use tools/runcreateluh2modiscurrentallrawfiles LUH2_HIST_GEOG_DEG025 CTSM52_MODISEarthStat CTSM52_MODISCurrent 2005
   Use tools/runcreatemodiscurrentsurffile CTSM52_MODISCurrent 2005

3. Generate CRU Climate surface data

   Use tools/runcreateclimatesurffile CRUSeriesName

4. Generate CRU CTSM LUH2 climate vegetation cover lookup files

   Use tools/runcreateluh2modiscruclimatevegetationlutfiles
   Must have CTSM52LUH2.Deg025.Hist.001 processed for year 2005

5. Generate CTSM52 MODIS EarthStat and Initial Potential Vegetation Continuous Fields (VCF) in workrawdata

   Use tools/runcreateluh2modiscurrentvcfrawfiles CTSM52_MODISCurrent 2005
   Use tools/runcreateluh2potvegforestinitialvcfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGInit 2005
   Use tools/runcreateluh2potvegnonforestinitialvcfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGInit 2005
   Use tools/runcreateluh2potvegrangeinitialvcfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGInit 2005
   Check Initial Vegetation Continuous Fields (VCF) netcdf files in worksurfdata
   Use tools/runcreatevcfsurffile CTSM52_POTVEGInit LUH2Component_VCF 2005

6. Generate Combined Vegetation Continuous Fields (VCF) for Climate Potential Vegetation in workrawdata

   Use tools/runcreateluh2potvegallcombinedvcfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGInit CTSM52_POTVEGComb 2005
   Check Vegetation Continuous Fields (VCF) netcdf files in worksurfdata
   Use tools/runcreatevcfsurffile CTSM52SeriesName LUH2Component_VCF year

7. Generate Current Day Scaled Vegetation Continuous Fields (VCF) for Climate Potential Vegetation in workrawdata

   Use tools/runcreateluh2potvegallscaledvcfrawfiles LUH2_HIST_GEOG_DEG025 CTSM52_MODISCurrent CTSM52_POTVEGComb CTSM52_POTVEGScaled 2005
   Check Vegetation Continuous Fields (VCF) netcdf files in worksurfdata
   Use tools/runcreatevcfsurffile CTSM52SeriesName LUH2Component_VCF year

8. Generate Maximum Vegetation Continuous Fields (VCF) for Climate Potential Vegetation in workrawdata. Potential Vegetation is updated where current day MODIS EarthStat vegetation is higher.

   Use tools/runcreateluh2potvegallmaxvcfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGComb CTSM52_POTVEGScaled CTSM52_POTVEGMax 2005
   Check Vegetation Continuous Fields (VCF) netcdf files in worksurfdata
   Use tools/runcreatevcfsurffile CTSM52SeriesName LUH2Component_VCF year

9. Generate All of the raw Merge files for primf secdf primn secdn range for generating the timeseries Descriptor files

   Use tools/runcreateluh2mergeallhistoricalrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005 1700 2015
   Check Merge netcdf files in worksurfdata for LUH2Type (primf,secdf,primn,secdn,range) FractionType (TREEFRAC,HERBFRAC,BAREFRAC)
   Use tools/runcreatemergesurffile CTSM52SeriesCurrentName CTSM52MergeSeriesName LUH2Group VCFGroup ReferenceYear StartYear EndYear

10. Generate All the VCF Tree, Herb and Grass PFT extrapolation files

Use tools/runcreateluh2modisearthstatextraptreerawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax 2005
Use tools/runcreateluh2modisearthstatextrapherbrawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax 2005
Use tools/runcreateluh2modisearthstatextrapgrassrawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax 2005
Check Extrapolation netcdf files in worksurfdata for PFT composition (TREE_EXTRAP,HERB_EXTRAP,GRASS_EXTRAP)
Use tools/runcreateworksurffile CTSM52_MODISEarthStat GRASS_EXTRAP 2005

11. Generate All of the raw LUH2 component PFT Descriptor files for primf secdf primn secdn range pastr for a timeseries

Generate the Descriptor files in rawdata:
Use tools/runcreateluh2descprimfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax CTSM52_POTVEGScaled CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreateluh2descsecdfrawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax CTSM52_POTVEGScaled CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreateluh2descprimnrawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax CTSM52_POTVEGScaled CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreateluh2descsecdnrawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax CTSM52_POTVEGScaled CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreateluh2descrangerawfiles CTSM52_MODISCurrent CTSM52_POTVEGMax CTSM52_POTVEGScaled CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreateluh2descpastrrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005

Generate the Descriptor files in surfdata:
Use tools/runcreateprimfdescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreatesecdfdescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreateprimndescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreatesecdndescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreaterangedescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1700 2015
Use tools/runcreatepastrdescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 2005 2005

12. Generate All of the raw LUH2 component CFT Descriptor files for c3ann c4ann c3per c4per c3nfx for a timeseries

Generate the Descriptor files in rawdata:
Use tools/runcreateluh2descc3annrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreateluh2descc4annrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreateluh2descc3perrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreateluh2descc4perrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreateluh2descc3nfxrawfiles CTSM52_MODISCurrent CTSM52_LUH2Hist 2005 1961 2015

Generate the Descriptor files in surfdata:
Use tools/runcreatec3anndescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreatec4anndescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreatec3perdescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreatec4perdescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1961 2015
Use tools/runcreatec3nfxdescsurffile CTSM52_MODISEarthStat CTSM52_LUH2Hist 2005 1961 2015

13. Generate All the raw LAI and Soil Color files for a timeseries these need to already exist as raw data for the timeseries

In surfdata:
Use tools/runcreatelaimksurffile CTSM52_LUH2Hist
Use tools/runcreatesoilcolormksurffile CTSM52_LUH2Hist

[mvdebolskiy]

Moving tools from Peter to ESCOMP:

1. Make it as a CTSM external for ctsm-tools
2. Leverage CIME to set up configuration and compile options
3. Tools will not be in the CESM3 release
4. Tech note is more realistic for tech note ( Ask Peter for the .doc before June to convert into .rst

Peter will present on CESM version for CCIS with datasets created with the previous version of mksurf tools.

/glade/campaign/cesm/development/lmwg/ is now readable for others

Newer versions of landuse tools are not on github. (for scenarios).

[slevis-lmwg]

@mvdebolskiy offered to make a list of TODO checkboxes here in the discussion. Possibly at the next meeting we can decide to start a project board.

[mvdebolskiy]

Notes from 17th May 2024:

• Need to run ssp245 to test with anomaly forcing? (blocked by SSP cases get rcp45 anomalies #2301)
• For SE testing: multiple resolutions. (primarily ne30)
• For scientific testing: f09 historical and spinup.
• @slevis-lmwg has started testing new pft datasets Run mksurfdata_esmf with the new vegetation raw data #2452
• Baselines with 5.2 datasets to compare with new pft datasets 5.3
• Run diagnostics on the scientific runs.