Python system tests fails in a CESM checkout

[ekluzek]

Brief summary of bug

You can't run the python system tests if you are in a CESM or CAM checkout (rather than a CTSM standalone checkout).

General bug information

CTSM version you are using: ctsm5.2.0

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: python sys tests

Details of bug

python system tests fail when run

cd components/clm/python
./run_ctsm_py_tests --sys

Important output or errors that show the problem

./run_ctsm_py_tests --sys
..............E
Stdout:
command is /glade/derecho/scratch/erik/tmp/tmpezv1xh1j/gen_mksurfdata_namelist --start-year 2000 --end-year 2000 --vic                           --res 360x720cru --silent --namelist crop-global-present_360x720cru.namelist
......EEEEEE
Inactive Modules:
  1) hdf5/1.12.2     2) intel/2023.0.0     3) ncarcompilers/1.0.0     4) netcdf/4.9.2

Due to MODULEPATH changes, the following have been reloaded:
  1) conda/latest     2) craype/2.7.20

The following have been reloaded with a version change:
  1) cdo/2.1.1 => cdo/2.3.0     2) ncarenv/23.06 => ncarenv/23.09     3) nco/5.1.4 => nco/5.1.9     4) ncview/2.1.8 => ncview/2.1.9

The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) cesmdev/1.0   2) ncarenv/23.09
Done converting /glade/derecho/scratch/erik/tmp/tmp48z4sg4n/scrip.nc
.
Inactive Modules:
  1) hdf5/1.12.2     2) intel/2023.0.0     3) ncarcompilers/1.0.0     4) netcdf/4.9.2

Due to MODULEPATH changes, the following have been reloaded:
  1) conda/latest     2) craype/2.7.20

The following have been reloaded with a version change:
  1) cdo/2.1.1 => cdo/2.3.0     2) ncarenv/23.06 => ncarenv/23.09     3) nco/5.1.4 => nco/5.1.9     4) ncview/2.1.8 => ncview/2.1.9

The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) cesmdev/1.0   2) ncarenv/23.09
Done converting /glade/derecho/scratch/erik/tmp/tmplc6833t9/scrip.nc
...E
Stdout:
in neonsite adding usermodsdirs
usermodsdirs: ['/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/cime_config/usermods_dirs/NEON/BART']
---- building a base case -------
---- creating a base case -------

======================================================================
ERROR: test_simple_jobscript_multi (test.test_sys_gen_mksurfdata_jobscript_multi.TestSysGenMkSurfJSMulti)
Test that a standard simple namelist works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_jobscript_multi.py", line 76, in test_simple_jobscript_multi
    self.createJS(nodes="4", tasks_per_node="12", scenario="crop-global-present")
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_jobscript_multi.py", line 68, in createJS
    main()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_jobscript_multi.py", line 395, in main
    runfile,
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_jobscript_multi.py", line 137, in write_runscript
    main_nml()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 329, in main
    determine_output_mesh(res, force_model_mesh_file, input_path, rawdata_files, tool_path)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 760, in determine_output_mesh
    tree2 = ET.parse(xml_path)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/tools/mksurfdata_esmf/../../ccs_config/component_grids_nuopc.xml'

Stdout:
command is /glade/derecho/scratch/erik/tmp/tmpezv1xh1j/gen_mksurfdata_namelist --start-year 2000 --end-year 2000 --vic                           --res 360x720cru --silent --namelist crop-global-present_360x720cru.namelist

======================================================================
ERROR: test_hires_namelist (test.test_sys_gen_mksurfdata_namelist.TestSysGenMkSurfNML)
Test that a high resolution namelist works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_namelist.py", line 106, in test_hires_namelist
    main()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 329, in main
    determine_output_mesh(res, force_model_mesh_file, input_path, rawdata_files, tool_path)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 760, in determine_output_mesh
    tree2 = ET.parse(xml_path)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/tools/mksurfdata_esmf/../../ccs_config/component_grids_nuopc.xml'

======================================================================
ERROR: test_potveg_namelist (test.test_sys_gen_mksurfdata_namelist.TestSysGenMkSurfNML)
Test that a potential vegetation namelist works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_namelist.py", line 146, in test_potveg_namelist
    main()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 329, in main
    determine_output_mesh(res, force_model_mesh_file, input_path, rawdata_files, tool_path)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 760, in determine_output_mesh
    tree2 = ET.parse(xml_path)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/tools/mksurfdata_esmf/../../ccs_config/component_grids_nuopc.xml'

======================================================================
ERROR: test_simple_namelist (test.test_sys_gen_mksurfdata_namelist.TestSysGenMkSurfNML)
Test that a standard simple namelist works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_namelist.py", line 62, in test_simple_namelist
    main()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 329, in main
    determine_output_mesh(res, force_model_mesh_file, input_path, rawdata_files, tool_path)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 760, in determine_output_mesh
    tree2 = ET.parse(xml_path)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/tools/mksurfdata_esmf/../../ccs_config/component_grids_nuopc.xml'

======================================================================
ERROR: test_ssp_transient_namelist (test.test_sys_gen_mksurfdata_namelist.TestSysGenMkSurfNML)
Test that a SSP transient namelist works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_namelist.py", line 127, in test_ssp_transient_namelist
    main()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 329, in main
    determine_output_mesh(res, force_model_mesh_file, input_path, rawdata_files, tool_path)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 760, in determine_output_mesh
    tree2 = ET.parse(xml_path)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/tools/mksurfdata_esmf/../../ccs_config/component_grids_nuopc.xml'

======================================================================
ERROR: test_vic_nocrop_inlandwet_glc_namelist (test.test_sys_gen_mksurfdata_namelist.TestSysGenMkSurfNML)
Test a namelist with several options on
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_gen_mksurfdata_namelist.py", line 84, in test_vic_nocrop_inlandwet_glc_namelist
    main()
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 329, in main
    determine_output_mesh(res, force_model_mesh_file, input_path, rawdata_files, tool_path)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/toolchain/gen_mksurfdata_namelist.py", line 760, in determine_output_mesh
    tree2 = ET.parse(xml_path)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/xml/etree/ElementTree.py", line 587, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/tools/mksurfdata_esmf/../../ccs_config/component_grids_nuopc.xml'

======================================================================
ERROR: test.test_sys_lilac_build_ctsm (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test.test_sys_lilac_build_ctsm
Traceback (most recent call last):
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/glade/work/erik/conda-envs/ctsm_pylib/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_lilac_build_ctsm.py", line 18, in <module>
    _CIME_PATH = add_cime_lib_to_path(standalone_only=True)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/path_utils.py", line 92, in add_cime_lib_to_path
    cime_path = path_to_cime(standalone_only=standalone_only)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/path_utils.py", line 53, in path_to_cime
    raise RuntimeError("Cannot find cime within standalone CTSM checkout")
RuntimeError: Cannot find cime within standalone CTSM checkout


======================================================================
ERROR: test_one_site (test.test_sys_run_neon.TestSysRunNeon)
This test specifies a site to run
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/test/test_sys_run_neon.py", line 57, in test_one_site
    main("")
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/site_and_regional/run_neon.py", line 227, in main
    cesmroot, output_root, res, compset, user_mods_dirs, overwrite, setup_only
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/site_and_regional/neon_site.py", line 52, in build_base_case
    case_path = super().build_base_case(cesmroot, output_root, res, compset, user_mods_dirs)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/python/ctsm/site_and_regional/tower_site.py", line 121, in build_base_case
    driver="nuopc",
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/case/case.py", line 2505, in create
    gpu_offload=gpu_offload,
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/case/case.py", line 1308, in configure
    compset_alias, science_support = self._set_compset(compset_name, files)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/case/case.py", line 720, in _set_compset
    self._compsetname, compset_alias, files
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/case/case.py", line 915, in valid_compset
    drv_comp = Component(drv_config_file, "CPL")
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/XML/component.py", line 25, in __init__
    EntryID.__init__(self, infile)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/XML/entry_id.py", line 15, in __init__
    GenericXML.__init__(self, infile, schema, read_only=read_only)
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/XML/generic_xml.py", line 84, in __init__
    "Makes no sense to have empty read-only file: {}".format(infile),
  File "/glade/derecho/scratch/erik/cesm2_3_alpha17d/cime/CIME/utils.py", line 176, in expect
    raise exc_type(msg)
CIME.utils.CIMEError: ERROR: Makes no sense to have empty read-only file: /glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/components/cmeps/cime_config/config_component.xml

Stdout:
in neonsite adding usermodsdirs
usermodsdirs: ['/glade/derecho/scratch/erik/cesm2_3_alpha17d/components/clm/cime_config/usermods_dirs/NEON/BART']
---- building a base case -------
---- creating a base case -------

----------------------------------------------------------------------
Ran 32 tests in 20.015s

FAILED (errors=8)

[ekluzek]

The tests that fail are:

• LILAC
• run_neon
• mksurfdata_esmf tools (being addressed in MKSURFDATAESMF build fails in cesm2_3_alpha17d and cesm2_3_alpha17e testing #2519)

LILAC failing is probably OK. But, we likely want to fix this for NEON?

[ekluzek]

For mksurfdata I addressed this by using path_to_cime rather than path_to_ctsm. Likely this can be done in run_neon and LILAC to do something similar. See

b9e947d