Integrate calibration pipeline into atmos #2895
Conversation
nefrathenrici
force-pushed
the
ne/calibrate
branch
from
fb79171
to
dbee822
Compare
nefrathenrici
force-pushed
the
ne/calibrate
branch
from
d8a5a2d
to
8154925
Compare
nefrathenrici
force-pushed
the
ne/calibrate
branch
from
a2f4074
to
4f8a838
Compare
nefrathenrici
force-pushed
the
ne/calibrate
branch
from
e2fc12b
to
16c9a61
Compare
nefrathenrici
force-pushed
the
ne/calibrate
branch
from
16c9a61
to
9ea6670
Compare
| file_path = joinpath(member_path, "parameters.toml") | ||
| mkpath(dirname(file_path)) | ||
| touch(file_path) | ||
|
|
||
| atmos_config = ClimaCalibrate.set_up_forward_model(1, 1, experiment_dir) | ||
| (; parsed_args) = atmos_config | ||
|
|
||
| @testset "Atmos Configuration" begin | ||
| @test parsed_args["moist"] == "equil" | ||
| @test parsed_args["toml"] == [file_path] | ||
| @test parsed_args["output_dir"] == member_path | ||
| @test parsed_args["restart_file"] == | ||
| "/groups/esm/ClimaArtifacts/artifacts/atmos_held_suarez_obs/day200.0.hdf5" | ||
| end | ||
|
|
||
| rm(file_path) |
There was a problem hiding this comment.
Use temporary folders instead of relying on creating a new folder and removing it. The folder will not be cleaned in case of failing tests.
| @test parsed_args["toml"] == [file_path] | ||
| @test parsed_args["output_dir"] == member_path | ||
| @test parsed_args["restart_file"] == | ||
| "/groups/esm/ClimaArtifacts/artifacts/atmos_held_suarez_obs/day200.0.hdf5" |
There was a problem hiding this comment.
You should use Artifacts or ClimaArtifacts instead of hardcoding the path.
This test has to be reproducible even outside of the Caltech cluster
|
I am not too familiar with the calibration pipeline, so it will take me a little bit of time to understand and properly review this PR. |
| YAML = "ddb6d928-2868-570f-bddf-ab3f9cf99eb6" | ||
|
|
||
| [compat] | ||
| ClimaAtmos = "=0.24.0" |
There was a problem hiding this comment.
If ClimaAtmos is pinned to a specific version, why do we need to have a job in buildkite?
| Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f" | ||
| EnsembleKalmanProcesses = "aa8a2aa5-91d8-4396-bcef-d4f2ec43552d" | ||
| JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819" | ||
| NetCDF = "30363a11-5582-574a-97bb-aa9a979735b9" |
| try | ||
| G_ensemble[:, m] .= process_member_data(simdir) | ||
| catch err | ||
| @info "Error during observation map for ensemble member $m" err | ||
| G_ensemble[:, m] .= NaN | ||
| end |
There was a problem hiding this comment.
What are you trying to try-catch here?
| const config = ExperimentConfig(@__DIR__) | ||
| function observation_map(iteration) | ||
| (; ensemble_size, output_dir) = config | ||
| dims = 1 |
There was a problem hiding this comment.
Maybe dims can have a more descrpitive name. dims of what?
| ensemble_error += abs(i - theta_star)^2 | ||
| ensemble_spread += abs(i - ensemble_mean)^2 | ||
|
|
||
| end |
There was a problem hiding this comment.
| ensemble_error += abs(i - theta_star)^2 | |
| ensemble_spread += abs(i - ensemble_mean)^2 | |
| end | |
| ensemble_error += abs(i - theta_star)^2 | |
| ensemble_spread += abs(i - ensemble_mean)^2 | |
| end |
| ensemble_error = 0 | ||
| ensemble_spread = 0 |
There was a problem hiding this comment.
| ensemble_error = 0 | |
| ensemble_spread = 0 | |
| ensemble_error = 0.0 | |
| ensemble_spread = 0.0 |
| set_up_forward_model(member, iteration, experiment_dir::AbstractString) | ||
|
|
||
| Returns an AtmosConfig object for the given member and iteration. | ||
| If given an experiment id string, it will load the config from the corresponding YAML file. |
| if haskey(config_dict, "restart_file") && | ||
| !isabspath(config_dict["restart_file"]) | ||
| config_dict["restart_file"] = | ||
| joinpath(experiment_dir, config_dict["restart_file"]) | ||
| end |
There was a problem hiding this comment.
Shouldn't this be handled internally by Atmos?
|
What is the ultimate intent of this PR? Is it to test that things work? To teach people how to do calibration? Other reasons? |
This PR adds the framework for reproducible calibration experiments, starting with one example experiment.
calibration/README.mdprovides an overview. Most of the actual code is in ClimaCalibrateContent
calibration/model_interface.jl: Model interface hooks for CalibrateAtmos. This will be used in every atmos calibration.test/calibration_interface.jl: Basic tests for the model interface file.calibration/experiments: Folder containing subfolders of calibration experiments. Currently only containssphere_held_suarez_rhoe_equilmoistsphere_held_suarez_rhoe_equilmoist/
experiment_config.yml: Stores filepaths for calibration artifacts and other configuration datamodel_config.yml: Atmos yaml config for the calibrationpipeline.sbatch: Script to run calibration experiment on centralobservation_map.jl: Code to process model output into observation spacepostprocessing.jl: Plotting scriptprior.toml: Prior distribution file