cycle4

[hakaseh]

@aekiss After setting and testing master+bgc, I'm thinking to create a branch cycle4 for our upcoming cycle 4 run. let me know if you have a better branch name in mind.

For this, I'd like to create diag_table. can you locate me to diag_table_source.yaml that has all the physical diagnostics for cycle 4? I can add BGC diagnostics to it.

Also do you have any recommendation for the duration of 0.1-deg run for one job? For physics only, it seems walltime is set to 5:00:00 for 3 months? If it's good to keep 3 months at a time, and BGC requires approximately nearly doubling, I can try 10:00:00 for 3 months?

[hakaseh]

I just noticed that there is already a branch for Cycle 4, https://github.com/COSIMA/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4.

Should I pull this and start adding BGC diagnostics and all other bgc settings?

[aekiss]

Yes, https://github.com/COSIMA/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4 is the branch intended for this purpose.

diag_table_source.yaml in that branch has the physical diagnostics from cycle3 so that is a good starting point for you to add BGC diags to.

There's a 5-hour walltime limit with this number of cores https://opus.nci.org.au/display/Help/Queue+Limits
so I think we'll need to use 1mo runs, at least to start with.

[aekiss]

FYI the actual walltime for cycles 2 and 3 is in these tables
https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle2/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf_cycle2.csv
https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle3/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf_cycle3.csv
it's under 3hr on average but gets close to 4hr when we need to reduce the timestep.

[aekiss]

If we have 1mo runs we'll need to set new_file_freq: 1 in the global section of diag_table_source.yaml

[hakaseh]

my test runs with master+bgc took just under 3 hours for 1 month simulation. but cycle4 will take longer because of substantially larger diagnostics.

[aekiss]

Wow, so 3x slower... What timestep did you use?

[hakaseh]

300s, i did not change from the default. should i try increase?

[aekiss]

I guess you were starting from rest? If so, 300s is a good initial choice. 540s will probably only work after ~1.5yr, as it needs time to settle down from the IC. See timesteps in first cycle: https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf.csv

So that makes it more like 1.7x slower if we take timestep into account.
How many cores are you using?

[hakaseh]

yes, from rest, using the same number of cores as master of https://github.com/COSIMA/01deg_jra55_ryf

[hakaseh]

I haven't sorted out the walltime for Cycle 4 yet, but I have created an BGC-enabled branch for Cycle 4 in my fork repo:
https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc
This should have all the changes needed to enable BGC runs. The next step is to set up restart files for this run, which requires manually adding sea-ice BGC tracers to sea-ice restart file, which I have tested with 1-degree in the past.

Will this be the restart directory used for the start of cycle 4?: /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731

[aekiss]

I've just merged master+bgc into 01deg_jra55v140_iaf_cycle4, so your https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc could be merged into 01deg_jra55v140_iaf_cycle4 via a pull request when we're ready.

Yes, /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731 is the restart we'll start from.
See https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle3/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf_cycle3.csv

metadata.yaml will need to be updated to document the BGC initial conditions used.

[hakaseh]

I actually created https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc so that it can be merged into 01deg_jra55v140_iaf_cycle4 without conflicts. There will be conflicts in BGC setup between master+bgc and cycle4 for example diag_table.

I can do pull request but is it possible to try merge with the version of 01deg_jra55v140_iaf_cycle4 before merging with master+bgc? I think this should be easier than resolving conflicts between master+bgc and https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc.

I have updated metadata.yaml to include BGC initial conditions in https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc.

[hakaseh]

Regarding the restart files for sea-ice BGC tracers, I added them and created the following new files based on /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731/ice/:

iced.2019-01-01-00000.nc, i2o.nc, o2i.nc.

The shell script to create these files are below (I couldn't attach *.sh here). Should I add this script to input_om2-bgc or create a separate github repo?

#Add sea-ice BGC tracers to the restart file

#Define the path to the restart file you want to add BGC tracers
path2restart=/g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731/ice

#Define the name of the restart file
filename=iced.2019-01-01-00000.nc

#Copy the restart file to pwd
cp ${path2restart}/${filename} .

#Add 2D fields.
for j in algalN nit
do
	ncap2 -O -s ${j}=iceumask*0 ${filename} ${filename}
done

#Add 3D fields.
for j in bgc_N_sk bgc_Nit_sk
do
	ncap2 -O -s ${j}=aicen*0 ${filename} ${filename}
done

#Next i2o.nc and o2i.nc

filename=i2o.nc

#Copy the restart file to pwd
cp ${path2restart}/${filename} .

for j in wnd10_io nit_io alg_io
do
    ncap2 -O -s ${j}=licefh_io*0 ${filename} ${filename}
done

filename=o2i.nc

#Copy the restart file to pwd
cp ${path2restart}/${filename} .

for j in ssn_i ssalg_i
do
    ncap2 -O -s ${j}=sst_i*0 ${filename} ${filename}
done

[aekiss]

Re your previous post, I've already resolved the conflicts between master+bgc and 01deg_jra55v140_iaf_cycle4 when I merged them: https://github.com/COSIMA/01deg_jra55_iaf/network

The differences between 01deg_jra55v140_iaf_cycle4 and 01deg_jra55v140_iaf_cycle4_bgc are not that big:
https://github.com/COSIMA/01deg_jra55_iaf/compare/01deg_jra55v140_iaf_cycle4..hakaseh:01deg_jra55v140_iaf_cycle4_bgc but there are a few conflicts that would need to be manually resolved.

You could try this in your fork by pulling the latest 01deg_jra55v140_iaf_cycle4 and either merging 01deg_jra55v140_iaf_cycle4 into 01deg_jra55v140_iaf_cycle4_bgc or vice-versa.
Have you resolved merge conflicts before? I could do it if you prefer.

[aekiss]

I think that script should be added to input_om2-bgc

[hakaseh]

Re your previous post, I've already resolved the conflicts between master+bgc and 01deg_jra55v140_iaf_cycle4 when I merged them: https://github.com/COSIMA/01deg_jra55_iaf/network

The differences between 01deg_jra55v140_iaf_cycle4 and 01deg_jra55v140_iaf_cycle4_bgc are not that big: https://github.com/COSIMA/01deg_jra55_iaf/compare/01deg_jra55v140_iaf_cycle4..hakaseh:01deg_jra55v140_iaf_cycle4_bgc but there are a few conflicts that would need to be manually resolved.

You could try this in your fork by pulling the latest 01deg_jra55v140_iaf_cycle4 and either merging 01deg_jra55v140_iaf_cycle4 into 01deg_jra55v140_iaf_cycle4_bgc or vice-versa. Have you resolved merge conflicts before? I could do it if you prefer.

I'm still not sure why we need to merge master+bgc into 01deg_jra55v140_iaf_cycle4 (and resolve conflicts) before merging 01deg_jra55v140_iaf_cycle4_bgc. Is it because so that master+bgc will recognise 01deg_jra55v140_iaf_cycle4 and vice versa? Still learning and trying to understand the github workflow.

I have resolved the conflicts and pushed the commit. will pull request now.

[aekiss]

No particular reason, it's just that I'd already merged master+bgc into 01deg_jra55v140_iaf_cycle4 before I realised you had a 01deg_jra55v140_iaf_cycle4_bgc branch.

[hakaseh]

I think that script should be added to input_om2-bgc

I just added the script to https://github.com/COSIMA/input_om2-bgc. Could you test the script?

[aekiss]

It worked for me. I've just pushed up a few tweaks. finalise.sh now adds provenance info to the warm start files too.

[hakaseh]

great!

what's the next step? we need to create /g/data/ik11/inputs/access-om2/input_bgc_20211105/01deg-cycle4 and add restarts and we are ready for testing??

[aekiss]

Yes I think this is getting close! Did you want me to make the cycle 4 inputs?

Also, just to double-check (sorry, I probably already asked about most of these): did/should we resolve/do any of these?

• did everything work well with the 1deg test run? I guess you fixed this coupling issue BGC included 1deg_jra55_iaf 1deg_jra55_iaf#11 (comment) but I can't recall - were you able to check for conservation? BGC included 1deg_jra55_iaf 1deg_jra55_iaf#11 (comment)
• was @PaulSpence going to try some BGC runs at 0.25deg with https://github.com/COSIMA/025deg_jra55_iaf/tree/master+bgc?
• gas fluxes via atmospheric pressure? BGC updates for the 4th cycle of 0.1-degree run and IAMIP2 mom-ocean/MOM5#338 (comment)
• fix CICE diag bugs diag bugs with AusCOM driver (affecting ACCESS-OM2) cice5#61 and diagnostic bug in CICE when outputting 4d fields cice5#62 ?

[hakaseh]

yes, could you please make the cycle 4 input? (not so familiar with the finalise.sh step, and since you made /g/data/ik11/inputs/access-om2/input_bgc_20211105/1deg, for consistency.
Note that it's only csiro_bgc.res.nc in the cycle 4 input that is different from /g/data/ik11/inputs/access-om2/input_bgc_20211105/1deg.

[hakaseh]

• the coupling issue is fixed, but conservation is not checked thoroughly. I'm going to do some test runs next week and will get back to you on this.
• atmospheric pressure is set to 1 atm at all times for now, but this is of secondary importance for gas fluxes.
• CICE diag bugs remain, but for BGC, it's not a problem (diag bugs with AusCOM driver (affecting ACCESS-OM2) cice5#61 (comment)).
• my test runs indicate that brine height tracer needs to be turned off to work well when sea-ice BGC is turned off (when brine height tracer is turned on while sea-ice BGC is off, spurious ice-to-ocean BGC flux exists, creating huge amount of nitrate in surface ocean). so for now, i will make the same change as d658de2 to master+bgc (e.g. https://github.com/COSIMA/1deg_jra55_iaf/tree/master+bgc).

Have a nice weekend 👋

[aekiss]

Thanks, you too - hope your weather there is better than here!
I'm making csiro_bgc.res.nc for cycle 4 now.

[aekiss]

OK those initial conditions are now in /g/data/ik11/inputs/access-om2/input_bgc_20211105/01deg-cycle4. I made them with https://github.com/COSIMA/input_om2-bgc/tree/afad69d which has a slightly tweaked version of the notebook.

The restart files are here: /g/data/v45/aek156/input_om2-bgc/*.nc

[hakaseh]

Just an update that intmld and int100 errors have been resolved, i think! (see the original issue: #11 (comment))

Here is the updated wombat with the fix: hakaseh/mom@398a0ae

I'm leaving my notes and test runs for record here (may not be useful to others)
intmld_int100_fix.pptx

[aekiss]

Awesome, thanks @hakaseh!
I'm not sure I understand the nature of the error - will it be possible to correct the existing intmld and int100 outputs, or should they be deleted?

[hakaseh]

so it's only dic_intmld,dic_int100, adic_intmld, and adic_int100 that need to be deleted (not recoverable). The rests are fine.

[aekiss]

The oxygen units bug has been fixed in /g/data/ik11/inputs/access-om2/input_bgc_20220224 at all resolutions.
I've also added long_name and units metadata to most fields.

[aekiss]

Just so I don't forget, the updated list of config changes is:

1. Jan 1958 - Nov 1959 inclusive: MOM outputs as in diag_table from here and these CICE outputs
2. Dec 1959 - Dec 1978 inclusive: as for 1, but without daily and monthly Sinz and Tinz CICE output
3. 1979 - 2013 inclusive: update diag_table with https://github.com/COSIMA/make_diag_table/tree/cycle4_1979-2013_2017-2018
4. 1987 - 2018 inclusive: add extra ice diagnostics for Pat Wongpan: f_albsni = 'md', dardg2dt = 'md', fswabs_ai = 'md', fhocn_ai = 'md' (NB: the last two now include daily), and these additional ones: f_bgc_n_sk = 'md', f_bgc_nit_sk = 'md', f_ppnet = 'md'
5. 2014 - 2015 inclusive: enable daily Sinz and Tinz output (would be good to fix diagnostic bug in CICE when outputting 4d fields cice5#62 first), and daily f_fswup, f_alvdr_ai, f_alidr_ai, f_alvdf_ai, f_alidf_ai, and 6-hourly aice, and update diag_table with https://github.com/COSIMA/make_diag_table/tree/cycle4_2014-2016
6. 2016 only: as for 5, but add daily aicen, vicen and 3-hourly uvel, vvel, divu, shear (generated in re-run - see below)
7. 2017 - 2018 inclusive: revert to config 4

[aekiss]

Cycle 4 is underway again. Although we had calculated up to Oct 1991 we've decided to discard this and re-run from 1 Jan 1979 onwards, resetting o2 to the corrected climatology so that we will have a longer o2 record.

The MOM code has also been fixed:

• dic_intmld, dic_int100, adic_intmld, and adic_int100 are now correct
• fe outputs have the correct units (involving umol, not mmol)
• _intmld and _int100 outputs have correct units and better longnames

The continuing run (backtracked to restart from 1979) is on the 01deg_jra55v140_iaf_cycle4 branch, and the abandoned previous run has been moved to branch 01deg_jra55v140_iaf_cycle4_OLD.

[aekiss]

For the record: All the daily and monthly ice data from April 2002 to the end of 2018 were accidentally deleted during post-processing. All MOM output, sea ice data prior to April 2002, and 6-hourly ice area from 2014-2016 inclusive were unaffected.

To fix this, 2002-2018 inclusive is being re-run on branch 01deg_jra55v140_iaf_cycle4_rerun_from_2002 to recreate the missing outputs following the schedule above (but not including the MOM output or 6-hourly ice output which we already have), and adding daily aicen, vicen and 3-hourly uvel, vvel, divu, shear for 2016 only.

[aekiss]

The cycle 4 run was completed a few weeks ago. Here's my announcement to the COSIMA mailing list:

---

01deg_jra55v140_iaf_cycle4, the 4th 1958-2018 interannually-forced ACCESS-OM2-01 cycle at 0.1 degree, is now available via the COSIMA Cookbook and from
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle4/

This is the first cycle to include any biogeochemistry, and also has nutrient and algae coupled between the ocean and sea ice (many thanks to Hakase Hayashida and Russ Fiedler for their contributions to the BGC configuration). The sea ice data is also complete, as I have re-generated the files I accidentally deleted.

Info on this run is here and an overview of the outputs (which includes a large number of BGC diagnostics) is given here.

There's also a new experiment 01deg_jra55v140_iaf_cycle4_jra55v150_extension which uses JRA55-do v1.5.0 and 1.5.0.1 (see COSIMA/access-om2#247) to extend cycle 4 to March 2022. This will be continued into the future, about 4 months behind real time. It includes the same diagnostics as the final year of cycle 4 (including BGC). It's available via the COSIMA Cookbook and from
/g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle4_jra55v150_extension
and metadata is here.

The outputs from both experiments can be searched via the COSIMA Cookbook database explorer
https://nbviewer.org/github/COSIMA/cosima-recipes/blob/master/Tutorials/Using_Explorer_tools.ipynb#Exploring-a-Cookbook-Database

You'll need to be a member of cj50 and ik11 to access them - apply at https://my.nci.org.au/mancini/project-search if you need to.

Outputs from cycles 1-3 are described here, and I'll add info for these new experiments when I get a chance.
http://cosima.org.au/index.php/2020/07/29/data-available-0-1-1958-2018-access-om2-iaf-run/

[aekiss]

Some of the BGC output is saved as one time level per file. In these cases the COSIMA Cookbook indexing considers it "static", so the frequency filter in cc.querying.getvar doesn't work.

The solution is to filter on the filename, e.g.

data = cc.querying.getvar('01deg_jra55v140_iaf_cycle4', 'phy', session, ncfile = '%monthly%')

See here for more: COSIMA/cosima-cookbook#306

[aekiss]

I've just updated this summary of IAF outputs to include cycle 4 and its extension:
http://cosima.org.au/index.php/2020/07/29/data-available-0-1-1958-2018-access-om2-iaf-run/