Difference between revisions of "ACCESS-ESM 1.5"

(Checking)
 
(29 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{ stub }}
 
  
Payu configuration: https://github.com/coecms/esm-pre-industrial
+
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them.  
  
See the README on the github page for information on how to run ACCESS-ESM with Payu
+
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:
  
CSIRO documentation: https://access-svn.nci.org.au/trac/cmip5/wiki
+
{| class="wikitable"
 +
|-
 +
! Submodel !! Model !! Version !! Notes
 +
|-
 +
| Atmosphere || [[UM]] || 7.3 ||
 +
|-
 +
| Land Surface || [[CABLE]] || Part of UM ||
 +
|-
 +
| Sea Ice || CICE || ||
 +
|-
 +
| Ocean || [[MOM]] || 5 ||
 +
|-
 +
| Coupler || OASIS-MCT || 3 ||
 +
|}
 +
 
 +
__TOC__
  
 
= Payu Experiments =
 
= Payu Experiments =
  
=== piControl ===
+
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu.
  
Default configuration, based off CSIRO ksh scripts
+
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.
  
=== Last Interglacial ===
+
* [https://github.com/coecms/esm-pre-industrial piControl]
 +
* [https://github.com/coecms/esm-historical historical]
 +
* [https://github.com/coecms/esm-ssp585 ssp585]
 +
* [https://github.com/coecms/esm-pmip-last-mil last millenium]
 +
* [https://github.com/coecms/esm-mh mid holocene]
 +
* [https://github.com/coecms/esm-lig last interglacial]
 +
* [https://github.com/coecms/access-esm-amip AMIP run (non-coupled atmosphere only)]
  
Science Contacts: Nick Yeung, Laurie Menviel
+
== Model Spinup ==
  
Note: Needs an updated UM build with different orbital parameters
+
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)
  
=CSRIO KSH Scripts=
+
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run
  
There is some documentation on google drive, CMS > Coupled Models
+
== Using Payu ==
  
 +
1. Load the [[Conda]] environment to access Payu
 
<syntaxhighlight>
 
<syntaxhighlight>
git clone git@bitbucket.org:climate-cms/csiro-scripts.git
+
module use /g/data/hh5/public/modules
cd csiro-scripts/original/build
+
module load conda/analysis3
make
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Turns out I needed access to the CABLE repo, see https://trac.nci.org.au/trac/cable/wiki/CableRegistration
+
2. Download the Payu experiment configuration
 
 
Other than that, compile worked fine.
 
 
 
===Running first test===
 
 
 
copy the scripts <span style="font-family:monospace">PI-C2C-1p5r29*}} from the {{original</span> directory to a new experiment directory.
 
 
 
Rename the scripts with
 
 
 
 
<syntaxhighlight>
 
<syntaxhighlight>
rename PI-C2C-1p5r29 test1 PI-C2C-1p5r29*
+
git clone https://github.com/coecms/esm-historical
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Make sure that the correct project is set in <span style="font-family:monospace">test1</span>:
+
3. Edit and run the warm-start script if needed
 
 
 
<syntaxhighlight>
 
<syntaxhighlight>
#PBS -P w35
+
./warm-start.sh
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Change initial date and final date in <span style="font-family:monospace">test1.init</span>:
+
4. Run the model
 
 
 
<syntaxhighlight>
 
<syntaxhighlight>
#-- Initial and Final Date of the Experiment
+
payu run
iniyear=1;      finalyear=1;            typeset -Z4 iniyear  finalyear
 
inimonth=1;    finalmonth=1;          typeset -Z2 inimonth finalmonth
 
iniday=1;      finalday=5;            typeset -Z2 iniday  finalday
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Then run
+
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.
  
<syntaxhighlight>
+
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration
qsub test1
 
</syntaxhighlight>
 
  
Link the <span style="font-family:monospace">build</span> directory to the experiment directory.
+
== Post-Processing ==
  
====Issues====
+
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.
  
The <span style="font-family:monospace">RUNID}} of {{PI-C2C-1p5r29}} was hardcoded in the script. I removed that hardcoded line and left the {{export RUNID=`basename $PBS_O_WORKDIR`</span> in because the experiment directory has the same name as the scripts.
+
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.
  
It didn't find the build directory in the experiment directory. I linked the build directory there.
+
== Experiment Details ==
  
The model failed asking for a restart dump that didn't exist. Deleting the <span style="font-family:monospace">test1.date</span> file solved that.
+
=== piControl ===
 +
You can download the configuration by typing
 +
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git
 +
</syntaxhighlight>
  
Didn't find <span style="font-family:monospace">create_rankfile.py}} -- copied from {{original}} directory. (Had to delete {{test1.date</span> again.)
+
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.
  
I also pre-emptively linked the <span style="font-family:monospace">runscripts}} directory from {{original</span> to my experiment directory.
+
Default configuration, based off CSIRO ksh scripts
  
==How ACCESS works==  
+
=== PMIP4 Runs ===
 +
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]
  
===<RUNID>===
+
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene.
  
The actual sumit script does not have an extension, and is a <span style="font-family:monospace">ksh</span> script.
+
Please refer to the main page to know where to download the configurations from.
It sets a whole lot of environment variables, most of them exported.
 
  
It then sources, if present, <span style="font-family:monospace">runscripts/umprofile}} which also sets some variables, however, many of these variables don't seem to refer to a path that exists on {{raijin</span>.
+
=== ''Last Interglacial'' ===
Then, again, if present, runs <span style="font-family:monospace">runscripts/setglobalvars</span>, again, the contents of which seem to be outdated.
+
Science Contacts: Nick Yeung, Laurie Menviel
  
It sets a few more variables, then sources <span style="font-family:monospace"><RUNID>.init</span>.
+
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at
 +
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe
 +
</syntaxhighlight>
  
Many more variable declarations, then it sources <span style="font-family:monospace">UMScr_Toplevel</span>.
+
=== ''Mid-Holocene'' ===
 +
Science Contacts: Josephine Brown
  
Finally, it sources <span style="font-family:monospace"><RUNID>.fin</span>
+
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at
 +
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div>
  
===<RUNID>.init===
 
  
This script is *sourced* by <span style="font-family:monospace"><RUNID></span>, so it's still a ksh script.
+
== AMIP ==
It begins with a function declaration, which sets certain ancillary files depending on the year.
 
  
Then come many more variables, including:
+
Science Contacts: Mustapha Adamu, Shayne McGregor
  
* <span style="font-family:monospace">CMIP5RUN}}, which can be any of these: {{picontrolv,historical,pi4xCO2,pi1pcntCO2,rcp45,rcp85,rcp26</span>.
+
An uncoupled, atmosphere only version of ACCESS-ESM is provided at [https://github.com/coecms/access-esm-amip this Github repository].
* <span style="font-family:monospace">nproc_ice</span> (12 currently)
 
* <span style="font-family:monospace">oce_nx}} and {{oce_ny</span> for ocean decomposition (currently 12 and 4, respectively)
 
* <span style="font-family:monospace">ntproc}} as the total number of cores for this job ({{UM_NPES}} + {{nproc_ice}} + {{(oce_nx * oce_ny)</span>)
 
* <span style="font-family:monospace">iniyear}}, {{inimonth}}, and {{iniday}} as the initial date of the model, which is then compressed into {{inidate}} ({{YYYYMMDD</span>)
 
* <span style="font-family:monospace">finalyear}}, {{finalmonth}}, {{finalday}} as the final date of the model, compressed into {{finaldate</span>.
 
  
Then it looks for <span style="font-family:monospace"><RUNID>.date</span>:
+
= CSIRO KSH Scripts =
  
If it '''doesn't''' exist, then it assumes that it's a new run. It creates the file and dumps the initial dates in there.
+
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].
If the file '''does''' exist, it reads the date out of it (only last line).
 
  
It then uses <span style="font-family:monospace">~access/bin/calendar_more</span> to calculate the times for this run (initial date, end date, first date of next run, et cetera).
+
= Fixing Ice Restart Dates (Payu) =
  
From this, it gets, amongst other things, the <span style="font-family:monospace">days_in_run}}, which it then multiplies by 86,400 (secs per day) to get {{runtime</span>.
+
In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).
  
Next, the script creates it work and run directories, copying files there as needed. (Apparently, for historical runs, it also needs to change the dates in the UM files, changing the reference date from the 16th of the month to the first.)
+
Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')
This is also where it uses the function it declared at the beginning.
 
  
Finally, it makes several (currently obscure) changes to a lot of namelists, and then creates the <span style="font-family:monospace">ACCESSRUNCMD</span>.
+
We'll need to work out some numbers to put into the restart directory, which can be done in Python
 +
<syntaxhighlight lang="python">
 +
import cftime
  
And it loads new modules, replacing others if they had been loaded before.
+
init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml
 +
dt = 3600 # Model timestep as set in ice/input_ice.nml
  
===<RUNID>.fin===
+
this_run_start = cftime.datetime(1996,1,1) # Date we want to start at
 +
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at
  
Check whether variable <span style="font-family:monospace">FCODE</span> is 0 (presumably the return code of the Model run?), only then will it do anything.
+
runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run
 +
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run
 +
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run
 +
istep0 = time / dt # Model steps between init_date and start of this run
  
It calculates the name of the restart file <span style="font-family:monospace">restartfile="aiihca.da${umdate}"}} where {{umdate}} has been created by {{datetoum}} or {{datetoum2</span>.
+
print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")
 +
</syntaxhighlight>
  
Next, it moves what I think are the coupling restart files, <span style="font-family:monospace">${cplrundir}/?2?.nc}} to the archive {{${archivedir}/restart/cpl/$resfile-${enddate}</span>
+
== Restart namelists ==
  
Then, it accesses <span style="font-family:monospace">$atmrundir}}: Validates the date of the restart file, moves it to {{${archivedir}/restart/atm/${restartarch}}} (({{restartarch}} had been set to {{"${RUNID}.astart-${nextdate}"</span>).
+
Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.
  
Next, it moves all UM output files (<span style="font-family:monospace">aiihca.p*</span>) to the archive dir, renaming them in the process.
+
In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.
  
==Modifications==  
+
== Restart file ==
  
Created a new branch: <span style="font-family:monospace">holger_testing</span>
+
Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like
 
+
<syntaxhighlight>
===MOM5 Version Control===
+
scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000
 
+
</syntaxhighlight>
MOM5 is under version control: <span style="font-family:monospace">https://github.com/OceansAus/ACCESS-ESM1.5-MOM5.git}} - this was added to the {{Makefile</span>.
 
  
===ummodel_hg3===
+
Give the output file a bigger number than any other restart to make sure Payu picks it up.
  
<span style="font-family:monospace">ummodel_hg3}} was still copied from {{/short/p66/txz599/ACCESSHOME/submodels/UM/ummodel_hg3/</span>. That directory itself is under version control, but with some non-checked-in.
+
== Checking ==
  
===umbase_hg3===
+
As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.
  
<span style="font-family:monospace">umbase_hg3}} was still copied. But there weren't any interesting changes in the directory compared the the repository. (Only {{fcm_env.sh}} and {{parsed_bld.cfg</span>)
+
Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'
So I've changed it to point to the svn repo.
 
  
===bld-hadgem3-mct.cfg===
+
Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like
 +
<pre>
 +
  istep0                    =   52584
 +
istep1:    52584    idate:  18611231    sec:        0
 +
Restart read at istep=       52584  189302400.000000
 +
istep1:    52584    idate:  18560101    sec:        0
 +
istep1:    52608    idate:  18560102    sec:        0
 +
istep1:    52632    idate:  18560103    sec:        0
 +
istep1:    52656    idate:  18560104    sec:        0
 +
istep1:    52680    idate:  18560105    sec:        0
 +
istep1:    52704    idate:  18560106    sec:        0
 +
istep1:    52728    idate:  18560107    sec:        0
 +
...
 +
</pre>
  
The file <span style="font-family:monospace">bld-hadgem3-mct.cfg}} was missing from the {{ummodel_hg3}} repo. For now, I've made it part of the access-esm repo, and added a line in the {{Makefile</span> to copy it over.
+
After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.
  
 +
After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.
  
[[Category: ACCESS]][[Category: Coupled Models]]
+
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]

Latest revision as of 19:10, 30 September 2021

This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them.

ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:

Submodel Model Version Notes
Atmosphere UM 7.3
Land Surface CABLE Part of UM
Sea Ice CICE
Ocean MOM 5
Coupler OASIS-MCT 3

Payu Experiments

We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu.

Please contact the CMS team if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.

Model Spinup

If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)

The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run

Using Payu

1. Load the Conda environment to access Payu

module use /g/data/hh5/public/modules
module load conda/analysis3

2. Download the Payu experiment configuration

git clone https://github.com/coecms/esm-historical

3. Edit and run the warm-start script if needed

./warm-start.sh

4. Run the model

payu run

Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.

We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration

Post-Processing

Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.

Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.

Experiment Details

piControl

You can download the configuration by typing

$ git clone https://github.com/coecms/esm-pre-industrial.git

The website of the configuration also contains detailed instructions on how to run the model.

Default configuration, based off CSIRO ksh scripts

PMIP4 Runs

Main Page:ACCESS-ESM-PMIP4

Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene.

Please refer to the main page to know where to download the configurations from.

Last Interglacial

Science Contacts: Nick Yeung, Laurie Menviel

Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at

/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe

Mid-Holocene

Science Contacts: Josephine Brown

Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at

/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe


AMIP

Science Contacts: Mustapha Adamu, Shayne McGregor

An uncoupled, atmosphere only version of ACCESS-ESM is provided at this Github repository.

CSIRO KSH Scripts

CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look here.

Fixing Ice Restart Dates (Payu)

In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).

Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')

We'll need to work out some numbers to put into the restart directory, which can be done in Python

import cftime

init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml
dt = 3600 # Model timestep as set in ice/input_ice.nml

this_run_start = cftime.datetime(1996,1,1) # Date we want to start at
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at

runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run
istep0 = time / dt # Model steps between init_date and start of this run

print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")

Restart namelists

Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.

In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.

Restart file

Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like

scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000

Give the output file a bigger number than any other restart to make sure Payu picks it up.

Checking

As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.

Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'

Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like

  istep0                    =    52584
istep1:     52584    idate:  18611231    sec:         0
 Restart read at istep=       52584   189302400.000000
istep1:     52584    idate:  18560101    sec:         0
istep1:     52608    idate:  18560102    sec:         0
istep1:     52632    idate:  18560103    sec:         0
istep1:     52656    idate:  18560104    sec:         0
istep1:     52680    idate:  18560105    sec:         0
istep1:     52704    idate:  18560106    sec:         0
istep1:     52728    idate:  18560107    sec:         0
...

After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.

After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.