Difference between revisions of "ACCESS-ESM 1.5"

(Checking)
 
(32 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Temporary page while I'm getting familiar with ACCESS-ESM 1.5.
 
  
Will be converted into a documentation later on.
+
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them.  
  
=Getting ACCESS-ESM1.5=
+
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:
  
There is some documentation on google drive, CMS > Coupled Models
+
{| class="wikitable"
 +
|-
 +
! Submodel !! Model !! Version !! Notes
 +
|-
 +
| Atmosphere || [[UM]] || 7.3 ||
 +
|-
 +
| Land Surface || [[CABLE]] || Part of UM ||
 +
|-
 +
| Sea Ice || CICE || ||
 +
|-
 +
| Ocean || [[MOM]] || 5 ||
 +
|-
 +
| Coupler || OASIS-MCT || 3 ||
 +
|}
  
<syntaxhighlight>
+
__TOC__
git clone git@bitbucket.org:climate-cms/csiro-scripts.git
+
 
cd csiro-scripts/original/build
+
= Payu Experiments =
make
+
 
</syntaxhighlight>
+
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu.
 +
 
 +
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.
  
Turns out I needed access to the CABLE repo, see https://trac.nci.org.au/trac/cable/wiki/CableRegistration
+
* [https://github.com/coecms/esm-pre-industrial piControl]
 +
* [https://github.com/coecms/esm-historical historical]
 +
* [https://github.com/coecms/esm-ssp585 ssp585]
 +
* [https://github.com/coecms/esm-pmip-last-mil last millenium]
 +
* [https://github.com/coecms/esm-mh mid holocene]
 +
* [https://github.com/coecms/esm-lig last interglacial]
 +
* [https://github.com/coecms/access-esm-amip AMIP run (non-coupled atmosphere only)]
  
Other than that, compile worked fine.
+
== Model Spinup ==
  
===Running first test===
+
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)
  
copy the scripts <span style="font-family:monospace">PI-C2C-1p5r29*}} from the {{original</span> directory to a new experiment directory.
+
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run
  
Rename the scripts with
+
== Using Payu ==
  
 +
1. Load the [[Conda]] environment to access Payu
 
<syntaxhighlight>
 
<syntaxhighlight>
rename PI-C2C-1p5r29 test1 PI-C2C-1p5r29*
+
module use /g/data/hh5/public/modules
 +
module load conda/analysis3
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Make sure that the correct project is set in <span style="font-family:monospace">test1</span>:
+
2. Download the Payu experiment configuration
 
 
 
<syntaxhighlight>
 
<syntaxhighlight>
#PBS -P w35
+
git clone https://github.com/coecms/esm-historical
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Change initial date and final date in <span style="font-family:monospace">test1.init</span>:
+
3. Edit and run the warm-start script if needed
 
 
 
<syntaxhighlight>
 
<syntaxhighlight>
#-- Initial and Final Date of the Experiment
+
./warm-start.sh
iniyear=1;      finalyear=1;            typeset -Z4 iniyear  finalyear
 
inimonth=1;    finalmonth=1;          typeset -Z2 inimonth finalmonth
 
iniday=1;      finalday=5;            typeset -Z2 iniday  finalday
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Then run
+
4. Run the model
 
 
 
<syntaxhighlight>
 
<syntaxhighlight>
qsub test1
+
payu run
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Link the <span style="font-family:monospace">build</span> directory to the experiment directory.
+
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.
  
====Issues====
+
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration
  
The <span style="font-family:monospace">RUNID}} of {{PI-C2C-1p5r29}} was hardcoded in the script. I removed that hardcoded line and left the {{export RUNID=`basename $PBS_O_WORKDIR`</span> in because the experiment directory has the same name as the scripts.
+
== Post-Processing ==
  
It didn't find the build directory in the experiment directory. I linked the build directory there.
+
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.
  
The model failed asking for a restart dump that didn't exist. Deleting the <span style="font-family:monospace">test1.date</span> file solved that.
+
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.
  
Didn't find <span style="font-family:monospace">create_rankfile.py}} -- copied from {{original}} directory. (Had to delete {{test1.date</span> again.)
+
== Experiment Details ==
  
I also pre-emptively linked the <span style="font-family:monospace">runscripts}} directory from {{original</span> to my experiment directory.
+
=== piControl ===
 +
You can download the configuration by typing
 +
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git
 +
</syntaxhighlight>
  
==How ACCESS works==
+
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.
  
===<RUNID>===
+
Default configuration, based off CSIRO ksh scripts
  
The actual sumit script does not have an extension, and is a <span style="font-family:monospace">ksh</span> script.
+
=== PMIP4 Runs ===
It sets a whole lot of environment variables, most of them exported.
+
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]
  
It then sources, if present, <span style="font-family:monospace">runscripts/umprofile}} which also sets some variables, however, many of these variables don't seem to refer to a path that exists on {{raijin</span>.
+
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene.  
Then, again, if present, runs <span style="font-family:monospace">runscripts/setglobalvars</span>, again, the contents of which seem to be outdated.
 
  
It sets a few more variables, then sources <span style="font-family:monospace"><RUNID>.init</span>.
+
Please refer to the main page to know where to download the configurations from.
  
Many more variable declarations, then it sources <span style="font-family:monospace">UMScr_Toplevel</span>.
+
=== ''Last Interglacial'' ===
 +
Science Contacts: Nick Yeung, Laurie Menviel
  
Finally, it sources <span style="font-family:monospace"><RUNID>.fin</span>
+
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at
 +
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe
 +
</syntaxhighlight>
  
===<RUNID>.init===  
+
=== ''Mid-Holocene'' ===
 +
Science Contacts: Josephine Brown
  
This script is *sourced* by <span style="font-family:monospace"><RUNID></span>, so it's still a ksh script.
+
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at
It begins with a function declaration, which sets certain ancillary files depending on the year.
+
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div>
  
Then come many more variables, including:
 
  
* <span style="font-family:monospace">CMIP5RUN}}, which can be any of these: {{picontrolv,historical,pi4xCO2,pi1pcntCO2,rcp45,rcp85,rcp26</span>.
+
== AMIP ==
* <span style="font-family:monospace">nproc_ice</span> (12 currently)
 
* <span style="font-family:monospace">oce_nx}} and {{oce_ny</span> for ocean decomposition (currently 12 and 4, respectively)
 
* <span style="font-family:monospace">ntproc}} as the total number of cores for this job ({{UM_NPES}} + {{nproc_ice}} + {{(oce_nx * oce_ny)</span>)
 
* <span style="font-family:monospace">iniyear}}, {{inimonth}}, and {{iniday}} as the initial date of the model, which is then compressed into {{inidate}} ({{YYYYMMDD</span>)
 
* <span style="font-family:monospace">finalyear}}, {{finalmonth}}, {{finalday}} as the final date of the model, compressed into {{finaldate</span>.
 
  
Then it looks for <span style="font-family:monospace"><RUNID>.date</span>:
+
Science Contacts: Mustapha Adamu, Shayne McGregor
  
If it '''doesn't''' exist, then it assumes that it's a new run. It creates the file and dumps the initial dates in there.
+
An uncoupled, atmosphere only version of ACCESS-ESM is provided at [https://github.com/coecms/access-esm-amip this Github repository].
If the file '''does''' exist, it reads the date out of it (only last line).
 
  
It then uses <span style="font-family:monospace">~access/bin/calendar_more</span> to calculate the times for this run (initial date, end date, first date of next run, et cetera).
+
= CSIRO KSH Scripts =
  
From this, it gets, amongst other things, the <span style="font-family:monospace">days_in_run}}, which it then multiplies by 86,400 (secs per day) to get {{runtime</span>.
+
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].
  
Next, the script creates it work and run directories, copying files there as needed. (Apparently, for historical runs, it also needs to change the dates in the UM files, changing the reference date from the 16th of the month to the first.)
+
= Fixing Ice Restart Dates (Payu) =
This is also where it uses the function it declared at the beginning.
 
  
Finally, it makes several (currently obscure) changes to a lot of namelists, and then creates the <span style="font-family:monospace">ACCESSRUNCMD</span>.
+
In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).
  
And it loads new modules, replacing others if they had been loaded before.
+
Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')
  
===<RUNID>.fin===
+
We'll need to work out some numbers to put into the restart directory, which can be done in Python
 +
<syntaxhighlight lang="python">
 +
import cftime
  
Check whether variable <span style="font-family:monospace">FCODE</span> is 0 (presumably the return code of the Model run?), only then will it do anything.
+
init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml
 +
dt = 3600 # Model timestep as set in ice/input_ice.nml
  
It calculates the name of the restart file <span style="font-family:monospace">restartfile="aiihca.da${umdate}"}} where {{umdate}} has been created by {{datetoum}} or {{datetoum2</span>.
+
this_run_start = cftime.datetime(1996,1,1) # Date we want to start at
 +
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at
  
Next, it moves what I think are the coupling restart files, <span style="font-family:monospace">${cplrundir}/?2?.nc}} to the archive {{${archivedir}/restart/cpl/$resfile-${enddate}</span>
+
runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run
 +
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run
 +
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run
 +
istep0 = time / dt # Model steps between init_date and start of this run
  
Then, it accesses <span style="font-family:monospace">$atmrundir}}: Validates the date of the restart file, moves it to {{${archivedir}/restart/atm/${restartarch}}} (({{restartarch}} had been set to {{"${RUNID}.astart-${nextdate}"</span>).
+
print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")
 +
</syntaxhighlight>
  
Next, it moves all UM output files (<span style="font-family:monospace">aiihca.p*</span>) to the archive dir, renaming them in the process.
+
== Restart namelists ==
  
==Modifications==
+
Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.
  
Created a new branch: <span style="font-family:monospace">holger_testing</span>
+
In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.
  
===MOM5 Version Control===  
+
== Restart file ==
  
MOM5 is under version control: <span style="font-family:monospace">https://github.com/OceansAus/ACCESS-ESM1.5-MOM5.git}} - this was added to the {{Makefile</span>.
+
Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like
 +
<syntaxhighlight>
 +
scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000
 +
</syntaxhighlight>
  
===ummodel_hg3===
+
Give the output file a bigger number than any other restart to make sure Payu picks it up.
  
<span style="font-family:monospace">ummodel_hg3}} was still copied from {{/short/p66/txz599/ACCESSHOME/submodels/UM/ummodel_hg3/</span>. That directory itself is under version control, but with some non-checked-in.
+
== Checking ==
  
===umbase_hg3===
+
As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.
  
<span style="font-family:monospace">umbase_hg3}} was still copied. But there weren't any interesting changes in the directory compared the the repository. (Only {{fcm_env.sh}} and {{parsed_bld.cfg</span>)
+
Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'
So I've changed it to point to the svn repo.
 
  
===bld-hadgem3-mct.cfg===
+
Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like
 +
<pre>
 +
  istep0                    =   52584
 +
istep1:    52584    idate:  18611231    sec:        0
 +
Restart read at istep=       52584  189302400.000000
 +
istep1:    52584    idate:  18560101    sec:        0
 +
istep1:    52608    idate:  18560102    sec:        0
 +
istep1:    52632    idate:  18560103    sec:        0
 +
istep1:    52656    idate:  18560104    sec:        0
 +
istep1:    52680    idate:  18560105    sec:        0
 +
istep1:    52704    idate:  18560106    sec:        0
 +
istep1:    52728    idate:  18560107    sec:        0
 +
...
 +
</pre>
  
The file <span style="font-family:monospace">bld-hadgem3-mct.cfg}} was missing from the {{ummodel_hg3}} repo. For now, I've made it part of the access-esm repo, and added a line in the {{Makefile</span> to copy it over.
+
After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.
  
 +
After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.
  
[[Category: ACCESS]][[Category: Coupled Models]]
+
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]

Latest revision as of 20:10, 30 September 2021

This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them.

ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:

Submodel Model Version Notes
Atmosphere UM 7.3
Land Surface CABLE Part of UM
Sea Ice CICE
Ocean MOM 5
Coupler OASIS-MCT 3

Payu Experiments

We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu.

Please contact the CMS team if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.

Model Spinup

If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)

The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run

Using Payu

1. Load the Conda environment to access Payu

module use /g/data/hh5/public/modules
module load conda/analysis3

2. Download the Payu experiment configuration

git clone https://github.com/coecms/esm-historical

3. Edit and run the warm-start script if needed

./warm-start.sh

4. Run the model

payu run

Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.

We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration

Post-Processing

Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.

Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.

Experiment Details

piControl

You can download the configuration by typing

$ git clone https://github.com/coecms/esm-pre-industrial.git

The website of the configuration also contains detailed instructions on how to run the model.

Default configuration, based off CSIRO ksh scripts

PMIP4 Runs

Main Page:ACCESS-ESM-PMIP4

Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene.

Please refer to the main page to know where to download the configurations from.

Last Interglacial

Science Contacts: Nick Yeung, Laurie Menviel

Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at

/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe

Mid-Holocene

Science Contacts: Josephine Brown

Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at

/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe


AMIP

Science Contacts: Mustapha Adamu, Shayne McGregor

An uncoupled, atmosphere only version of ACCESS-ESM is provided at this Github repository.

CSIRO KSH Scripts

CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look here.

Fixing Ice Restart Dates (Payu)

In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).

Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')

We'll need to work out some numbers to put into the restart directory, which can be done in Python

import cftime

init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml
dt = 3600 # Model timestep as set in ice/input_ice.nml

this_run_start = cftime.datetime(1996,1,1) # Date we want to start at
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at

runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run
istep0 = time / dt # Model steps between init_date and start of this run

print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")

Restart namelists

Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.

In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.

Restart file

Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like

scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000

Give the output file a bigger number than any other restart to make sure Payu picks it up.

Checking

As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.

Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'

Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like

  istep0                    =    52584
istep1:     52584    idate:  18611231    sec:         0
 Restart read at istep=       52584   189302400.000000
istep1:     52584    idate:  18560101    sec:         0
istep1:     52608    idate:  18560102    sec:         0
istep1:     52632    idate:  18560103    sec:         0
istep1:     52656    idate:  18560104    sec:         0
istep1:     52680    idate:  18560105    sec:         0
istep1:     52704    idate:  18560106    sec:         0
istep1:     52728    idate:  18560107    sec:         0
...

After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.

After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.