Difference between revisions of "ACCESS-ESM 1.5"

(Checking)
 
(21 intermediate revisions by 3 users not shown)
Line 1: Line 1:
  
{{Stub}}
+
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them.
  
Payu configuration: [https://github.com/coecms/esm-pre-industrial https://github.com/coecms/esm-pre-industrial]
+
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:
  
See the README on the github page for information on how to run ACCESS-ESM with Payu
+
{| class="wikitable"
 +
|-
 +
! Submodel !! Model !! Version !! Notes
 +
|-
 +
| Atmosphere || [[UM]] || 7.3 ||
 +
|-
 +
| Land Surface || [[CABLE]] || Part of UM ||
 +
|-
 +
| Sea Ice || CICE || ||
 +
|-
 +
| Ocean || [[MOM]] || 5 ||
 +
|-
 +
| Coupler || OASIS-MCT || 3 ||
 +
|}
  
CSIRO documentation: [https://access-svn.nci.org.au/trac/cmip5/wiki https://access-svn.nci.org.au/trac/cmip5/wiki]
+
__TOC__
 
 
 
 
  
 
= Payu Experiments =
 
= Payu Experiments =
  
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. You can download the configuration by typing
+
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu.  
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git
 
</syntaxhighlight>
 
 
 
The website ([https://github.com/coecms/esm-pre-industrial https://github.com/coecms/esm-pre-industrial]) of the configuration also contains detailed instructions on how to run the model.
 
 
 
== Configurations ==
 
 
 
=== piControl ===
 
 
 
Default configuration, based off CSIRO ksh scripts
 
  
 +
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.
  
=== PMIP4 Runs ===
+
* [https://github.com/coecms/esm-pre-industrial piControl]
 +
* [https://github.com/coecms/esm-historical historical]
 +
* [https://github.com/coecms/esm-ssp585 ssp585]
 +
* [https://github.com/coecms/esm-pmip-last-mil last millenium]
 +
* [https://github.com/coecms/esm-mh mid holocene]
 +
* [https://github.com/coecms/esm-lig last interglacial]
 +
* [https://github.com/coecms/access-esm-amip AMIP run (non-coupled atmosphere only)]
  
Main Page:&nbsp;[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]
+
== Model Spinup ==
  
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene
+
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)
  
==== Last Interglacial ====
+
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run
  
Science Contacts: Nick Yeung, Laurie Menviel
+
== Using Payu ==
  
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is at
+
1. Load the [[Conda]] environment to access Payu
<syntaxhighlight lang="unknown">/short/public/access-esm/payu/bin/coe/um_hg3.exe-20190412-pmip-r327
+
<syntaxhighlight>
 +
module use /g/data/hh5/public/modules
 +
module load conda/analysis3
 
</syntaxhighlight>
 
</syntaxhighlight>
  
= CSIRO KSH Scripts =
+
2. Download the Payu experiment configuration
 
+
<syntaxhighlight>
There is some documentation on google drive, CMS > Coupled Models
+
git clone https://github.com/coecms/esm-historical
 
 
&nbsp;
 
<syntaxhighlight lang="unknown">
 
git clone git@bitbucket.org:climate-cms/csiro-scripts.git
 
cd csiro-scripts/original/build
 
make
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Turns out I needed access to the CABLE repo, see [https://trac.nci.org.au/trac/cable/wiki/CableRegistration https://trac.nci.org.au/trac/cable/wiki/CableRegistration]
+
3. Edit and run the warm-start script if needed
 
+
<syntaxhighlight>
Other than that, compile worked fine.
+
./warm-start.sh
 
 
=== Running first test ===
 
 
 
copy the scripts <code>PI-C2C-1p5r29*</code> from the <code>original</code> directory to a new experiment directory.
 
 
 
Rename the scripts with
 
 
 
&nbsp;
 
<syntaxhighlight lang="unknown">
 
rename PI-C2C-1p5r29 test1 PI-C2C-1p5r29*
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Make sure that the correct project is set in <code>test1</code>:
+
4. Run the model
 
+
<syntaxhighlight>
&nbsp;
+
payu run
<syntaxhighlight lang="unknown">
 
#PBS -P w35
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Change initial date and final date in <code>test1.init</code>:
+
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.
  
&nbsp;
+
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration
<syntaxhighlight lang="unknown">
 
#-- Initial and Final Date of the Experiment
 
iniyear=1;      finalyear=1;            typeset -Z4 iniyear  finalyear
 
inimonth=1;    finalmonth=1;          typeset -Z2 inimonth finalmonth
 
iniday=1;      finalday=5;            typeset -Z2 iniday  finalday
 
</syntaxhighlight>
 
  
Then run
+
== Post-Processing ==
  
&nbsp;
+
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.
<syntaxhighlight lang="unknown">
 
qsub test1
 
</syntaxhighlight>
 
  
Link the <code>build</code> directory to the experiment directory.
+
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.
  
==== Issues ====
+
== Experiment Details ==
  
The RUNID of <code>PI-C2C-1p5r29</code> was hardcoded in the script. I removed that hardcoded line and left the <code>export RUNID=`basename $PBS_O_WORKDIR`</code> in because the experiment directory has the same name as the scripts.
+
=== piControl ===
 +
You can download the configuration by typing
 +
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git
 +
</syntaxhighlight>
  
It didn't find the build directory in the experiment directory. I linked the build directory there.
+
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.
  
The model failed asking for a restart dump that didn't exist. Deleting the <code>test1.date</code> file solved that.
+
Default configuration, based off CSIRO ksh scripts
  
Didn't find <code>create_rankfile.py</code> -- copied from <code>original</code> directory. (Had to delete <code>test1.date</code> again.)
+
=== PMIP4 Runs ===
 +
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]
  
I also pre-emptively linked the <code>runscripts</code> directory from <code>original</code> to my experiment directory.
+
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene.  
  
== How ACCESS works ==
+
Please refer to the main page to know where to download the configurations from.
  
=== <RUNID> ===
+
=== ''Last Interglacial'' ===
 +
Science Contacts: Nick Yeung, Laurie Menviel
  
The actual sumit script does not have an extension, and is a <code>ksh</code> script. It sets a whole lot of environment variables, most of them exported.
+
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at
 +
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe
 +
</syntaxhighlight>
  
It then sources, if present, <code>runscripts/umprofile</code> which also sets some variables, however, many of these variables don't seem to refer to a path that exists on <code>raijin</code>. Then, again, if present, runs <code>runscripts/setglobalvars</code>, again, the contents of which seem to be outdated.
+
=== ''Mid-Holocene'' ===
 +
Science Contacts: Josephine Brown
  
It sets a few more variables, then sources <code><RUNID>.init</code>.
+
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at
 +
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div>
  
Many more variable declarations, then it sources <code>UMScr_Toplevel</code>.
 
  
Finally, it sources <code><RUNID>.fin</code>
+
== AMIP ==
  
=== <RUNID>.init ===
+
Science Contacts: Mustapha Adamu, Shayne McGregor
  
This script is *sourced* by <code><RUNID></code>, so it's still a ksh script. It begins with a function declaration, which sets certain ancillary files depending on the year.
+
An uncoupled, atmosphere only version of ACCESS-ESM is provided at [https://github.com/coecms/access-esm-amip this Github repository].
  
Then come many more variables, including:
+
= CSIRO KSH Scripts =
  
*<code>CMIP5RUN</code>, which can be any of these: <code>picontrolv,historical,pi4xCO2,pi1pcntCO2,rcp45,rcp85,rcp26</code>.
+
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].
*<code>nproc_ice</code> (12 currently)
 
*<code>oce_nx</code> and <code>oce_ny</code> for ocean decomposition (currently 12 and 4, respectively)
 
*<code>ntproc</code> as the total number of cores for this job (<code>UM_NPES</code> + <code>nproc_ice</code> + <code>(oce_nx * oce_ny)</code>)
 
*<code>iniyear</code>, <code>inimonth</code>, and <code>iniday</code> as the initial date of the model, which is then compressed into <code>inidate</code> (<code>YYYYMMDD</code>)
 
*<code>finalyear</code>, <code>finalmonth</code>, <code>finalday</code> as the final date of the model, compressed into <code>finaldate</code>.  
 
  
Then it looks for <code><RUNID>.date</code>:
+
= Fixing Ice Restart Dates (Payu) =
  
If it '''doesn't''' exist, then it assumes that it's a new run. It creates the file and dumps the initial dates in there. If the file '''does''' exist, it reads the date out of it (only last line).
+
In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).
  
It then uses <code>~access/bin/calendar_more</code> to calculate the times for this run (initial date, end date, first date of next run, et cetera).
+
Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')
  
From this, it gets, amongst other things, the <code>days_in_run</code>, which it then multiplies by 86,400 (secs per day) to get <code>runtime</code>.
+
We'll need to work out some numbers to put into the restart directory, which can be done in Python
 +
<syntaxhighlight lang="python">
 +
import cftime
  
Next, the script creates it work and run directories, copying files there as needed. (Apparently, for historical runs, it also needs to change the dates in the UM files, changing the reference date from the 16th of the month to the first.) This is also where it uses the function it declared at the beginning.
+
init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml
 +
dt = 3600 # Model timestep as set in ice/input_ice.nml
  
Finally, it makes several (currently obscure) changes to a lot of namelists, and then creates the <code>ACCESSRUNCMD</code>.
+
this_run_start = cftime.datetime(1996,1,1) # Date we want to start at
 +
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at
  
And it loads new modules, replacing others if they had been loaded before.
+
runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run
 +
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run
 +
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run
 +
istep0 = time / dt # Model steps between init_date and start of this run
  
=== <RUNID>.fin ===
+
print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")
 +
</syntaxhighlight>
  
Check whether variable <code>FCODE</code> is 0 (presumably the return code of the Model run?), only then will it do anything.
+
== Restart namelists ==
  
It calculates the name of the restart file <code>restartfile="aiihca.da${umdate}"</code> where <code>umdate</code> has been created by <code>datetoum</code> or <code>datetoum2</code>.
+
Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.
  
Next, it moves what I think are the coupling restart files, <code>${cplrundir}/?2?.nc</code> to the archive <code>${archivedir}/restart/cpl/$resfile-${enddate}</code>
+
In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.
  
Then, it accesses <code>$atmrundir</code>: Validates the date of the restart file, moves it to <code>${archivedir}/restart/atm/${restartarch</code>} ((<code>restartarch</code> had been set to <code>"${RUNID}.astart-${nextdate}"</code>).
+
== Restart file ==
  
Next, it moves all UM output files (<code>aiihca.p*</code>) to the archive dir, renaming them in the process.
+
Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like
 
+
<syntaxhighlight>
== Modifications ==
+
scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000
 
+
</syntaxhighlight>
Created a new branch: <code>holger_testing</code>
 
 
 
=== MOM5 Version Control ===
 
  
MOM5 is under version control: <code>[https://github.com/OceansAus/ACCESS-ESM1.5-MOM5.git https://github.com/OceansAus/ACCESS-ESM1.5-MOM5.git]</code> - this was added to the <code>Makefile</code>.
+
Give the output file a bigger number than any other restart to make sure Payu picks it up.
  
=== ummodel_hg3 ===
+
== Checking ==
  
<code>ummodel_hg3</code> was still copied from <code>/short/p66/txz599/ACCESSHOME/submodels/UM/ummodel_hg3/</code>. That directory itself is under version control, but with some non-checked-in.
+
As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.
  
=== umbase_hg3 ===
+
Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'
  
<code>umbase_hg3</code> was still copied. But there weren't any interesting changes in the directory compared the the repository. (Only <code>fcm_env.sh</code> and <code>parsed_bld.cfg</code>) So I've changed it to point to the svn repo.
+
Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like
 +
<pre>
 +
  istep0                    =    52584
 +
istep1:    52584    idate:  18611231    sec:        0
 +
Restart read at istep=      52584  189302400.000000
 +
istep1:    52584    idate:  18560101    sec:        0
 +
istep1:    52608    idate:  18560102    sec:        0
 +
istep1:    52632    idate:  18560103    sec:        0
 +
istep1:    52656    idate:  18560104    sec:        0
 +
istep1:    52680    idate:  18560105    sec:        0
 +
istep1:    52704    idate:  18560106    sec:        0
 +
istep1:    52728    idate:  18560107    sec:        0
 +
...
 +
</pre>
  
=== bld-hadgem3-mct.cfg ===
+
After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.
  
The file <code>bld-hadgem3-mct.cfg</code> was missing from the <code>ummodel_hg3</code> repo. For now, I've made it part of the access-esm repo, and added a line in the <code>Makefile</code> to copy it over.
+
After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.
  
 
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]
 
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]

Latest revision as of 20:10, 30 September 2021

This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them.

ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:

Submodel Model Version Notes
Atmosphere UM 7.3
Land Surface CABLE Part of UM
Sea Ice CICE
Ocean MOM 5
Coupler OASIS-MCT 3

Payu Experiments

We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu.

Please contact the CMS team if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.

Model Spinup

If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)

The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run

Using Payu

1. Load the Conda environment to access Payu

module use /g/data/hh5/public/modules
module load conda/analysis3

2. Download the Payu experiment configuration

git clone https://github.com/coecms/esm-historical

3. Edit and run the warm-start script if needed

./warm-start.sh

4. Run the model

payu run

Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.

We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration

Post-Processing

Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.

Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.

Experiment Details

piControl

You can download the configuration by typing

$ git clone https://github.com/coecms/esm-pre-industrial.git

The website of the configuration also contains detailed instructions on how to run the model.

Default configuration, based off CSIRO ksh scripts

PMIP4 Runs

Main Page:ACCESS-ESM-PMIP4

Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene.

Please refer to the main page to know where to download the configurations from.

Last Interglacial

Science Contacts: Nick Yeung, Laurie Menviel

Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at

/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe

Mid-Holocene

Science Contacts: Josephine Brown

Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at

/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe


AMIP

Science Contacts: Mustapha Adamu, Shayne McGregor

An uncoupled, atmosphere only version of ACCESS-ESM is provided at this Github repository.

CSIRO KSH Scripts

CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look here.

Fixing Ice Restart Dates (Payu)

In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).

Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')

We'll need to work out some numbers to put into the restart directory, which can be done in Python

import cftime

init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml
dt = 3600 # Model timestep as set in ice/input_ice.nml

this_run_start = cftime.datetime(1996,1,1) # Date we want to start at
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at

runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run
istep0 = time / dt # Model steps between init_date and start of this run

print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")

Restart namelists

Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.

In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.

Restart file

Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like

scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000

Give the output file a bigger number than any other restart to make sure Payu picks it up.

Checking

As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.

Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'

Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like

  istep0                    =    52584
istep1:     52584    idate:  18611231    sec:         0
 Restart read at istep=       52584   189302400.000000
istep1:     52584    idate:  18560101    sec:         0
istep1:     52608    idate:  18560102    sec:         0
istep1:     52632    idate:  18560103    sec:         0
istep1:     52656    idate:  18560104    sec:         0
istep1:     52680    idate:  18560105    sec:         0
istep1:     52704    idate:  18560106    sec:         0
istep1:     52728    idate:  18560107    sec:         0
...

After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.

After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.