Difference between revisions of "ACCESS1.0 CMIP"

Line 12: Line 12:
  
 
NOTE: Consider using [[ACCESS-ESM 1.5]] for new runs on Gadi, it is much simpler to use
 
NOTE: Consider using [[ACCESS-ESM 1.5]] for new runs on Gadi, it is much simpler to use
 +
 +
The component models of ACCESS-CM 1.0 need to be rebuilt in order to run on Gadi, and there are few modifications required to the run script. The instructions here are for the historical experiment mentioned at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin, you may need to modify them if running a different experiment.
  
 
1. Build the models
 
1. Build the models
Line 25: Line 27:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
3. Set the recommended PBS headers for Gadi
+
3. Set the recommended PBS headers for Gadi in your main run script (e.g. exp/a10_hist/a10_hist)
 
<syntaxhighlight lang=text>
 
<syntaxhighlight lang=text>
 
#PBS -q normal
 
#PBS -q normal
Line 36: Line 38:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
4. Set the recommended Decomposition (for a total of 144 cpus, 384 gb memory)
+
4. Set the recommended Decomposition (for a total of 144 cpus, 384 gb memory) by modifying variables in the main and init scripts (e.g. exp/a10_hist/a10_hist, exp/a10_hist/a10_hist.init)
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
Line 54: Line 56:
 
|}
 
|}
  
5. Fix the main script (e.g. 'a10_hist')
+
5. Fix the main script for Gadi (e.g. 'exp/a10_hist/a10_hist')
  
 
A few edits are required throughout the script:
 
A few edits are required throughout the script:
Line 78: Line 80:
 
export SHORTDIR="/scratch/$PROJECT/$USER"
 
export SHORTDIR="/scratch/$PROJECT/$USER"
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
6. Perform a test run
 +
 +
Change the run length settings in the init script (e.g. 'exp/a10_hist/a10_hist.init') to perform a 10 model day run:
 +
<syntaxhighlight lang=bash>
 +
#-- Initial and Final Date of the Experiment
 +
iniyear=1951;  finalyear=1951;        typeset -Z4 iniyear  finalyear
 +
inimonth=1;    finalmonth=1;          typeset -Z2 inimonth finalmonth
 +
iniday=1;      finalday=10;            typeset -Z2 iniday  finalday
 +
 +
inidate=${iniyear}${inimonth}${iniday}                  #YYYYMMDD
 +
finaldate=${finalyear}${finalmonth}${finalday}          #YYYYMMDD
 +
 +
#-- Duration of a run/chunk (HERE is most often visited for test/short runs!)
 +
nyear=0                # number of years
 +
nmonth=0                # number of months
 +
nday=10                # number of days
 +
</syntaxhighlight>
 +
 +
Then qsub the main script (e.g. 'exp/a10_hist/a10_hist')
 +
 +
Raw model output gets sent to <code>$SHORTDIR/archive/history/$EXPT</code> (e.g. /short/w35/saw562/archive/history/a10_hist)
 +
 +
MOM, CICE & Oasis Model logs and namelists are in <code>$SHORTDIR/work/um_coupled/$EXPT</code>
 +
 +
UM logs are in <code>$SHORTDIR/data/um_coupled/$EXPT</code>
  
 
=Model Configurations=
 
=Model Configurations=

Revision as of 17:53, 2 February 2020

Template:Stub This is a stub page and needs expansion

Model Name

ACCESS 1.0 Coupled Model as used in CMIP5. Includes UM 7.3 atmosphere, MOSES land surface, CICE sea ice, MOM 4 ocean coupled using Oasis3

The model is not available in the UMUI, instead it is run from a set of shell scripts.

A guide to setting up and running the model on the old supercomputer Raijin can be found at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin

Running on Gadi

NOTE: Consider using ACCESS-ESM 1.5 for new runs on Gadi, it is much simpler to use

The component models of ACCESS-CM 1.0 need to be rebuilt in order to run on Gadi, and there are few modifications required to the run script. The instructions here are for the historical experiment mentioned at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin, you may need to modify them if running a different experiment.

1. Build the models

Follow the instructions at https://github.com/coecms/access-cm10-build-gadi to download the source code and build the model

2. Change the executable paths in your *.init file to point to the newly built model executables (note the timestamp in the file name will need to be changed)

cp -f $ACCESS_BINDIR/oasis3_MPI1.20200130.RJ         $cpl_exe
cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20200130.VAYU     $ocn_exe
cp -f $ACCESS_BINDIR/cice_MPI1.VAYU_6p_20200130      $ice_exe
cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe-20200131        $atm_exe

3. Set the recommended PBS headers for Gadi in your main run script (e.g. exp/a10_hist/a10_hist)

#PBS -q normal
#PBS -l ncpus=144
#PBS -l mem=192gb
#PBS -l walltime=00:10:00
#PBS -l wd
#PBS -W umask=0022
#PBS -l storage=scratch/access+gdata/access

4. Set the recommended Decomposition (for a total of 144 cpus, 384 gb memory) by modifying variables in the main and init scripts (e.g. exp/a10_hist/a10_hist, exp/a10_hist/a10_hist.init)

Component Variable Value
UM UM_ATM_NPROCX 12
UM UM_ATM_NPROCY 8
MOM oce_nx 8
MOM oce_ny 5
MOM nproc_ocn 40 (oce_nx * oce_ny)
CICE nproc_ice 6

5. Fix the main script for Gadi (e.g. 'exp/a10_hist/a10_hist')

A few edits are required throughout the script:

# Should point to the 'bin' directory from building the model
ACCESS_BINDIR=$PWD/../../bin

# Also load openmpi here
module load openmpi/4.0.2

# ...

# Replace the square brackets and their contents with 'true'
# if [[ "$PBS_O_HOST" == "raijin*" || "$PBS_O_HOST" == "r"*{0,2}(\d) ]] ; then
if true; then
         export exec_site="NCI"
         export exec_system="raijin"

# ...

# Change '/short' to '/scratch'
export SHORTDIR="/scratch/$PROJECT/$USER"

6. Perform a test run

Change the run length settings in the init script (e.g. 'exp/a10_hist/a10_hist.init') to perform a 10 model day run:

#-- Initial and Final Date of the Experiment
iniyear=1951;   finalyear=1951;         typeset -Z4 iniyear  finalyear
inimonth=1;     finalmonth=1;           typeset -Z2 inimonth finalmonth
iniday=1;       finalday=10;            typeset -Z2 iniday   finalday

inidate=${iniyear}${inimonth}${iniday}                  #YYYYMMDD
finaldate=${finalyear}${finalmonth}${finalday}          #YYYYMMDD

#-- Duration of a run/chunk (HERE is most often visited for test/short runs!)
nyear=0                 # number of years
nmonth=0                # number of months
nday=10                 # number of days

Then qsub the main script (e.g. 'exp/a10_hist/a10_hist')

Raw model output gets sent to $SHORTDIR/archive/history/$EXPT (e.g. /short/w35/saw562/archive/history/a10_hist)

MOM, CICE & Oasis Model logs and namelists are in $SHORTDIR/work/um_coupled/$EXPT

UM logs are in $SHORTDIR/data/um_coupled/$EXPT

Model Configurations