Difference between revisions of "ACCESS1.0 CMIP"
(8 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
+ | |||
ACCESS 1.0 Coupled Model as used in CMIP5. Includes UM 7.3 atmosphere, MOSES land surface, CICE sea ice, MOM 4 ocean coupled using Oasis3 | ACCESS 1.0 Coupled Model as used in CMIP5. Includes UM 7.3 atmosphere, MOSES land surface, CICE sea ice, MOM 4 ocean coupled using Oasis3 | ||
The model is not available in the UMUI, instead it is run from a set of shell scripts. | The model is not available in the UMUI, instead it is run from a set of shell scripts. | ||
− | A guide to setting up and running the model on the old supercomputer Raijin can be found at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin | + | A guide to setting up and running the model on the old supercomputer Raijin can be found at [https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin] |
== Running on Gadi == | == Running on Gadi == | ||
− | '''NOTE''': ACCESS-CM1 is only maintained for historical reference. New work should use [[ACCESS-ESM 1.5]], | + | '''NOTE''': <font color="red">ACCESS-CM1 is only maintained for historical reference and is '''no longer supported''' (existing projects excepted). New work should use [[ACCESS-ESM_1.5|ACCESS-ESM 1.5]] or ACCESS-CM2, those are much simpler to use and supported. If you really need to use ACCESS-CM1, please contact [mailto:c.carouge@unsw.edu.au Claire Carouge] before starting your project to assess your needs.</font> |
− | The component models of ACCESS-CM 1.0 need to be rebuilt in order to run on Gadi, and there are few modifications required to the run script. The instructions here are for the historical experiment mentioned at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin, you may need to modify them if running a different experiment. | + | The component models of ACCESS-CM 1.0 need to be rebuilt in order to run on Gadi, and there are few modifications required to the run script. The instructions here are for the historical experiment mentioned at [https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin], you may need to modify them if running a different experiment. |
On Gadi the model should take approximately '''1,400 SU per model year''', and run at a speed of about '''5 model years per day'''. | On Gadi the model should take approximately '''1,400 SU per model year''', and run at a speed of about '''5 model years per day'''. | ||
Line 15: | Line 16: | ||
1. '''Build the models''' | 1. '''Build the models''' | ||
− | Follow the instructions at https://github.com/coecms/access-cm10-build-gadi to download the source code and build the model | + | Follow the instructions at [https://github.com/coecms/access-cm10-build-gadi https://github.com/coecms/access-cm10-build-gadi] to download the source code and build the model |
− | 2. '''Change the executable paths in your | + | 2. '''Change the executable paths in your init file to point to the newly built model executables''' (e.g. exp/a10_hist/a10_hist.init) |
− | <syntaxhighlight lang= | + | |
− | cp -f $ACCESS_BINDIR/oasis3_MPI1. | + | Note the newly built executables have a timestamp in the file name that will need to be changed, e.g. edit the lines |
− | cp -f $ACCESS_BINDIR/mom4_MPI1.exe. | + | <syntaxhighlight lang="unknown"> |
− | cp -f $ACCESS_BINDIR/cice_MPI1. | + | cp -f ~access/apps/oasis3/3/bin/oasis3.MPI1.x $cpl_exe |
− | cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe- | + | cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20131017.RAIJIN $ocn_exe |
+ | cp -f $ACCESS_BINDIR/cice_MPI1.RAIJIN_6p_20131028 $ice_exe | ||
+ | cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe-20131017 $atm_exe | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | to instead say (this happens two times in the script): | ||
+ | <syntaxhighlight lang="bash"> | ||
+ | EXE_TIMESTAMP=20200204 | ||
+ | cp -f $ACCESS_BINDIR/oasis3_MPI1.${EXE_TIMESTAMP}.RJ $cpl_exe | ||
+ | cp -f $ACCESS_BINDIR/mom4_MPI1.exe.${EXE_TIMESTAMP}.VAYU $ocn_exe | ||
+ | cp -f $ACCESS_BINDIR/cice_MPI1.VAYU_6p_${EXE_TIMESTAMP} $ice_exe | ||
+ | cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe-${EXE_TIMESTAMP} $atm_exe | ||
</syntaxhighlight> | </syntaxhighlight> | ||
3. '''Set the recommended PBS headers for Gadi in your main run script''' (e.g. exp/a10_hist/a10_hist) | 3. '''Set the recommended PBS headers for Gadi in your main run script''' (e.g. exp/a10_hist/a10_hist) | ||
− | <syntaxhighlight lang= | + | <syntaxhighlight lang="bash"> |
#PBS -q normal | #PBS -q normal | ||
#PBS -l ncpus=144 | #PBS -l ncpus=144 | ||
Line 36: | Line 48: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
− | 4. '''Set the recommended Decomposition (for a total of 144 cpus, | + | 4. '''Set the recommended Decomposition (for a total of 144 cpus, 192 gb memory) by modifying variables in the main and init scripts''' (e.g. exp/a10_hist/a10_hist, exp/a10_hist/a10_hist.init) |
+ | |||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
− | ! Component | + | ! Component |
+ | ! Variable | ||
+ | ! Value | ||
|- | |- | ||
− | | UM | + | | UM |
+ | | UM_ATM_NPROCX | ||
+ | | 12 | ||
|- | |- | ||
− | | UM | + | | UM |
+ | | UM_ATM_NPROCY | ||
+ | | 8 | ||
|- | |- | ||
− | | MOM | + | | MOM |
+ | | oce_nx | ||
+ | | 8 | ||
|- | |- | ||
− | | MOM | + | | MOM |
+ | | oce_ny | ||
+ | | 5 | ||
|- | |- | ||
− | | MOM | + | | MOM |
+ | | nproc_ocn | ||
+ | | 40 (oce_nx * oce_ny) | ||
|- | |- | ||
− | | CICE | + | | CICE |
+ | | nproc_ice | ||
+ | | 6 | ||
|} | |} | ||
Line 58: | Line 85: | ||
A few edits are required throughout the script: | A few edits are required throughout the script: | ||
− | <syntaxhighlight lang=bash> | + | |
+ | <syntaxhighlight lang="bash"> | ||
# Should point to the 'bin' directory from building the model | # Should point to the 'bin' directory from building the model | ||
ACCESS_BINDIR=$PWD/../../bin | ACCESS_BINDIR=$PWD/../../bin | ||
Line 71: | Line 99: | ||
if true; then | if true; then | ||
export exec_site="NCI" | export exec_site="NCI" | ||
− | export exec_system=" | + | export exec_system="gadi" |
# ... | # ... | ||
Line 77: | Line 105: | ||
# Change '/short' to '/scratch' | # Change '/short' to '/scratch' | ||
export SHORTDIR="/scratch/$PROJECT/$USER" | export SHORTDIR="/scratch/$PROJECT/$USER" | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Also in the .fin file (e.g. 'exp/a10_hist/a10_hist.fin') change 'mppnccombine.RAIJIN' to 'mppnccombine.VAYU' in the 'archjob' line | ||
+ | <syntaxhighlight lang=bash> | ||
+ | archjob=`qsub -v PROJECT,sdir=$sdir,tdir=$tdir,idate=${enddate},mppncombine_exec=$ACCESS_BINDIR/mppnccombine.VAYU $ACCESS_BINDIR/do_mppncombine-cmip5.ksh` | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 82: | Line 115: | ||
Change the run length settings in the init script (e.g. 'exp/a10_hist/a10_hist.init') to perform a 1 model month run: | Change the run length settings in the init script (e.g. 'exp/a10_hist/a10_hist.init') to perform a 1 model month run: | ||
− | <syntaxhighlight lang=bash> | + | <syntaxhighlight lang="bash"> |
#-- Initial and Final Date of the Experiment | #-- Initial and Final Date of the Experiment | ||
iniyear=1951; finalyear=1951; typeset -Z4 iniyear finalyear | iniyear=1951; finalyear=1951; typeset -Z4 iniyear finalyear | ||
Line 107: | Line 140: | ||
This should take approx 30 minutes to complete | This should take approx 30 minutes to complete | ||
− | =Model Configurations= | + | = Model Configurations = |
− | * [[ | + | *[[ACCESS-CM_1.0|ACCESS-CM 1.0]] |
− | * [[ACCESS1. | + | *[[ACCESS1.3_CMIP|ACCESS-CM 1.3]] |
− | [[Category:ACCESS 1.0]][[Category:CMIP5 models and configuration]][[Category: Coupled Models]] | + | [[Category:ACCESS 1.0]] [[Category:CMIP5 models and configuration]] [[Category:Coupled Models]] |
Latest revision as of 20:14, 11 February 2020
ACCESS 1.0 Coupled Model as used in CMIP5. Includes UM 7.3 atmosphere, MOSES land surface, CICE sea ice, MOM 4 ocean coupled using Oasis3
The model is not available in the UMUI, instead it is run from a set of shell scripts.
A guide to setting up and running the model on the old supercomputer Raijin can be found at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin
Running on Gadi
NOTE: ACCESS-CM1 is only maintained for historical reference and is no longer supported (existing projects excepted). New work should use ACCESS-ESM 1.5 or ACCESS-CM2, those are much simpler to use and supported. If you really need to use ACCESS-CM1, please contact Claire Carouge before starting your project to assess your needs.
The component models of ACCESS-CM 1.0 need to be rebuilt in order to run on Gadi, and there are few modifications required to the run script. The instructions here are for the historical experiment mentioned at https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin, you may need to modify them if running a different experiment.
On Gadi the model should take approximately 1,400 SU per model year, and run at a speed of about 5 model years per day.
1. Build the models
Follow the instructions at https://github.com/coecms/access-cm10-build-gadi to download the source code and build the model
2. Change the executable paths in your init file to point to the newly built model executables (e.g. exp/a10_hist/a10_hist.init)
Note the newly built executables have a timestamp in the file name that will need to be changed, e.g. edit the lines
cp -f ~access/apps/oasis3/3/bin/oasis3.MPI1.x $cpl_exe
cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20131017.RAIJIN $ocn_exe
cp -f $ACCESS_BINDIR/cice_MPI1.RAIJIN_6p_20131028 $ice_exe
cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe-20131017 $atm_exe
to instead say (this happens two times in the script):
EXE_TIMESTAMP=20200204
cp -f $ACCESS_BINDIR/oasis3_MPI1.${EXE_TIMESTAMP}.RJ $cpl_exe
cp -f $ACCESS_BINDIR/mom4_MPI1.exe.${EXE_TIMESTAMP}.VAYU $ocn_exe
cp -f $ACCESS_BINDIR/cice_MPI1.VAYU_6p_${EXE_TIMESTAMP} $ice_exe
cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe-${EXE_TIMESTAMP} $atm_exe
3. Set the recommended PBS headers for Gadi in your main run script (e.g. exp/a10_hist/a10_hist)
#PBS -q normal
#PBS -l ncpus=144
#PBS -l mem=192gb
#PBS -l walltime=00:30:00
#PBS -l wd
#PBS -W umask=0022
#PBS -l storage=scratch/access+gdata/access
4. Set the recommended Decomposition (for a total of 144 cpus, 192 gb memory) by modifying variables in the main and init scripts (e.g. exp/a10_hist/a10_hist, exp/a10_hist/a10_hist.init)
Component | Variable | Value |
---|---|---|
UM | UM_ATM_NPROCX | 12 |
UM | UM_ATM_NPROCY | 8 |
MOM | oce_nx | 8 |
MOM | oce_ny | 5 |
MOM | nproc_ocn | 40 (oce_nx * oce_ny) |
CICE | nproc_ice | 6 |
5. Fix the main script for Gadi (e.g. 'exp/a10_hist/a10_hist')
A few edits are required throughout the script:
# Should point to the 'bin' directory from building the model
ACCESS_BINDIR=$PWD/../../bin
# Also load openmpi here
module load openmpi/4.0.2
# ...
# Replace the square brackets and their contents with 'true'
# if [[ "$PBS_O_HOST" == "raijin*" || "$PBS_O_HOST" == "r"*{0,2}(\d) ]] ; then
if true; then
export exec_site="NCI"
export exec_system="gadi"
# ...
# Change '/short' to '/scratch'
export SHORTDIR="/scratch/$PROJECT/$USER"
Also in the .fin file (e.g. 'exp/a10_hist/a10_hist.fin') change 'mppnccombine.RAIJIN' to 'mppnccombine.VAYU' in the 'archjob' line
archjob=`qsub -v PROJECT,sdir=$sdir,tdir=$tdir,idate=${enddate},mppncombine_exec=$ACCESS_BINDIR/mppnccombine.VAYU $ACCESS_BINDIR/do_mppncombine-cmip5.ksh`
6. Perform a test run
Change the run length settings in the init script (e.g. 'exp/a10_hist/a10_hist.init') to perform a 1 model month run:
#-- Initial and Final Date of the Experiment
iniyear=1951; finalyear=1951; typeset -Z4 iniyear finalyear
inimonth=1; finalmonth=1; typeset -Z2 inimonth finalmonth
iniday=1; finalday=31; typeset -Z2 iniday finalday
inidate=${iniyear}${inimonth}${iniday} #YYYYMMDD
finaldate=${finalyear}${finalmonth}${finalday} #YYYYMMDD
#-- Duration of a run/chunk (HERE is most often visited for test/short runs!)
nyear=0 # number of years
nmonth=1 # number of months
nday=0 # number of days
Then qsub the main script (e.g. 'exp/a10_hist/a10_hist')
Raw model output gets sent to $SHORTDIR/archive/history/$EXPT
(e.g. /short/w35/saw562/archive/history/a10_hist)
MOM, CICE & Oasis Model logs and namelists are in $SHORTDIR/work/um_coupled/$EXPT
UM logs are in $SHORTDIR/data/um_coupled/$EXPT
This should take approx 30 minutes to complete