ACCESS-S

Revision as of 23:13, 27 November 2019 by C.carouge (talk | contribs)
Template:Unsupported: CLEX CMS does not ordinarily provide support for this configuration, either due to its age or it being outside the scope of the Centre's research. Please contact us before starting any major work with this configuration

ACCESS-S

This is an as-of-yet unlisted document to chart my progress with ACCESS-S. Once it's running, hopefully this will make it easier to convert it into a document.

Getting ACCESS-S

This is the initial email I got from Hailin:

Hi Holger,

If you'd like to play the ACCESS-S1 suite, you can copy my suite au-aa563.

The followings are the key parameters to run the suite.

In rose-suite.conf:
MAKE_BUILDS=true               #set true to compile source codes
N_GSHC_MEMBERS=3          #num of ensemble members, as for MEMBERS=-m in app/glosea_init_cntl_file/rose-app.conf
N_GSHC_STEPS=2                  #number of RESUBMIT (number of chunk runs)
RESUB_DAYS=1                     #number of days per chunk run

In app/glosea_init_cntl_file/rose-app.conf:
GS_HCST_START_DATE=1990050100   #start date, it is 01 of May in this case
MEMBERS=-m 3                   #total number of ensembles, must be the same as the N_GSHC_MEMBERS in rose-suite.conf
GS_YEAR_LIST=1997            #the year of the run

After you compiled the codes and run the job successfully, you could maintain your own INSTALL_DIR which is defined in suite.rc:
INSTALL_DIR = "/short/dx2/hxy599/gc2-install"

If you have any problems please let me know.

Regards,
Hailin

So I made a copy of that, new rose is au-aa566, most of the things were already set to the values that Hailin initiated in his email. I've changed the INSTALL_DIR}} to {{/short/${PROJECT}/${USER}/gc2-install}} but I'm also not a member of the group Template:Dx2 or Template:Ub7, so I'm also trying to copy the Template:DUMP DIR and Template:DUMP DIR BOM directories to my {{/short/${PROJECT}/${USER}/dump}} and {{/short/${PROJECT}/${USER}/dump-bom, respectively, but there is 27TB of data, and I can't do that.

Getting ACCESS-S to run

I've copied the job, and just tried to run it, but it failed with error messages, culminating in Illegal item: [scheduling]initial cycle time

The solution to this is to use older versions of CYLC and ROSE with this command:

?'"`UNIQ--syntaxhighlight-00000002-QINU`"'?

First hurdles:

  1. gsfc_get_analysis gets a submit-failed
  2. GSHC_M1-3 get failed

For now, I've reset the suite.rc to point to the BoM directories, to see whether that changes anything -- It didn't

Looking at the job activity log and the job itself of gsfc_get_analysis}}, I notice strange PBS directives: Template:ConsumableMemory(2GB) and Template:Wall clock limit. I find these strings in suite.rc, and replace them with Template:-l vmem=2GB and Template:-l walltime=01:11:00. (I also find another reference to these Values for {{glosea_joi_prods, and change them as well.)

This seems to have succeeded for gsfc_get_analysis}}, but the {{GSHC_M1-3 still fail. I found this error message:

?'"`UNIQ--syntaxhighlight-00000003-QINU`"'?

It seems in the namelist entered, there's a value for control_resubmit}}, which the UM doesn't understand. Since rose considers this variable to be compulsory, I've had to remove it from the file {{~roses/au-aa566/app/coupled/rose-app.conf, and now I've submitted it again. (Or I could have disabled all metadata from the menu option...)

Second issues:

gsfc_get_analysis fails at the end, but it seems that it's not doing all that much:

?'"`UNIQ--syntaxhighlight-00000004-QINU`"'?

CPU time used is only 3 seconds, while it ran out of walltime after almost 1h15m.

So it seems that, since SUITE_TYPE}} is set to Template:Research (and thereby {{GS_SUITE_TYPE is also research, some environment variables are set to directories that might exist on the MetOffice computer, but not on raijin:

?'"`UNIQ--syntaxhighlight-00000005-QINU`"'?

For now I replaced the else clause above with the same data from as the original and try again.

Full Reset

Scott noticed that there were some new changes to the configuration file, namely RUN_GSFC and RUN_GSMN were set to true.

Since I couldn't remember ever changing them, I just made a full reset, changed and only changed the project.


Take 2

Zoe Gillett needs to run ACCESS-S1, and Angus "Gus" Gray-Weale helped us set up a new version of ACCESS-S1. He copied his version of ACCESS-S1, which turned out to be under git version control, which was copied over together with his configuration.

New repo

I've created a new repository on MOSRS for the model. It's u-bk374.

Work done

I added a script set_versions.sh to set the versions of rose and cylc, similarly to what's discussed above. It has to be sourced, not run, because it's setting environment variables.

Gus also recommended we become members of his group qa11 to get read access to specific ancillary files on /short/qa11/aag562/group/as specifically for testing purposes.

The main configuration file is rose-suite.conf:

 

[env]

[jinja2:suite.rc]
ACCESS_S_FOLDER="/short/qa11/aag562/group/as"
#ENS_NO=1
MAKE_BUILDS=true
N_GSHC_MEMBERS=3
N_GSHC_STEPS=2
RESUB_DAYS=1
RSTYR=2010
RSTMN=01
RSTDY=01
#------------
ARCHIVE_FCST_ANALYSES=false
ARCHIVE_TO_MASSR=false
#DISABLE_RESERVATIONS=true
START_CYCLE="2014071500"
END_CYCLE="2014071500"
EXTRACT_HOST='$ROSE_ORIG_HOST'
#GET_ANALY=false
GS_PS_NUM=34
HCST_CNTL_FILE_PATH='\${ROSE_SUITE_DIR}/hcst_control_file.txt'
HPC_HOST='raijin'
MN_OFFSET=30
NUM_RETRIES=1
#N_GSFC_MEMBERS=1
#N_GSFC_STEPS=2
#N_GSMN_MEMBERS=1
#N_GSMN_STEPS=4
PROD_FILT="product_filter.cfg"
#PROD_RUN=false
#RUN_GSFC=false
#RUN_GSHC=true
#RUN_GSMN=false
SEND_TO_DART=false
#SUITE_TYPE='research'
UM_STEPS_PER_DAY=96

Of specific interest are the lines:

Key Description
MAKE_BUILDS Whether to create the binaries
N_GSHC_MEMBERS Number of ensemble members
ACCESS_S_FOLDER Where to find the ancillries
START_CYCLE Start date
END_CYCLE End date