DataMover

Revision as of 22:39, 4 December 2019 by C.carouge (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Template:Consider Delete This information is especially outdated, and we're considering deleting it

Forword

This script dataMover has been used for CMIP5 simulations that the CoE has been running on the VAYU cluster at NCI. It basically does below:

(a) extracting the fields 91(Temperature at 1.5m), 158 (total precipitation rate) and 192(Pressure at mean sea level) from monthly historical output before moving model outputs to MDSS and write them to a netcdf file (using convsh)

(b) archiving, compressing and uploading PBS job outputs (*.o* and *.e*) to MDSS;

(c) archiving, compressing and uploading model outputs to MDSS;

(d) completing processing (a), (b) and (c) on a daily basis like system Cron job.

How to get the scripts

svn co https://trac.nci.org.au/svn/access_tools/cmip5_utils/trunk

This includes two scripts dataMover and convert. The script dataMover will call the script convert to do (a) mentioned above

Note that if you got the error "403 Forbidden" and then need email [| climate help desk] for access permission.

How to modify this script to meet your needs

If you need to use this script for other UM outputs other than CMIP5 (CoE has been running) , a few places should be changed, such as NCI project name, output directory, etc.

Note that you need to have the allocated storage space on MDSS (at least make sure that the size should be larger than the estimated size of all model outputs and restart files (if needed).

Any questions, please email [| climate_help on NCI] and you are welcome to discuss.