climate-cms wikis.unsw.edu.au - User contributions [en]

File:Dev-vabga.png

2019-01-27T15:15:23Z

ClaireCarouge:

File:Dev-vabg.png

2019-01-27T15:15:16Z

ClaireCarouge:

File:Devumui.png

2019-01-27T15:15:10Z

ClaireCarouge:

File:NU-WRF workflow schematic 2.jpg

2019-01-27T15:12:00Z

ClaireCarouge:

File:LVT usersguide.pdf

2019-01-27T15:09:59Z

ClaireCarouge:

File:LIS usersguide.pdf

2019-01-27T15:09:52Z

ClaireCarouge:

File:LDT usersguide.pdf

2019-01-27T15:09:45Z

ClaireCarouge:

File:Inputs3.txt

2019-01-27T15:07:28Z

ClaireCarouge:

File:Inputs2.txt

2019-01-27T15:07:17Z

ClaireCarouge:

File:Inputs.txt

2019-01-27T15:07:07Z

ClaireCarouge:

Admin

2019-01-17T05:46:48Z

ClaireCarouge: Imported from Wikispaces

=Organisation=
==Research programs (RPs) and RP leads==
{|
| Research Program || RP leads || CMS person
|-
| Climate Variability || Andy Hogg, Mat England, Dietmar Dommenget, Neil Holbrooke || Aidan Heerdegen, Holger Wolff
|-
| Heatwaves and Cold air outbreaks || Michael Reeder, Jason Evans || Paola Petrelli
|-
| Rainfall Extremes || Todd Lane, || Scott Wales
|-
| Drought || Mike Roderick, Andy Pitman (?) || Danny Eisenberg
|-
|}
==CIs==
Andy Pitman: Director
Todd Lane: Vice-director
Christian Jakob
Michael Reeder
Dietmar Dommenget
Steven Sherwood
Lisa Alexander
Julie Arblaster
Nerilie Abram
Gab Abramowitz
Nathaniel Bindoff
Jason Evans
Andy Hogg
Peter Strutton
----
=Policies=
==Blogposts==
The roster for the blogposts is:
* Claire - start week of 03/08
* Holger - start week of 10/08
* Paola - start week of 17/08
* Aidan - start week of 24/08
* Danny - start week of 31/08
* Scott - start week of 7/09
Follow instructions on this Trello card: [[https://trello.com/c/sWrPyxAv/24-readme-how-do-i-use-this-list]]
----
==News==
To communicate new work:
# update all documentation
# advertise on [[news | CMS news]] page
----
==Help desk==
There is now a roster for the help desk:
{|
| Monday || Aidan
|-
| Tuesday || Scott
|-
| Wednesday || Holger
|-
| Thursday || Claire / Danny
|-
| Friday || Paola
|-
|}
Main tasks on "help desk day" are:
# Helpdesk
# Documentation / training update
# Other work
----
==Priorities==
We need to update our priorities regularly (~3 months). Update on Trello (and/or spreadsheet?) in consultation with the Infrastructure Committee. Communicate to CIs.

----
==NCI Projects husbandry==
# Every 6 months, send a list of members for each CoE projects to CIs. It's a reminder to clean up the memberships.
# Everyone is in charge of caring for one or several projects. See tables below.
# Need to keep track every week or so of the storage and SU usage (use ncimonitor)
# '''Contact users''' if people are on track to exceed '''storage'''
# '''Contact Aidan''' if people are on track to exceed '''SU'''.
=NCI Projects=
'''FlagshipC:'''

{|
! Project !! Lead CI !! CIs !! Description !! Carer
|-
| w48 || Dietmar Dommenget || || Global scale decadal climate variability in a ACCESS hierarchy of climate models || Holger
|-
| w42 || Abhnil Prasad || || The effects of tropical convection on Australia's climate || Claire
|-
| v45 || Andy Hogg || || Mechanisms and attribution of past and future ocean circulation change || Aidan
|-
| w97 || Jason Evans || Lisa Alexander, Andrew King || Mechanisms and attribution of changes in Australian climate extremes || Scott
|-
| w40 || Todd Lane || || Atmospheric and oceanic processes and dynamics || Scott
|-
| w58 || Michael Reeder (superscience) || || The dynamics of subtropical anticyclones and the connection to drought, heatwaves and bushfires in southern Australia || Holger
|-
| w35 || Claire Carouge || || Terrestrial modelling within the Centre of Excellence regionalizing land surface processes || Danny
|-
| access || Claire Carouge or Scott Wales || || Group to run the ACCESS model and get access to some additional modules (useful for Python). ||
|-
|}
==Other CoE-Related:==
===Compute===
{|
| dt6 || Andy Pitman || Land stuff
|-
|}

===Storage===
{|
! Project !! Lead CI !! Description !! Carer
|-
| ua8 || Paola Petrelli || ARCCSS and CLEX data || Paola
|-
| ub4 || Paola Petrelli || Era-Interim Data || Paola
|-
| rr3 || Kate Snow || ESGF CMIP5 Australian Data Publication || Paola
|-
| al33 || Kate Snow || ESGF CMIP5 Data Replication || Paola
|-
| oi10 || Kate Snow || ESGF CMIP6 Data Replication || Paola
|-
| p66 || Tony Hirst || ACCESS GCM ||
|-
| rq5 || Paola Petrelli || OFES || Paola
|-
| rq7 || Paola Petrelli || YOTC || Paola
|-
| rr4 || || BoM ACCESS NWP data ||
|-
| rr7 || Aurel Moise, Paola Petrelli || Reanalyses || Paola
|-
| wd9 || Paola Petrelli || CABLE evaluation datasets || Paola
|-
| hh5 || Claire Carouge || Climate LIEF storage grant || Paola
|-
|}

===Cloud projects===
{|
| fe2_2 || || ARCCSS cloud
|-
| ju7_1 || || CWSlab
|-
|}
==Requesting Met Office accounts for users==

New accounts need a sponsor from their institution
Sponsors are listed at https://code.metoffice.gov.uk/trac/admin/wiki/GroupSponsors, Current ARCCSS sponsors are Scott, Claire and Holger
Instructions for an account request are at https://code.metoffice.gov.uk/trac/admin/wiki/NewUserChecklist

In brief, create a ticket at https:''code.metoffice.gov.uk/trac/admin/newticket?type=account-request with the user's name and email address, like https:''code.metoffice.gov.uk/trac/admin/ticket/758
----
=Useful contacts=

{|
! DATA !!
|-
| Ben Evans || ben.evans@anu.edu.au
|-
| Claire Trenham ||
|-
| Clare Richards || clare.richards@anu.edu.au
|-
| Jon Smilie ||
|-
|}

{|
! User Support !!
|-
| Roger Edberg || roger.edberg@anu.edu.au
|-
| Brian Davis || brian.davis@anu.edu.au
|-
|}

{|
! ACCESS !! !!
|-
| Bob Pitt || || In charge of access-svn.nci.org.au. (e.g. certificate renewal etc)
|-
|}

=conda environment=

CMS maintains an anaconda installation on the NCI systems. To add new packages to the unstable analysis environments clone the [https://github.com/coecms/conda-envs | conda-envs repository]. The master branches contains nothing but a README. The magic happens in other branches. There are two branches, analysis27}} and{{analysis3, for the python2 and python3 environments.

Checkout an analysis branch, add the package name to environment.yml and then commit your change. If the package is to be added to both branches checkout the other branch and make the same change and commit.

Next push altered branches to the repository. If both branches have been updated push both at the same time:

<syntaxhighlight>
git push origin analysis27 analysis3
</syntaxhighlight>

Pushing the branches will trigger testing and updating on the jenkins instance. The status of the conda environment update is viewable here:

https://accessdev.nci.org.au/jenkins/blue/pipelines?search=conda

===Tips and tricks if problems with conda environments===
* Scott said: "for whatever reason (something to do with external c libraries I think) loading packages is sometimes order-dependent." If a package breaks the tests, put its import at the top of the list. It would then be loaded before whatever package is causing the issue.

NCI Guidelines gdata

2019-01-17T05:38:34Z

ClaireCarouge: Imported from Wikispaces

Gdata is a disk storage with high-speed access. It is meant to be used to store data files used in analysis. This disk is not backed up
The location is always /g/dataN with the N identifying physically different disks, all of the Centre’s projects are on /g/data1 or on /g/data3. This means that they are effectively separate filesystems so if you’re moving files from a project in /g/data1 to a project in /g/data3 for example, you are effectively copying the files to a new filesystem. If you are moving files between two projects on the same disk, the transfer will be basically immediate because their physical location won’t change.

===Proper usage===

Gdata is used for large collections that users access often, so it is appropriate for:

* Published or shared datasets: all the data projects such as ub4 (ERA-Interim), rr7 (Reanalysis products), rr3, al33 and oi10 (CMIP), ua8 (CoE Published Data) have disk storage allocated as they are large, and frequently accessed by a large number of users. It is also necessary for files that are served through web services via THREDDS to be stored on disk with fast access.
* Model output or other collection of files that a user is currently analysing and any intermediate files produced by that analysis.
* Software not otherwise available, although it is always best to check with the CMS and/or NCI if a shared installation is possible

===What should not be stored on gdata:===
* Your own code files: gdata is not backed up so it’s always better to have them in your home directory or you can have a local copy on gdata but use bitbucket or github as your original repository
* Gdata is not to store model logs and other standard output errors produced by software, unless you are analysing them. If you want to keep these files to have a future reference then you should archive (tar) them and move them to tape (MDSS)
*Any files that are no longer used after you finished to analyse your data you should “clean” it and archive it. If you think you might reuse the data in the future you should evaluate how likely this is and act accordingly. It’s very easy to leave the data there for the moment and never get back to it. Storage is becoming more and more scarce and this is by far the most expensive storage option.

===Good practices:===
* All netcdf files should be compressed
* Use group permissions wherever you can
* Organise your work space (READMEs, data management plan, etc).
* Clean up regularly

===Big files:===
* A big file is anything bigger than several GB.
* Avoid creating any file bigger than 20GB. If you really need to, you should use techniques like:
* Chunking (netcdf4): http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters
* Striping: https://opus.nci.org.au/display/Help/Lustre+Basics

[[include component="page" wikiName="climate-cms" page="nci_guidelines_index" editable="1"]]

News

2019-01-11T06:02:18Z

ClaireCarouge: Imported from Wikispaces

==='''WRF v4.0.2'''===
* This WRF version is now available on Raijin. Please refer to instructions [[WRF v4.0.2 installation | here]].
* The UNSW-CCRC-WRF code has also been updated to this version. Please see the V4.0.2 branch of [https://bitbucket.org/dargueso/unsw-ccrc-wrf/src/V4.0.2/ | the repository].
=='''2018-10-24'''==
==='''Python 2 end of support'''===
* Numpy has announced it will stop support for Python 2 from January 2019. This means numpy will still be available for Python 2 but the version for Python 2 will be frozen from then on. Following this announcement, several packages built upon numpy have announced the end of support for Python 2 as well. As such, we will stop updating our conda environments at NCI for Python 2 from January 2019 onwards. This means, the environments will still be available but no update will be made to them. We strongly advise all Python users to transition to Python 3.
===mppnccombine-fast===
* If you are running MOM, you may need to collate your outputs or even your restarts. There is a new mppnccombine availble, faster than the old one. Aidan has mentioned during [https://www.youtube.com/watch?v=p1Sm_NGOuPc&t=257s | the training on payu new features ]and he put the information on using it in [https://climate-cms.org/mom,/collate,/mpi/2018/10/19/mppnccombine-fast.html | this blogpost].
=='''2018-09-25'''==
==='''ERA5'''===
* We know accept some requests for downloading ERA5 data. Please refer to [[ERA5 | the ERA5 wiki page]] for more information on the specific conditions.
==='''CMEMS-sea level v4.0'''===
* There's a new version 4.0 available for CMEMS-sea level (previously known as AVISO). For more info see the relevant [[sealevel_GLO_PHY_L4_REP_observations_008_047 | wiki page]]
==='''JRA55 moving on Monday 13th August'''===
* We are going to move JRA55 (not JRA55-do!!!) 6hr and day data, from
>> /g/data1/rr7/ana4MIPs/reanalysis/JRA/JRA55 to
>> /g/data1/rr7/JRA55
===CMS Trainings===
* Some trainings were recorded. You can find them on [https://www.youtube.com/channel/UCSmoK6oWV9O0Hmyt9UdDNsQ | the CMS YouTube channel]
* 3D Visualisation - Martin Jucker, UNSW https://youtu.be/ya-xXY5uzeo
* Nectar Research Cloud - Just Berkhout, TPAC https://youtu.be/8jluqwOOS0U

=='''2018-08-01'''==
==='''New blog'''===
* We are starting a blog to share answers to queries we receive. The aim is to help newcomers get started. The blog is at this address: [[http://climate-cms.org/]]
===CMORPH moving===
* We are going to move CMORPH to a different project, from ua8 to rr7, this is to group all re-analysis that do not have specific licensing terms under one project. The new location will be /g/data/rr7/CMORPH/. More details will follow.
===Aerosol data from MACC reanalysis===
* A small subset of the Monitoring Atmospheric Composition and Climate reanalysis from ECMWF is now available in the ub4 project. We downloaded only aerosols variables for the entire period (2003-2012). See [[MACC | MACC page]] for details
===CMORPH update===
* Since there are delays in releasing new data for V1-0, we downloaded its precursor V0.x from July 2017 to date included. We also concatenated the daily data into monthly files. See [[CMORPH | CMORPH page]] for more information.
===Python package for selecting regions===
* We came across [http://regionmask.readthedocs.io/en/stable/index.html | this package]. It is designed to help select geographical regions either predefined or defined by the users. Let us know if you would be interested in it and we'll get it installed in the conda package.
==2018-05-14==
===GLEAM v3.2===

* updated versions of the GLEAM global datasets of terrestrial evaporation and root-zone soil moisture (GLEAM v3.2a and GLEAM v3.2b) are available in /g/data/wd9/BenchMarking/GLEAM_v3-2/

===UM performance with GA7===
* [[GA7 Performance | At this page]], you'll see a summary of the performance of the latest UM versions with GA7 configuration.
==2018-03-28==
===Publishing data on the NOAA/WDS Paleoclimatology archive===
* We have started listing our published paleoclimate datasets on the NOAA/WDS Paleoclimatology archive:https://www.ncdc.noaa.gov/paleo-search/study/23470
> If you would like to do the same for your data or you know of another discipline specific data repository please contact us on climate-help
===CESM - CCSM data===
* We moved the CESM1-CAM5-BGC-LE, CESM1-LME and CCSM dataset to ua6, in line with their licensing terms. We also re-organised the data in a CMIP-like structure. Work is still going on, as well new variables have been added. We will update the wiki asap.
===CMIP5 - CORDEX - CNRM5 6hrLev recall===
* There is an issue with CNRM5 historical 6hrLev, this can affect also CORDEX data for more information please refer to the [https://opus.nci.org.au/pages/viewpage.action?pageId=6324228 | CMIP confluence page]
===GSWP3 data===
* There is a new version for this dataset in the CABLE collection (wd9), the older version is considered obsolete now, it will be still available for a month in /g/data/ua8/GSWP3_obsolete and then deleted. We do not manage anymore this dataset so questions need to be directed to Bernard Pak (CSIRO).
===OSTIA data===
* We are stopping updates of this dataset since the users who requested originally do not need it anymore, if you are still using this data please contact us.
===Conda Updates===
The Conda environments will be updated to a new version on Tuesday 27th March. This will be a general update of all packages, please email climate_help if you have any issues with python scripts after the update

==2018-01-31==
===WRF===
* The Unified Post-Processing system (UPP) version 3.2 is now installed at NCI. It resides within the WRFV_3.9.1.1 directory but can be used with outputs from older WRF versions. For details on where to find it and how to compile it, refer to [[WRF#WRF-Post-processing%20software-UPP | this page]].
===Anaconda environment updates===
CMS are changing the way we manage our Anaconda environments at NCI (Raijin, VDI and accessdev). To date we've been updating the environments every week, to improve stability of the environments we're going to be creating a 'stable' version
of each environment which gets updated once per quarter.

How you use the environments will stay the same, you can still use:
<syntaxhighlight>
module use /g/data3/hh5/public/modules
module load conda/analysis3 # or conda/analysis27
</syntaxhighlight>
and get access to a wide variety of climate and weather focused Python libraries.

We'll still be installing new packages on request, you can ask for them by emailing us at climate_help@nci.org.au, but so that the 'stable' environments are not affected new packages will be installed into the environments 'analysis3-unstable' and 'analysis27-unstable'. During the quarterly updates all of the new packages added during the past quarter will be added to the 'stable' environments.

As the number of packages being installed into our Anaconda environments is getting quite large reducing the update frequency will allow us to more thouroughly test the interactions between the different packages and ensure we've got a stable platform for research.
===GPCP===
* Global Precipitation Climatology Project - daily precipitation analysis from surface and satellite measurements for 1997 onwards. Details in the [[GPCP | dataset page]] .
===GSMaP===
* Global Satellite Mapping Precipitation re-analysis-gauge product is available in ua8. Details in the [[GSMaP | dataset page]]
===MERRA-2 citation update===
* NASA has added DOIs and a citation for each of the MERRA-2 products. Details in the [[MERRA2 | dataset page]].
==2017-12-13==
==='''GIMMS AVHRR Global NDVI 1/12-degree'''===
* Update: added files for 2011, 2012 and 2013. Data is in /g/data/ua8/gimmsndvi3g/<YYYYs_new>/<files>
===UM 10.6 GA7.1 n96 'HighResMIP'===
* The Hadley Centre's CMIP6 HighResMIP model can now be run on Raijin at the lower n96 resolution using the Rose job u-as709 - http://climate-cms.unsw.wikispaces.net/UM+Experiments

==2017-11-30==
===ERA5===
* U,V and T operative analysis data for January 2016 is now available both as model level and pressure level (also Q) as grib and netcdf files. This data is provided as an example of ERA5. We'll be downloading also ENDA data for the same month. Data is in /g/data/ub4/era5-test/<format>/oper/<levels>/<files>

===New Wiki Pages===
* We've added a guide to how to use the grib_api module to work with GRIB format files: [[GRIB Format Files | http://climate-cms.unsw.wikispaces.net/GRIB+Format+Files]]
* Forcing files used for ACCESS CMIP5 runs: [[ACCESS CMIP5 Forcings | http://climate-cms.unsw.wikispaces.net/ACCESS+CMIP5+Forcings]]
* Information on NCI systems has been updated. We have also added some guidelines to using resources at NCI. [[NCI Guidelines | http://climate-cms.unsw.wikispaces.net/NCI+Guidelines]]

===Training material===
* All the material from the training day can be found [https://drive.google.com/open?id=0B1KncyYETiZqNGdHZ0VZSk02cUE | here].

===Unified Model Newsletter===
* The latest UM newsletter is available at https://code.metoffice.gov.uk/trac/jumps/raw-attachment/wiki/UMNewsletter/UMDec2017.pdf

==2017-09-12==
===WRF===
* WRF v3.9.1.1 has now been ported to NCI. See [[WRF v3.9.1.1 installation | here]].
===ACCESS Science Day===
* Claire and Scott presented talks on ESMValTool, ARCCSSive and higher resolution atmosphere models at the ACCESS Science Day, slides from all presentations are available at https://accessdev.nci.org.au/trac/wiki/ScienceDay

===CMIP5 clean up of extra copies===
* As announced to anyone in ua6 by e-mail, we have initiated a clean up of extra copies of CMIP5 data. The process and a list of the redundant directories that eventually will be removed are available on the NCI [https://opus.nci.org.au/display/CMIP/2017/08/22/Notice+of+ua6+data+migration+and+clean+up | confluence site]
===Updates to CMORPH data===
* We checked the entire datasets for errors in files, we also updated the collection that now has data up to July 2017 included
* The files for Sep 07 2003 and Jan 22 2014 were updated because they contained empty arrays.
===Downloading from Met Office Archives (MASS/MOOSE)===
* We are now able to download model outputs from the UK Met Office MASS archive system on request - [[Downloading from MASS]]

===Conda Updates===
* ===https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda===
** [https://github.com/ioos/compliance-checker | compliance-checker] IOOS compliance checker for validating CF-NetCDF and ACDD metadata
* There were some issues with broken packages installed in the shared conda environment from August 31st to September 4th. We apologise for any inconvenience and are working on a robust testing solution to ensure this does not happen again in the future
==2017-07-26==
===NOAA OISST===
* NOAA is changing the delivery mode of OISST and also switching to netcdf4 format. I'm currently reviewing our download procedure and we'll finally have a new raw and modified version for this data. To help to transition the v2-0_modifed version will stay in place.
===WRF v3.9===
* The compiling scripts for WRF and WPS have been revamped. They now accept command-line arguments for customisation rather than requiring users to hack into the script. See details on the [[WRF v3.9 installation | installation page for v3.9]]. If you would like more command-line options that's easy to add so please tell CMS about it.
===PyPI publishing example===
* A brief [[How to publish your Python code to PyPI | HOW-TO on publishing simple code to PyPI]] which will make it more visible, and easier to install and use.
===River runoff regridding tools===
* [https://github.com/OceansAus/runoff_tools | River runoff regridding tools for MOM] written by Russ Fiedler for COSIMA have been installed as module on Raijin and VDI. See [[River runoff regridding for MOM | here]] for an example of how they are used.
===JRA-55-do===
* Repeat Year Forcing (RYF) derivative datasets of the Japanese Reanalysis for Ocean Model forcing (JRA55-do) are available under the same directory as JRA55-do. In addition there are regridded runoff data for different MOM ocean configurations available in ua8. Contact climate_help@nci.org.au for more information.
===Conda Updates===
* ===https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda===
** [https://code.metoffice.gov.uk/doc/um/mule/2017.06.1 | mule] The UKMO library for reading and writing UM format files
** [https://www.ffmpeg.org | ffmpeg] Video and audio encoding library and utilities. Installed to support creating animations in matplotlib
** [https://github.com/pypa/twine | twine] Utility to support installing PyPI packages
** [https://github.com/pydap/pydap | pydap] Pydap is an implementation of the Opendap/DODS protocol

===STASH and Rose===
* How to copy STASH settings between suites with Rose - http://climate-cms.unsw.wikispaces.net/Running+the+UM+with+Rose#x-Copying%20STASH%20fields

==2017-06-06==
===ERA-Interim for WRF===
* Bug information: The information on how to use the ERA-Interim dataset in WPS on the wiki ([[How to run WRF with ERA-Interim data | here]]) and the automatic script [https://github.com/coecms/wps-era | wps-era] referenced on that page were erroneous. This code and instructions would potentially create erroneous fields along the coast lines. These instructions and script have now been updated. Please update your code and regenerate your WRF input files if you have used wps-era
===TRMM===
* Bug information: some files in the TRMM dataset at NCI got corrupted. The dataset has now been repaired and more stringent tests have been added to help avoid corrupted files in the future.
===ACCESS-MOSES with KPP===
* A version of ACCESS1.0 coupled to KPP using OASIS3-MCT was finalised. UMUI jobs vatae}} (N96 resolution) and {{vaski (N48) can be used as a basis. Further documentation in [[ACCESS-MOSES-KPP]].

==2017-05-19==
===CMIP induction and ARCCSSive training===
* We added to the NCI online training website a CMIP induction which includes training on how to use the ARCCSSive python module. To access the course use your NCI account to login [https://training.nci.org.au/course/view.php?id=26 | here]

===JRA55-do===

* JRA-55 based surface dataset for driving ocean-sea ice models, JRA55-do v 0.8 and v1.1 are now available in /g/data1/ua8/JRA55-do/v0-8 (v1-1). This dataset is covered by a license and it's not officially published yet, so you shouldn't re-distribute it. Please, follow Terms of Use available in the main directory.

===WRF v3.9===
* The latest version of WRF (v3.9) is now available on Raijin. Please find detailed instructions [[WRF v3.9 installation | here]]

===Conda Updates===
* ===https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda===
** [http://wrf-python.readthedocs.io/en/latest/index.html | wrf-python] A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model.
** [http://ferret.pmel.noaa.gov/Ferret/documentation/pyferret | pyferret] PyFerret is a Python module wrapping Ferret. The pyferret module provides Python functions so Python users can easily take advantage of the Ferret's abilities to retrieve, manipulate, visualize, and save data. (Python 2.7 only)
** [http://www.sympy.org/en/index.html | SymPy] SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python.

==2017-04-05==
===Data collections Updates===
* Expansion of MERRA-2 collection at NCI with the inst3d_3d_asm_Np data. [[MERRA2 | View MERRA-2 page]]

===UM10.6 GA7===
* The UK Met Office's Global Atmosphere 7 configuration is available on Raijin for UM 10.6 as job 'u-ak072' http://climate-cms.unsw.wikispaces.net/GlobalAtmosphere

===LROSE===
* NCAR's [https://www.eol.ucar.edu/content/lidar-radar-open-software-environment | Lidar Radar Open Software Environment (LROSE)] has been installed as a public module in /g/data3/hh5/public/modules. The commands to load the module are:
module use /g/data3/hh5/public/modules
module load lrose

===Conda Updates===
* ===Several packages have been added to the public Anaconda environments https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda===
** [https://unidata.github.io/MetPy/ | metpy]: MetPy is a collection of tools in Python for reading, visualizing and performing calculations with weather data.
** [http://ajdawson.github.io/windspharm/ | windspharm]: windspharm is a Python package for performing computations on global wind fields in spherical geometry.
** [https://code.zmaw.de/projects/cdo/wiki/Cdo%7Brbpy%7D | python-cdo]: Python bindings for CDO
** [http://arm-doe.github.io/pyart/ | pyart]: The Python ARM Radar Toolkit
** [http://bokeh.pydata.org/en/latest/ | bokeh]: Bokeh is a Python interactive visualization library that targets modern web browsers for presentation.

==2017-03-17==
===Python libraries at NCI===
* We've moved the Python libraries installed with Miniconda to a public space (/g/data3/hh5/public/modules) so there is no need anymore to be part of ACCESS to use it. See the [https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda | updated instructions]
===UM Newsletter===
* The latest newsletter is released: http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMNewsletterFeb2017
===Massdata utility===
* Are you moving data to/from massdata? Are you having issues with the copy stopping in the middle of things and impossibility to restart from where it failed? Aidan wrote a small utility to help with that. It’s available to anyone, info: [[https://github.com/coecms/mdssdiff]]
==2017-02-15==

===MPAS-Atmosphere===
* The latest release of [[MPAS | MPAS-Atmosphere]] has been installed on Raijin. This model will probably supplant WRF sometime in the future.

===Python libraries at NCI===
* We've installed some Python libraries using Miniconda in /g/data1/access. The aim is to facilitate access to and management of commonly used Python libraries on Raijin, Accessdev and VDI. It also enables users to locally install their own libraries using Miniconda without duplicating the centrally installed libraries. For the moment, users have to be part of the ACCESS group on Raijin to use these (if you are not, just ask access on my.nci.org.au) but we plan to make it public access. More details are [https://accessdev.nci.org.au/trac/wiki/User%20Guides/conda | here].

===Sharing facility at NCI===
* We worked at NCI to enable users to share data stored on Raijn with collaborators who don't have access to NCI. This is aimed for work in progress which can not be published and opened publicly. The files are accessible for a short time only to allow your collaborators to download them to their own storage. We are working on making it user-friendly but if you need to share some data in the meantime, please ask climate_help@nci.org.au.

===Updating OpenMPI library for MOM===
* After the new quarter maintenance, some MOM jobs experienced some stalling behaviour at the end of the job. This mainly affect large configurations (0.1°). The fix is to recompile MOM using the openmpi/10.2.1 version. If you don't know whether you should worry about it or how to recompile, please contact climate_help@nci.org.au.

===NU-WRF===
* The latest manual is now on the CMS wiki. Just follow the NU-WRF link in the navigation sidebar.

==2016-08-22==

===Configurations===
* ACCESS CM 1.0 pre-industrial control has been ported to work on Raijin https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin#piControl

===UM===
* The Accesscollab UMUI server is no longer supported, and we can't guarantee the safety of data on it. Please move any data on the 'Collab' page of the UMUI to the 'Accessdev' or 'Collab(local)' page. The 'Collab (local)' page contains copies of the Accescollab database from last year, if your jobs are present there no further action is required.
> To move an experiment folder right click on it and select 'copy' then go to the 'accessedev' or 'collab(local)' page and use the 'paste' button in the top right of umuix to copy all jobs in that folder.
* NIWA's um2netcdf.x conversion script is now available as ACCESS module um2netcdf-niwa https://github.com/adeytown/um2netcdf too

WRF

2018-11-16T04:33:07Z

ClaireCarouge: Imported from Wikispaces

=WRF=

==Overview==

The Weather Research and Forecasting (WRF) model is a next-generation mesoscale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs. It features multiple dynamical cores, a 3-dimensional variational (3DVAR) data assimilation system, and a software architecture allowing for computational parallelism and system extensibility. WRF is suitable for a broad spectrum of applications and across scales ranging from meters to thousands of kilometres.

If you have any corrections, comments or contributions, please feel free to email us on [[mailto:climate_help@nf.nci.org.au | climate_help@nci.org.au]] so that we can continue to update and upgrade the guides over time.

The installation (obtaining and compiling the source code) is NCI specific. The NCAR guides which we link to give instructions for compilation. DO NOT USE THOSE INSTRUCTIONS. They will not work. First, install WRF using this wiki's instructions and only follow the NCAR guides for running your simulation.

==Installation Guides:==
* [[WRF v4.0.2 installation]]
* [[WRF v3.9.1.1 installation]]
* [[WRF v3.9 installation]] * New installation instructions *
* [[WRF v3.8.1 installation]] * New installation instructions *
* [[WRF v3.7.1 installation]] * New installation instructions *
* [[WRF V3.6 installation]]
* [[WRF V3.5.1 Chem installation]]
* [[WRF V3.5.1 installation]]
* [[WRF V3.5 installation]]
* [[WRF V3.4.1 installation]]
* [[WRF V3.4 installation]]
* [[WRF V3.3.1 installation]]
* [[WRF installation | WRF V3.3 installation]]

==Running WRF at NCI==

[[How to run WRF]]

[[How to run WRF with ERA-Interim data]]

[[How to run WRF with inputs in netcdf format]]

[[WRFPLUS V3.4.1 installation]]

[[WRFDA V3.4.1 installation]]

[[WRF Workflow Tools]]

[[Tips and Tricks]]

==Datasets for WRF==
We keep an updated version of WPS Geog dataset on Raijin under /projects/WRF/data/WPS_GEOG
We also keep the data to run the January 2000 WRF tutorial under /projects/WRF/data/JAN00
We do not keep any other datasets specifically for WRF use. You may be able to use some of the [http://nci.org.au/data-collections/data-collections/ | RDSI datasets already stored at NCI] or you will have to bring your own dataset.

==Post-processing software==
===UPP===
The NCEP Unified Post-Processing system (UPP) v3.2 has been installed on Raijin. Although it can be used with several WRF versions, it is dependant on a compilation of WRFV3 and hence is installed in the WRF directory. It can be found under /projects/WRF/WRFV_3.9.1.1 and following versions. The simplest way to use it is to:
# Clone the WRFV_3.9.1.1 directory to your working area:
<syntaxhighlight>
git clone /projects/WRF/WRFV_3.9.1.1
</syntaxhighlight>
# Compile WRFV3 from WRFV_3.9.1.1 using the run_compile script under WRFV_3.9.1.1/WRFV3
# Compile UPP using the run_compile script under WRFV_3.9.1.1/UPPV3.2. Use the -h option to see the help.
You can then use UPP on your WRF outputs, no matter what WRF version was used to produce them (at the condition UPP v3.2 is compatible with this WRF version, see the NCAR documentation if you have any doubts).

UPP has been tested on the January 2000 tutorial outputs using the wrf_cntrl.parm file provided by default in UPP.

The CMS team will not provide support to users to run their own cases. Please refer to the [https://dtcenter.org/upp/users/docs/overview.php | UPP documentation] and [https://dtcenter.org/upp/users/support/index.php | user support].

**Related links:'''

[http://www.wrf-model.org | WRF Home Page]

[http://www.mmm.ucar.edu/wrf/users/docs/arw_v3.pdf | A Description of the Advanced Research WRF Version 3]

[[WRF tests]]

Process for [[WRF porting to NCI machine]]
[[Category:Main Menu]][[Category:wrf]]

WRF porting to NCI machine

2018-11-14T04:03:45Z

ClaireCarouge: Imported from Wikispaces

These are the steps to follow to install new WRF versions on NCI machine. Only for admins within the wrf group at NCI.
If starting from scratch:
# Update the Github fork of NCAR Github in coecms/WRFV3
# Under your own project, clone the WRF github repo: https://github.com/coecms/WRF and create a new branch for the new version (named Vx.x.x)
# Add https://github.com/coecms/WRFV3.git as a remote (named NCAR if you want to copy-paste the commands) and fetch
# In the new branch, run the command:
> <syntaxhighlight>
git merge -s recursive -Xsubtree=WRFV3 -Xtheirs --allow-unrelated-histories NCAR_WRF/master
</syntaxhighlight>
> You may need another branch than master to pick up a specific release.
# Clone into /projects/WRF with wrf group ownership. Make sure the master branch there points to the new version branch.

If you already have a clone of the coecms/WRF repo:
# Update the Github fork of NCAR Github in coecms/WRFV3
# In your local clone, fetch the NCAR remote
# Create a new branch for the new version (named Vx.x.x)
# In the new branch, run the command:
> <syntaxhighlight>
git merge -s recursive -Xsubtree=WRFV3 -Xtheirs --allow-unrelated-histories NCAR_WRF/master
</syntaxhighlight>
> You may need another branch than master to pick up a specific release.
# Clone into /projects/WRF with wrf group ownership. Make sure the master branch there points to the new version branch.

Follow same steps for WPS, but remote is called NCAR_WPS and -Xsubtree=WPS

==Jenkins setup==
# Add a new parameter for the version. Just the numbers, no "V".
# Create a branch for that version in wrf-testing repo. Exact same name as the previous step parameter.

How to run WRF with ERA-Interim data

2018-10-22T05:28:43Z

ClaireCarouge: Imported from Wikispaces

=Running WRF from ERA-Interim=

The ARCCSS CMS team helps to maintain a mirror of the ERA-Interim reanalysis data set at NCI, which can be used as the boundary conditions for a WRF run. The GRIB files may be found under /g/data1/ub4/erai/grib/, you will need to request access to the dataset first in order to access it.

You can use either Vtable.ERA-interim.ml}} or {{Vtable.ERA-interim.pl}} to ungrib the ERA-Interim data, using the grib files under directories {{oper_an_ml}} or {{oper_an_pl}} respectively. These Vtable files can be found in the WPS distribution under {{ungrib/Variable_Tables}}. You will also need to ungrib the files in directory {{oper_an_sfc to get the surface fields.

==Automatic WPS==

We have a script available at https://github.com/coecms/wps-era that will run all WPS steps automatically for you using the pressure level dataset. You can download the script with git:
<syntaxhighlight>
git clone https://github.com/coecms/wps-era
cd wps-era
</syntaxhighlight>

Edit the namelist.wps file as appropriate for your model, then run WPS with
<syntaxhighlight>
make WPSDIR=/path/to/WPS
</syntaxhighlight>
WPSDIR should be the path where you compiled WPS. The script will read the run dates from the namelist, and link in only the files required for the period you're using.

==Manual WPS==

===Time dependent data===

To run ungrib on both the surface and vertical data you can either create a temporary directory with both types of files, and run link_grib.csh on that directory:
<syntaxhighlight>
mkdir era-tmp
ln -s /g/data1/ub4/erai/grib/oper_an_pl/fullres/2006/ei_oper_an_pl_075x075_90N0E90S35925E_200603* era-tmp
ln -s /g/data1/ub4/erai/grib/oper_an_sfc/fullres/ei_oper_an_sfc_075x075_90N0E90S35925E_200603* era-tmp
./link_grib.csh era-tmp/*
./ungrib.exe
</syntaxhighlight>

OR you can use different ungrib prefixes for the surface and vertical fields, combining the files with metgrid:
<syntaxhighlight>
./link_grib.csh /g/data1/ub4/erai/grib/oper_an_pl/fullres/2006/ei_oper_an_pl_075x075_90N0E90S35925E_200603*
nano namelist.wps # Set ungrib/prefix to 'PL'
./ungrib.exe

./link_grib.csh /g/data1/ub4/erai/grib/oper_an_sfc/fullres/ei_oper_an_sfc_075x075_90N0E90S35925E_200603*
nano namelist.wps # Set ungrib/prefix to 'SFC'
./ungrib.exe
</syntaxhighlight>

===Invariant Data===

You will also need to generate the invariant data, which includes the land-sea mask used for interpolating the ERA Interim source dataset to your model's target resolution
The invariant data is only defined at 19890101T1200Z, so you need to set the start and end dates in `namelist.wps` to this value (if you're using multiple nests set all of the dates)
<syntaxhighlight>
&share
! Keep other settings the same, change dates to
start_date = '1989-01-01_12:00:00',
end_date = '1989-01-01_12:00:00',
/
&ungrib
out_format = 'WPS'
prefix = 'INV'
/
</syntaxhighlight>

<syntaxhighlight>
./link_grib.csh /g/data1/ub4/erai/grib/invariant/ei_oper_an_sfc_075x075_90N0E90S3585E_invariant
./ungrib.exe
</syntaxhighlight>

===Remaining processing===

Before running Metgrid you need to tell it about all of the input GRIB datasets. In `namelist.wps` set `fg_name` to a list of the time dependent datasets, and set `constants_name` to `INV:1989-01-01_12`

<syntaxhighlight>
&metgrid
fg_name = 'SFC', 'PL'
io_form_metgrid = 2,
constants_name = 'INV:1989-01-01_12'
/
</syntaxhighlight>

You can then run geogrid and metgrid as normal to generate the input files (don't forget to change back the run dates if you changed them for the invariant ungrib).

CESM1-LME

2018-10-04T06:45:46Z

ClaireCarouge: Imported from Wikispaces

The CESM Paleoclimate Working Group at NCAR conducted a series of Last Millennium community experiments, referred to as the Last Millennium Ensemble (LME). The LME used a ~2-degree atmosphere and land, ~1-degree ocean and sea ice version of CESM-CAM5_CN (1.9x2.5_gx1v6). Ensemble members extend from 850 to 2006 using reconstructions for the transient evolution of solar intensity, volcanic emissions, greenhouse gases, aerosols, land use conditions, and orbital parameters, together and individually. For more information: [http://www.cesm.ucar.edu/projects/community-projects/LME/ | CESM1-LME website]
You can find [http://www.cesm.ucar.edu/projects/community-projects/LME/known-issues.html | known issues] with this dataset listed in the same website.

===Terms of use===

From the CESM project website:
If you are interested in analyzing the CESM-CAM5 Last Millennium Ensemble, we kindly ask that you provide a short description of your proposed research focus and your contact information. Please send this information to Bob Tomas (tomas@ucar.edu) so that other users can see what projects are underway. A list of projects can be found [https://www2.cesm.ucar.edu/models/experiments/LME/projects | here].
A list of publications using the Last Millennium Ensemble can be found [https://www2.cesm.ucar.edu/models/experiments/LME/publications | here]. If you would like to add a publication to this list, please contact [[mailto:webhelp@cgd.ucar.edu | webhelp@cgd.ucar.edu]]. When presenting results either in oral or written form, please acknowledge the CESM1(CAM5) Last Millennium Ensemble Community Project and supercomputing resources provided by NSF/CISL/Yellowstone. An overview paper of the Last Millennium Ensemble Project should be cited as follows:
Otto-Bliesner, B.L., E.C. Brady, J. Fasullo, A. Jahn, L. Landrum, S. Stevenson, N. Rosenbloom, A. Mai, G. Strand. Climate Variability and Change since 850 C.E.: An Ensemble Approach with the Community Earth System Model (CESM), Bulletin of the American Meteorological Society, 735-754 (May 2016 issue)[http://www.cgd.ucar.edu/ccr/ottobli/pubs/Otto-Bliesner_BAMS-LME_2016.pdf | Article Download][http://www.cgd.ucar.edu/ccr/ottobli/pubs/BAMS-CESM-LME_accepted-25July2015.pdf | ]]

===CESM1-LME on raijin===

We downloaded a subset of the entire collection, mostly monthly fields. For each selected variables we downloaded all the available ensembles.
The data was downloaded from the NCAR [https://www.earthsystemgrid.org/home.html | earth system website] and it will be eventually hosted as part of the ua6 project.
The current location on raijin is:
/g/data1/ua6/CESM1-LME/<frequency>/<variable>/<version>/<files>

i.e.

/g/data1/ua6/CESM1-LME/mon/PS/v6/<files>

NB All the ensembles are stored in the same directory, this is a temporary arrangement and might change without notice!

**Atmosphere monthly fields:'''
* T - temperature V6
* TS - surface temperature V7
* TSMN - minimum temperature V6
* TSMX - maximum temperature V6
* PRECT - precipitation V2
* PS pressure V6
* PSL sea level pressure V7
* TAUX zonal surface stress V6
* TAUY meridional surface stress V6
* OMEGA - vertical velocity V6

**Atmosphere daily fields:'''
* Z500 - geopotential height at 500mbar V6
* TS - surface temperature V6
* PRECT - precipitation V2

**Ocean monthly fields:'''
* HBLT - boundary-layer depth V8
* HMXL - mixed layer depth V8
* SSH - sea surface height V8
* SST - potential temperature V8
* TEMP - potential temperature V8

**Ice monthly fields:'''
* aice - sea ice area V6

**Land monthly fields:'''
* QOVER - surface runoff V7
* QRUNOFF - total liquid runoff V7

WRF v3.9 installation

2018-09-10T06:59:58Z

ClaireCarouge: Imported from Wikispaces

This version is from 2017. Look for the latest version available at NCI at [[WRF | this page]].

='''Foreword'''=
----
This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modifications to WRF and WPS.
This document is only valid for version 3.9 of WRF and later. Please refer to the older version pages for other instructions.

=Source Code=
----
NCAR has moved the WRF source code to Git version control. The original code for WRF v3.9 can be found on [https://github.com/NCAR/WRFV3 | NCAR's release Github repository].
The source code ported to Raijin can be found both under /projects/WRF/WRFV_3.9 and on Github: [https://github.com/coecms/WRFV3 | https:''github.com/coecms/WRF]

Clone the WRF Github repository (using HTTPS or SSH protocol depending on your setup) to your own space under /short:
<syntaxhighlight>
cd /short/$PROJECT/$USER/
git clone -b V3.9 https://github.com/coecms/WRF.git
</syntaxhighlight>
or
<syntaxhighlight>
cd /short/$PROJECT/$USER/
git clone -b V3.9 git@github.com:coecms/WRF.git

</syntaxhighlight>
='''Building WRF (ARW)'''=
----
By default, WRF will output files in netcdf 3 format. If you would like to directly output files in netcdf 4 format with compression, you need to edit the file WRF/build.env before compilation. Note if you are planning to generate very frequent outputs, this will significantly slow down your simulation. You could be better off compressing the outputs afterwards but please don't forget to [[NetCDF Compression Tools | compress your data]]. 

Go to the WRFV3/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRF/WRFV3/
</syntaxhighlight>
There is a run_compile script to help you configure and compile WRF. This script by default will compile the WRFV3 software for a real experiment, with speed optimisations for the normal and express queues and with basic nesting. There is a range of command-line arguments accepted by the script. These are the command-line arguments accepted by the ''configure'' script, an option to clean everything prior to compilation, the choice of compilation flags and nesting type for the ''configure'' script and finally the compilation option for the ''compile'' script. You can access a full list and description by running:
<syntaxhighlight>
./run_compile -h
</syntaxhighlight>
To configure and compile WRFV3 with the default options, you simply run:
<syntaxhighlight>
./run_compile
</syntaxhighlight>
The configure step will be run interactively with the output on your screen, then the compilation step will be submitted to the express queue automatically. The compilation takes about 30-45 minutes on the express queue. To check the status, you can use the nqstat command.

After a successful build, one should see the following executable files for WRF (the ARW core):
<syntaxhighlight>
[abc123@raijin3 WRFV3]$ ls -l main/*.exe

-rwx------ 1 abc123 wrf 35559897 Aug 22 17:14 main/ndown.exe
-rwx------ 1 abc123 wrf 35471615 Aug 22 17:14 main/real.exe
-rwx------ 1 abc123 wrf 35081246 Aug 22 17:14 main/tc.exe
-rwx------ 1 abc123 wrf 38763058 Aug 22 17:13 main/wrf.exe
</syntaxhighlight>

**Tip''': if you are only running an idealised case and you are getting tired of specifying the option on the command-line, you can:
# change the default value of compile_case at the start of the run_compile script OR
# create an alias for the command in your .bashrc or .cshrc file (it is a good idea to change the name of the command for the alias so it is clear it is an alias if you need help later on):
bash syntax:
<syntaxhighlight>
alias my_compile='run_compile --compile_case <replace with the compile option here>'
</syntaxhighlight>
csh/tcsh syntax:
<syntaxhighlight>
alias my_compile 'run_compile --compile_case <replace with the compile option here>'
</syntaxhighlight>

**Tip:''' run_compile is sourcing the environment as defined in WRF/build.env. Feel free to change the versions of software you want to use in here. Note the ''module purge'' at the start. This means your currently loaded environment will be completely ignored.
=Debugging with WRF=
----
If you want to compile WRF for debugging, the default option for compilation is not valid because it includes some optimisation. You'll need to use one of the first four architecture options. For example, to debug with the distributed memory only option (dmpar):
<syntaxhighlight>
./run_compile -c -a 3 -D
</syntaxhighlight>
='''Building WPS'''=
----
Go to the WPS/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRF/WPS/
</syntaxhighlight>
There is another ''run_compile'' script to help with configuring and compiling WPS. You can again choose the input to the configure script from the command-line option as well as cleaning a previous compilation of WPS. The default is to compile with GRIB2 support and a distributed memory code. See the inline help for ''run_compile'' for a full explanation:
<syntaxhighlight>
./run_compile -h
</syntaxhighlight>

To configure and compile WPS with the default options, you simply run:
<syntaxhighlight>
./run_compile
</syntaxhighlight>
A file configure.wps should be created after configuring has completed.

After a successful build, one should see the following executable files:
<syntaxhighlight>
[abc123@raijin3 WPS]$ ls -l *.exe

lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:19 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:20 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
</syntaxhighlight>
<syntaxhighlight>
[abc123@raijin3 WPS]$ ls -l */src/*.exe

-rwx------ 1 abc123 wrf 3412782 Aug 23 10:19 geogrid/src/geogrid.exe
-rwx------ 1 abc123 wrf 3206974 Aug 23 10:20 metgrid/src/metgrid.exe
-rwx------ 1 abc123 wrf 1209622 Aug 23 10:20 ungrib/src/g1print.exe
-rwx------ 1 abc123 wrf 1407507 Aug 23 10:20 ungrib/src/g2print.exe
-rwx------ 1 abc123 wrf 2169010 Aug 23 10:20 ungrib/src/ungrib.exe
-rwx------ 1 abc123 wrf 1184660 Aug 23 10:20 util/src/avg_tsfc.exe
-rwx------ 1 abc123 wrf 1250303 Aug 23 10:20 util/src/calc_ecmwf_p.exe
-rwx------ 1 abc123 wrf 1217842 Aug 23 10:20 util/src/height_ukmo.exe
-rwx------ 1 abc123 wrf 953821 Aug 23 10:20 util/src/int2nc.exe
-rwx------ 1 abc123 wrf 1118699 Aug 23 10:20 util/src/mod_levs.exe
-rwx------ 1 abc123 wrf 3704627 Aug 23 10:20 util/src/plotfmt.exe
-rwx------ 1 abc123 wrf 3409302 Aug 23 10:20 util/src/plotgrids.exe
-rwx------ 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe
</syntaxhighlight>
Note that there are some warnings in the build output like "warning: overriding commands for target `.c.o'." These are harmless and can be ignored.

==Running WRF==
----
Please follow the [http://www2.mmm.ucar.edu/wrf/OnLineTutorial/Introduction/index.html | WRF tutorial] to learn how to run WRF. For convenience, there are 2 run scripts under WRFV3/run:
* run_real
* run_mpi
These can be used to submit both real.exe and wrf.exe to the queues.

=Tests run=
----
==Compilation==
* WRF compilation with dmpar
* WRF compilation with dm+sm
* WPS compilation with distributed memory with dm+sm compilation of WRF
==Tests simulations==
----
Tests simulations have been done on [https://accessdev.nci.org.au/jenkins/job/WRF/job/WRF-Core/ | Jenkins]. The simulations were:
* First tutorial case (Jan 00) with 1 nest
* First tutorial case (Jan 00) with 2 nests
* First tutorial case (Jan 00) with additional diagnostics turned on.
WRF outputs for these tests can be found under /projects/WRF/data/KGO/. The inputs for the tests can be found in https://github.com/coecms/wrf-testing

C20C

2018-08-15T00:25:58Z

ClaireCarouge: Imported from Wikispaces

==C20C+ - International CLIVAR C20C+ Detection and Attribution project==

This is an international project aiming to produce a large pool of output from climate models and impact models for use in improving our understanding of extreme weather in the context of past and current climate change.
Models are run under two families of scenarios: 
* All-Hist: under the time-varying boundary conditions observed during the past few decades 
* Nat-Hist: same as All-Hist but with the anthropogenic contribution to the boundary conditions removed
This and more detailed information can be found on the [http://portal.nersc.gov/c20c/main.html | C20C+ portal]

===License===
No registration is required. All output from this project published online is available according to the conditions of the [http://creativecommons.org/licenses/by-nc-sa/2.0/ | Creative Commons License].
The data is provided "as-is". The contributors have endeavoured to ensure quality control, however due to the large quantity of data it has not been possible to check everything in detail. If you spot anything odd in the data then please e-mail the people listed in the "contact" attribute of the NetCDF file. Errata are listed in the "details" files referenced in the [http://portal.nersc.gov/c20c/data.html | NERSC data portal].
If your analyses of this data themselves produce diagnostic data or output from "impacts" models that may be of interest to other researchers, please consider contributing them to the NERSC data portal. Protocols are described on the [http://portal.nersc.gov/c20c/experiment.html | Experiment] page.

===C20C+ on raijin===

We rsync data directly from the NERSC server, so we have all the files available on disk, which includes all the smaller and frequently accessed files.
* All "fixed" ("fx") variables
* All "month" ("mon") variables
* The following "day/atmos" ("Aday") variables: clt, hurs, huss, pr, prsn, prw, ps, psl, rlds, rsds, rsus, tas, tasmax, tasmin, ts, uas, vas
We are also contributing to the project, these files are stored separately.
The replica is currently stored in '''/g/data1/ua8/C20C/'''

the directory structure can vary depending on the institute but it's mostly:
* <institute-id>/<model>/<experiment>/<ensemble>/<version>/<frequency>/<realm>/<variable>/<run>/<files>
While this directory structure differs from the one used for CMIP data, the definitions of realm, frequency, mip and variable names are the same. Experiments are "All-Hist" and "Nat-Hist", ensembles are like "est1"

Also the filenames are similarly structured:
* <variable>_<mip>_<model>_<experiment>_<ensemble>_<version>_<run>_<time-period>.nc

Available sub-projects:
C20C (includes HadISST1 NOAA-OI-v2 WRAF)
LBNL
MIROC
UCT-CSAG
ETH
HAPPI
MOHC
NOAA-ESRLandCIRES
CCCma
NCC
MPI-M

The original ARCCSS contribution is stored in /g/data1/ua8/ARCCSS_Data/C20C/NERSC and it has a simplified directory structure

* <frequency>/<realm>/<variable_name>/<ensemble>/<files>
filenames are the same as for the replica, ensembles are like CMIP ensemble ie "r1i1p1".
More information on these experiment are available from their [https://researchdata.ands.org.au/atmospheric-access13-historical-sub-project/645160 | RDA record]

WRF v3.9.1.1 installation

2018-06-21T23:44:17Z

ClaireCarouge: Imported from Wikispaces

='''Foreword'''=
----
This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modifications to WRF and WPS.
This document is only valid for version 3.9.1.1 of WRF and later. Please refer to the older version pages for other instructions.

=Source Code=
----
The source code ported to Raijin can be found both under /projects/WRF/WRFV_3.9.1.1 and on Github: [[https://github.com/coecms/WRF]]

Clone the WRF Github repository (using HTTPS or SSH protocol depending on your setup) to your own space under /short:
<syntaxhighlight>
cd /short/$PROJECT/$USER/
git clone -b V3.9.1.1 https://github.com/coecms/WRF.git

</syntaxhighlight>
or
<syntaxhighlight>
cd /short/$PROJECT/$USER/
git clone -b V3.9.1.1 git@github.com:coecms/WRF.git

</syntaxhighlight>
='''Building WRF (ARW)'''=
----
The building instructions are the same as for WRF V3.9. Please refer to the [[WRF v3.9 installation | wiki page for WRF v3.9]] or to the [https://github.com/coecms/WRF | github repository] .

WRF v3.7.1 installation

2018-06-21T06:11:34Z

ClaireCarouge: Imported from Wikispaces

This version is from 2015. Look for the latest version available at NCI at [[WRF | this page]].
='''Foreword'''=
----
This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modification to WRF and WPS.
This document is only valid from version 3.7.1 of WRF. Please refer to the [[WRF Installation | old installation page]] for older versions.

=Source Code=
----
The source code tar files of WRF and WPS (V3.7.1) have been downloaded and placed in the subdirectory /projects/WRF/Downloads on the RAIJIN cluster. The uncompressed source code for V3.7.1 of WRF and WPS, which have been ported to build on RAIJIN cluster, is located in the subdirectory /projects/WRF/WRFV_3.7.1.

='''Prerequisites'''=
----
Get an NCI account, please refer to the [[Induction | Induction page]] of this wiki for instructions.
Check the loaded modules and you should already have the Intel compilers and OpenMPI loaded, run:
<syntaxhighlight>
module list -l
- Package -----------------------------+- Versions -+- Last mod. ------
Currently Loaded Modulefiles:
pbs 2013/05/13 0:17:55
dot 2008/11/25 19:24:19
intel-fc/12.1.9.293 2013/08/02 3:40:39
intel-cc/12.1.9.293 2013/08/02 3:47:33
openmpi/1.6.3 2013/04/02 4:20:55
</syntaxhighlight>
Load the required netcdf module if missing, run:
<syntaxhighlight>
module load netcdf/4.3.3.1
</syntaxhighlight>
If configuring to build GRIB2 support, run (tcsh):
<syntaxhighlight>
setenv JASPERINC /usr/include
setenv JASPERLIB /usr/lib64
</syntaxhighlight>
or in bash
<syntaxhighlight>
export JASPERINC=/usr/include
export JASPERLIB=/usr/lib64
</syntaxhighlight>
To enable large file support in NetCDF (This installation of NetCDF supports large file support), run(tcsh):
<syntaxhighlight>
setenv WRFIO_NCD_LARGE_FILE_SUPPORT 1
</syntaxhighlight>
or in bash:
<syntaxhighlight>
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
</syntaxhighlight>
Clone the WRF repository to your own space under /short:
<syntaxhighlight>
cd /short/$PROJECT/$USER/
git clone /projects/WRF/WRFV_3.7.1
</syntaxhighlight>
='''Building WRF (ARW)'''=
----
Go to the WRFV3/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRFV_3.7.1/WRFV3/
</syntaxhighlight>
Start with a clean installation before compiling WRF, run:
<syntaxhighlight>
./clean -a
</syntaxhighlight>
Configure and compile WRF, run:
<syntaxhighlight>
./run_compile
</syntaxhighlight>
You will be given four choices using Intel compiler. These choices range from compiling for a single processor job (serial), OpenMP shared-memory (smpar), to using distributed-memory (dmpar) or distributed Memory with shared memory (dm+sm).
<syntaxhighlight>
Please select from among the following Linux x86_64 options:
1. (serial) 2. (smpar) 3. (dmpar) 4. (dm+sm) INTEL (ifort/icc)
Enter selection [1-4] :
</syntaxhighlight>
Once an option is selected from above, a choice of what type of nesting (no nesting (0), basic (1), pre-set moves (2), or vortex following (3)) will be given.
<syntaxhighlight>
Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]:
</syntaxhighlight>
A file configure.wrf should be created after configuring has completed.
Then a compilation script, called wrf_compile, is submitted to the job queue by the run_compile script.

After a successful build, one should see the following executable files for WRF (the ARW core):
<syntaxhighlight>
[abc123@raijin3 WRFV3]$ ls -l main/*.exe

-rwx------ 1 abc123 wrf 35559897 Aug 22 17:14 main/ndown.exe
-rwx------ 1 abc123 wrf 35471615 Aug 22 17:14 main/real.exe
-rwx------ 1 abc123 wrf 35081246 Aug 22 17:14 main/tc.exe
-rwx------ 1 abc123 wrf 38763058 Aug 22 17:13 main/wrf.exe
</syntaxhighlight>
='''Building WPS'''=
----
Go to the WPS/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRF/WRFV_3.7.1/WPS/
</syntaxhighlight>
Start with a clean installation before compiling WPS, run:
<syntaxhighlight>
clean -a
</syntaxhighlight>
Run run_compile:
<syntaxhighlight>
./run_compile
</syntaxhighlight>
You will be given four choices using Intel compiler.
<syntaxhighlight>
Please select from among the following build methods.
1. Linux x86_64, Intel compiler (serial)
2. Linux x86_64, Intel compiler (serial_NO_GRIB2)
3. Linux x86_64, Intel compiler (dmpar)
4. Linux x86_64, Intel compiler (dmpar_NO_GRIB2)
Enter selection [1-4] :
</syntaxhighlight>
Choose one of options from above.

A file configure.wps should be created after configuring has completed. Then the run_compile script will send a compilation job, called wps_compile, to the queue.
After a successful build, one should see the following executable files:
<syntaxhighlight>
[abc123@raijin3 WPS]$ ls -l *.exe

lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:19 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:20 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
</syntaxhighlight>
<syntaxhighlight>
[abc123@raijin3 WPS]$ ls -l */src/*.exe

-rwx------ 1 abc123 wrf 3412782 Aug 23 10:19 geogrid/src/geogrid.exe
-rwx------ 1 abc123 wrf 3206974 Aug 23 10:20 metgrid/src/metgrid.exe
-rwx------ 1 abc123 wrf 1209622 Aug 23 10:20 ungrib/src/g1print.exe
-rwx------ 1 abc123 wrf 1407507 Aug 23 10:20 ungrib/src/g2print.exe
-rwx------ 1 abc123 wrf 2169010 Aug 23 10:20 ungrib/src/ungrib.exe
-rwx------ 1 abc123 wrf 1184660 Aug 23 10:20 util/src/avg_tsfc.exe
-rwx------ 1 abc123 wrf 1250303 Aug 23 10:20 util/src/calc_ecmwf_p.exe
-rwx------ 1 abc123 wrf 1217842 Aug 23 10:20 util/src/height_ukmo.exe
-rwx------ 1 abc123 wrf 953821 Aug 23 10:20 util/src/int2nc.exe
-rwx------ 1 abc123 wrf 1118699 Aug 23 10:20 util/src/mod_levs.exe
-rwx------ 1 abc123 wrf 3704627 Aug 23 10:20 util/src/plotfmt.exe
-rwx------ 1 abc123 wrf 3409302 Aug 23 10:20 util/src/plotgrids.exe
-rwx------ 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe
</syntaxhighlight>
Note that there are some warnings in the build output like warning: overriding commands for target `.c.o'. These are harmless and can be ignored.

=Tests run=
----
==Compilation==
* WRF compilation with dmpar
* WRF compilation with dm+sm
* WRF-Chem compilation with dmpar
* WPS compilation with dmpar_NO_GRIB2 with dmpar compilation of WRF
* WPS compilation with dmpar_NO_GRIB2 with dm+sm compilation of WRF
==Tests simulations==
* First tutorial case (Jan 00) with 1 nest
* First tutorial case (Jan 00) with 2 nests

WRF Installation

2018-06-21T06:06:00Z

ClaireCarouge: Imported from Wikispaces

This version is from 2011. Look for the latest version available at NCI at [[WRF | this page]].
='''WRF (version 3.3) Installation'''=

=='''Forword'''==

This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modification to WRF and WPS.

==Source Code==

The source code tar files of WRF and WPS (V3.3) have been downloaded and placed in the subdirectory /projects/WRF/Downloads on the RAIJIN cluster. The uncompressed source code for V3.3 of WRF and WPS, which have been ported to build on RAIJIN cluster, is located in the subdirectory /projects/WRF/WRFV_3.3.

=='''Prerequisites'''==

Get an NCI account, please refer to the NCI web site for more details on how to get an account ([[http://nf.nci.org.au/accounts/forms/user_registration.php]] and[[http://nf.nci.org.au/accounts/forms/user_connection.php]])

On approval, you will get

Project ID: $PROJECT (e.g. w35)
User ID: $USER (e.g. abc123)

Read the NCI email about your account as well as their user guide webpage http://nf.nci.org.au/wiki/RaijinUserGuide

After you log in, change your password, run:
<syntaxhighlight>
passwd
</syntaxhighlight>
Check the loaded modules and you should already have the Intel compilers and OpenMPI loaded, run:
<syntaxhighlight>
module list -l
- Package -----------------------------+- Versions -+- Last mod. ------
Currently Loaded Modulefiles:
pbs 2013/05/13 0:17:55
dot 2008/11/25 19:24:19
intel-fc/12.1.9.293 2013/08/02 3:40:39
intel-cc/12.1.9.293 2013/08/02 3:47:33
openmpi/1.6.3 2013/04/02 4:20:55
</syntaxhighlight>
Load the required netcdf module if missing, run:
<syntaxhighlight>
module load netcdf/4.2.1.1
</syntaxhighlight>
If configuring to build GRIB2 support, run (tcsh):
<syntaxhighlight>
setenv JASPERINC /usr/include
setenv JASPERLIB /usr/lib64
</syntaxhighlight>
or in bash
<syntaxhighlight>
export JASPERINC=/usr/include
export JASPERLIB=/usr/lib64
</syntaxhighlight>
To enable large file support in NetCDF (This installation of NetCDF supports large file support), run(tcsh):
<syntaxhighlight>
setenv WRFIO_NCD_LARGE_FILE_SUPPORT 1
</syntaxhighlight>
or in bash:
<syntaxhighlight>
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
</syntaxhighlight>

Create your personal subdirectory in /short/, run:
<syntaxhighlight>
mkdir -p /short/$PROJECT/$USER/WRF/WRFV_3.3
</syntaxhighlight>
Copy the WRF and WPS files to your /short/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRF/WRFV_3.3
cp -rf /projects/WRF/WRFV_3.3/WRFV3 .
cp -rf /projects/WRF/WRFV_3.3/WPS .
</syntaxhighlight>
=='''Building WRF (ARW)'''==

Go to the /WRFV3/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRF/WRFV_3.3/WRFV3/
</syntaxhighlight>
Start with a clean installation before compiling WRF, run:
<syntaxhighlight>
./clean -a
</syntaxhighlight>
Configure WRF, run:
<syntaxhighlight>
./configure
</syntaxhighlight>
You will be given four choices using Intel compiler. These choices range from compiling for a single processor job (serial), OpenMP shared-memory (smpar), to using distributed-memory (dmpar) or distributed Memory with shared memory (dm+sm).

----
<syntaxhighlight>
Please select from among the following build methods.
1. Linux x86_64 i486 i586 i686, ifort compiler with icc (serial)
2. Linux x86_64 i486 i586 i686, ifort compiler with icc (smpar)
3. Linux x86_64 i486 i586 i686, ifort compiler with icc (dmpar)
4. Linux x86_64 i486 i586 i686, ifort compiler with icc (dm+sm)
Enter selection [1-4] :
</syntaxhighlight>
Once an option is selected from above, a choice of what type of nesting (no nesting (0), basic (1), pre-set moves (2), or vortex following (3)) will be given.
----
<syntaxhighlight>
Compile for nesting? (0=no nesting, 1=basic, 2=preset moves, 3=vortex following) [default 0]: For options dmpar and dm+sm, the choice 0=no nesting is not applicable.
</syntaxhighlight>

A file configure.wrf should be created after configuring has completed.

To build WRF (the ARW core), use the script run_compile to submit it to the job queue, run:
<syntaxhighlight>
qsub run_compile
</syntaxhighlight>
After a successful build, one should see the following executable files for WRF (the ARW core):
<syntaxhighlight>
[abc123@vayu3 WRFV3]$ ls -l main/*.exe

-rwxr-xr-x 1 abc123 wrf 35559897 Aug 22 17:14 main/ndown.exe
-rwxr-xr-x 1 abc123 wrf 34988805 Aug 22 17:14 main/nup.exe
-rwxr-xr-x 1 abc123 wrf 35471615 Aug 22 17:14 main/real.exe
-rwxr-xr-x 1 abc123 wrf 35081246 Aug 22 17:14 main/tc.exe
-rwxr-xr-x 1 abc123 wrf 38763058 Aug 22 17:13 main/wrf.exe
</syntaxhighlight>
=='''Building WPS'''==

Go to the /WPS/ subdirectory, run:
<syntaxhighlight>
cd /short/$PROJECT/$USER/WRF/WRFV_3.3/WPS/
</syntaxhighlight>
Load required module, run:
<syntaxhighlight>
module load ncl/6.1.2
</syntaxhighlight>
Start with a clean installation before compiling WPS, run:
<syntaxhighlight>
clean -a
</syntaxhighlight>
Configure WPS, run:
<syntaxhighlight>
./configure
</syntaxhighlight>
You will be given four choices using Intel compiler.
----
<syntaxhighlight>
Please select from among the following build methods.
1. PC Linux x86_64, Intel compiler serial, NO GRIB2
2. PC Linux x86_64, Intel compiler serial
3. PC Linux x86_64, Intel compiler DM parallel, NO GRIB2
4. PC Linux x86_64, Intel compiler DM parallel
Enter selection [1-4] :
</syntaxhighlight>
Choose one of options from above.

A file configure.wps should be created after configuring has completed.

To build WPS, use the script run_compile to submit it to the job queue. This will build the em_real version.
<syntaxhighlight>
qsub run_compile
</syntaxhighlight>
After a successful build, one should see the following executable files:
<syntaxhighlight>
[abc123@vayu3 WPS]$ ls -l *.exe

lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:19 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:20 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
</syntaxhighlight>
<syntaxhighlight>
[abc123@vayu3 WPS]$ ls -l */src/*.exe

-rwxr-xr-x 1 abc123 wrf 3412782 Aug 23 10:19 geogrid/src/geogrid.exe
-rwxr-xr-x 1 abc123 wrf 3206974 Aug 23 10:20 metgrid/src/metgrid.exe
-rwxr-xr-x 1 abc123 wrf 1209622 Aug 23 10:20 ungrib/src/g1print.exe
-rwxr-xr-x 1 abc123 wrf 1407507 Aug 23 10:20 ungrib/src/g2print.exe
-rwxr-xr-x 1 abc123 wrf 2169010 Aug 23 10:20 ungrib/src/ungrib.exe
-rwxr-xr-x 1 abc123 wrf 1184660 Aug 23 10:20 util/src/avg_tsfc.exe
-rwxr-xr-x 1 abc123 wrf 1250303 Aug 23 10:20 util/src/calc_ecmwf_p.exe
-rwxr-xr-x 1 abc123 wrf 1217842 Aug 23 10:20 util/src/height_ukmo.exe
-rwxr-xr-x 1 abc123 wrf 1118699 Aug 23 10:20 util/src/mod_levs.exe
-rwxr-xr-x 1 abc123 wrf 3704627 Aug 23 10:20 util/src/plotfmt.exe
-rwxr-xr-x 1 abc123 wrf 3409302 Aug 23 10:20 util/src/plotgrids.exe
-rwxr-xr-x 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe
</syntaxhighlight>
Note that there are some warnings in the build output like warning: overriding commands for target `.c.o'. These are harmless and can be ignored.

Embarrassingly parallel job

2018-02-09T01:06:31Z

ClaireCarouge: Imported from Wikispaces

Let's say you want to run the same command a lot of time with different inputs. Each iteration of the command is totally independent of the other iterations. Or you could have a lot of different scripts you need to run which are independent of each other. You have what we call an embarrassingly parallel problem. This page is to explain how to setup this on PBS scheduler in order to run everything under one PBS job instead of lots of smaller jobs. Running several 1 CPU jobs on the HPC is not efficient as it is asking for more work from the scheduler and it means other people in the same group can run fewer jobs (since there is a limit on the number of jobs in the queue per project). At the same time, bigger jobs tends to get higher priority than smaller jobs hence you could see a decrease in the time spent in the queue.

This type of scheduling work is called array jobs and NCI already gives a lot of information on [https://opus.nci.org.au/display/Help/How+to+submit+array+jobs+on+Raijin | their help pages]. Here we try to give a few more details.

==Run the same scripts with a lot of inputs.==
Let's take an example. Let's say you have the script (make sure it is executable for you using ''chmod''):
<syntaxhighlight>
#!/bin/bash
sleep 10
echo "I am ${HOSTNAME}, my arguments are >$*<"
</syntaxhighlight>
And you want to run it with the inputs as in the linked file inputs.txt.
[[:File:inputs.txt]]

Scott Wales has written the following example to do this:
<syntaxhighlight>
#!/bin/bash
# Run an embarrassingly parallel job, where each command is totally independant
# Uses gnu parallel as a task scheduler, then executes each task on the available cpus with pbsdsh

#PBS -q express
#PBS -l ncpus=32
#PBS -l walltime=0:10:00
#PBS -l mem=2gb
#PBS -l wd

module load parallel

SCRIPT=./script.sh # Script to run.
INPUTS=inputs.txt # Each line in this file is used as arguments to ${SCRIPT}
# It's fine to have more input lines than you have requested cpus,
# extra jobs will be executed as cpus become available

# Here '{%}' gets replaced with the job slot ({1..$PBS_NCPUS})
# and '{}' gets replaced with a line from ${INPUTS}
parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- /bin/bash -l -c "${SCRIPT} {}" :::: ${INPUTS}
</syntaxhighlight>
GNU parallel manages the parallelism while pbsdsh makes sure the command is run on the appropriate node. You want to load the bash in this case because the environment created by pbsdsh is very basic.

==Run several scripts==
In the previous example, it is assumed the user has 1 script and a lot of inputs. If alternatively, you have several scripts you can modify the command in this way:
<syntaxhighlight>
parallel -j ${PBS_NCPUS} pbsdsh -n {%} -- /bin/bash -l -c {} :::: ${INPUTS}
</syntaxhighlight>
You want to load the bash in this case because the environment created by pbsdsh is very basic. And then, you need to put each ''command line'' inside inputs.txt as a separate line as in the inputs2.txt below. Note by command line we mean the script name and all arguments needed (again make sure the scripts are executable by you). The inputs.txt lines should look just the same as you would type on a terminal if running each script separately.
[[:File:inputs2.txt]]

==Outputs==
If your script is writing outputs to the standard output (for log information for example), you need to make sure to handle the outputs correctly. If you'd like each of your scripts to create its own output file, the simplest is probably to use the second method (Run several scripts) and to redirect the outputs for each script in the inputs.txt file. See the inputs3.txt example file below.
[[:File:inputs3.txt]]

FAQs

2018-02-08T01:17:13Z

ClaireCarouge: Imported from Wikispaces

This page lists FAQs for the different model supported by the CMS team, as well as FAQs around accessing and publishing data.
=FAQ pages=

[[UM and ACCESS FAQ]]
[[um-errors | UM error messages]]
[[WRF FAQ]]
[[LIS and NUWRF FAQ]]
[[MOM FAQ]]
[[Data FAQ]]

=FAQ miscellaneous=
==How do I run independent jobs in parallel?==
If you have to run the same script with a lot of independent sets of inputs, or you have lots of similar scripts to run at NCI. And these scripts run on 1 processor. You could submit each of your jobs independently via PBS but then small jobs are not efficient at NCI. It would probably be better for you to have a master script that dispatches your jobs to available processors. You then submit this master script to PBS and this script requires a lot more processors and hence might be more efficient with the scheduler. There is an example of such a master script [[Embarrassingly parallel job | on this page]].

==How to transform 1D lat/lon arrays into 2D lat/lon arrays with Python?==
If using numpy or masked arrays, one can use the resize() method. Assuming slat and slon are 1D latitude and longitude arrays, do this:
<syntaxhighlight lang=python>
tmp = numpy.resize(slat, (slon.size, slat.size))
slat2D = numpy.transpose(tmp)
slon2D = numpy.resize(slon, (slat.size, slon.size))

</syntaxhighlight>
If you want masked arrays, replace "numpy." by "numpy.ma."

==How do I push my local code changes on raijin back to github using ssh keys?==

You have an account on [http://github.com | github]. You have cloned a repository to raijin and made changes to it. Now you want to push those changes back to the github repository but you get errors like this (where <username> is your github username):
<syntaxhighlight>
error: The requested URL returned error: 403 Forbidden while accessing https://github.com/<username>/test.git/info/refs

fatal: HTTP request failed
</syntaxhighlight>
Firstly, you need to [https://help.github.com/articles/generating-ssh-keys/ | make an ssh key and add the public part of the key to your account on github].

Check the url to which you are trying to push your changes:

<syntaxhighlight>
git remote -v
origin https://github.com/<username>/test.git (fetch)
origin https://github.com/<username>/test.git (push)
</syntaxhighlight>

If it looks like the output above you need to alter the url and put in a login name in front of the address. For ssh access the login name is always git. The command to set the remote url is:
<syntaxhighlight>
git remote set-url origin ssh://git@github.com/<username>/test.git

</syntaxhighlight>
but replace <username> with your github username.

==Why do I get segfaults?==

A segmentation fault means the program you are using has tried to read or write a memory location it does not have access to. This can mean there is an error in the code, or there is a system limit set. If the program runs for others, but not you, it may well be a system limit issue. This seems to be quite common when trying to write netcdf files. In some cases this can be overcome by setting your stack size to unlimited

For tcsh/csh add this to your .cshrc:
<syntaxhighlight>
limit stacksize unlimited
</syntaxhighlight>

For bash add this to your .bashrc:
<syntaxhighlight>
ulimit -s unlimited
</syntaxhighlight>

Note: you must log out and log back in again for this change to take effect.

Nesting Suite

2017-12-14T04:49:52Z

ClaireCarouge: Imported from Wikispaces

Up-to-date information and news about the nested suite can be found on the RMED wiki and Rose issue tracker:
* https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting
* https://code.metoffice.gov.uk/trac/roses-u/ticket/52

==Nesting inside ERA-Interim==

The suite can be driven from ERA-Interim analyses with a small amount of setup.

At NCI we have 6 hourly ERA-Interim analyses in GRIB format under the directory /g/data1/ub4/erai/grib. As ERA-Interim is under licence, make sure you comply with requirements [[ERA INTERIM | here]] before starting

Currently code changes required to run from ERA-Interim are in the job '''u-at264''', which is based on the 2017 nesting suite.

Make a copy of this job with
<syntaxhighlight>
rosie copy u-at264
</syntaxhighlight>

Open the job in `rose edit` and make the following changes to '''suite conf''' to enable ERA-Interim driving:
* In 'General run options' make sure SITE is NCI
* In 'Driving model setup', set:
** 'dm_name' to 'ECMWF forecast fields'
** 'dm_ec_ic_lbc_dir' to '\$ROSE_DATA/erai'
** 'dm_ec_cb_freq' to '21600' (6 hours, the ERA-Interim analysis frequency)

To run from ERA interim the suite needs to generate lateral boundary conditions from the ERA-interim GRIB files. To enable this you need to create a new 'Resolution setup' that is a bit larger than your coarsest LAM domain. This resolution won't be used to run the model, instead the ERA interim data will be interpolated to it so that the boundary conditions can be cut out for the LAM. For instance if your coarsest LAM is a 200x200 grid with resolution 0.1 x 0.1 degrees, a 220x220 grid at the same resolution would be fine - this gives a border of 10 grid points around each edge to use for the boundary conditions. You could also use less grid points at a coarser resolution, though be cautious as this adds an extra interpolation step.

You will need some extra ancillary files when using ERA-Interim, set the following in your new resolution, as well as your coarsest LAM resolution:
* vegsoil_anc: True
* lai_anc: True
* alb_anc: False
* orog_strm: False
* 3D_ancils: True
* sstsmc_anc: True

Set the `nmods` to 0, since we just want the ancillaries at this resolution (this will print a warning in Rose, which can be ignored)

To make the ERA interim driver use this resolution you just set up, in the 'Driving model setup' page set 'dm_ec_lam_ancil_dir}}' to the new resolution's ancillary directory. For instance, if the new resolution is `resn_2` of region `Regn1`, set `{{dm_ec_lam_ancil_dir}}` to `{{\$ROSE_DATA/ancils/Regn1/resn_2`

Note that the resolution of the ERA Interim source dataset is ~ 0.75 x 0.75 degrees, keep this in mind when choosing the resolution of your coarsest LAM (i.e. make sure your LAM isn't inside a single ERA Interim grid point!).

https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting/ECdriver

[[include component="page" wikiName="climate-cms" page="Index UM" editable="1"]]

UM Environment

2017-12-12T00:01:50Z

ClaireCarouge: Imported from Wikispaces

=How to set up the Unified Model for your first climate simulation=

The UK Met Office's Unified Model is the atmospheric part of the ACCESS model, it was used by CAWCR for the AMIP and CMIP model inter-comparison projects and is the main atmospheric model used by BoM for weather forecasting.

Researchers at the CoE access the UM by logging in to the supercomputers at the National Compute Infrastructure (NCI). NCI hosts a range of computing facilities, from supercomputers to data archiving & analysis, and is used by a wide range of researchers in different fields from across Australia.

==Step 1: NCI Account==

Obtain a [[NCI Guidelines Requesting Access | NCI account]] and get connected to a [[NCI Guidelines Projects | compute project]] so that you can log on to the supercomputer

==Step 2: ACCESS Registration==

Due to licensing restrictions the UM is not available to just anyone with an NCI account. Join the [https://my.nci.org.au/mancini/project/access/join | ACCESS project] to get access to the model.

==Step 3: Connect to NCI==

Once your NCI account is set up and you've been registered as an ACCESS member you can start using all these accounts by connecting to the supercomputer. To do this run:
<syntaxhighlight>
ssh -Y abc123@raijin.nci.org.au
</syntaxhighlight>
from the command line, substituting <nowiki>abc123</nowiki> with your own username.You can update your NCI password by going to https://my.nci.org.auTry running
<syntaxhighlight>
qstat
</syntaxhighlight>
This gives you the full list of what everyone's running on the supercomputer, usually there's a bit under a thousand different jobs running at any one time.Other helpful commands to know are
<syntaxhighlight>
nqstat
nci_account
lquota
</syntaxhighlight>
<nowiki>nqstat`` shows the running jobs for only your own project, which is in general a more manageable list. ``nci_account`` show you what resources are available in your current project for compute time and storage, ``lquota</nowiki> shows you how much space is available in your home directory.

==Step 4: Setting up Accessdev==

The user interface for the UM is hosted on a server called accessdev. You can connect to accessdev by running:
<syntaxhighlight>
ssh -Y abc123@accessdev.nci.org.au
</syntaxhighlight>
If this does not work, email [[mailto:climate_help@nci.org.au | climate_help]] so we can check your account has been added properly.

The UM's source code is stored in a subversion repository. Building the model happens automatically, however the system needs to be able to see the source code. Check this works by running
<syntaxhighlight>
svn info https://access-svn.nci.org.au/svn/um
</syntaxhighlight>
If you have any trouble accessing this repository, email [[mailto:climate_help@nci.org.au | climate_help]] - your account may need to be given read permissions. The repository will ask to store your NCI password in plain text. Agree to this, otherwise the automatic build system won't be able to get access.

To download the model source code the UMUI will need to access your NCI password. Your password is stored securely using a program called `gpg-agent`. To set this up, run
<syntaxhighlight>
accesssvn-setup
</syntaxhighlight>
This will set up gpg-agent for your account, but you'll need to log out of accessdev and then back in again in order to start it up. Once it's started, you can save your NCI password with the command
<syntaxhighlight>
access-auth
</syntaxhighlight>
Passwords are saved for 12 hours.

The UM's interface is on accessdev, however the model actually runs on the supercomputer. To make this work you need to set things up so that accessdev can talk to the supercomputer, which is done using SSH keys. These allow you to connect to another computer without having to enter your password every time - for the details on how to do this see the wiki entry on [[SSH Agents]].

You'll also need to set the same DATAOUTPUT variable you created on Raijin. If you use csh add to ~/.login:
<syntaxhighlight>
setenv DATAOUTPUT /short/$PROJECT/$USER/UM_ROUTDIR
</syntaxhighlight>
or if you use bash add to ~/.bashrc:
<syntaxhighlight>
export DATAOUTPUT=/short/$PROJECT/$USER/UM_ROUTDIR
</syntaxhighlight>

==Step 5: The UM User Interface==

With all of this setup out of the way you can finally start the UM's user interface and run a model, by running
<syntaxhighlight>
umuix
</syntaxhighlight>

[[File:devumui.png]]

You'll be greeted with a table showing a list of UM experiments. A couple are available by default, these are what we call 'standard runs', commonly used & tested jobs set up to be simple to use and base new experiments off of. If you press the 'All' button at the bottom of the window you can see everybody's UM experiments, press 'User' to go back to just your own.

The majority of existing runs in the database have been set up for NCI's old Vayu supercomputer. Converting runs is pretty easy, but for the moment we'll look at runs that have already been set up to run on Raijin.

The usual workflow with the UM is to copy an existing job, modify it to your liking then submit it to the supercomputer. To start off with we'll forgo most of that and try submitting one of the standard runs. Click the folder next to ''''vabg'''' to show the model runs available.

[[File:dev-vabg.png]]

At the moment this experiment has the AMIP configuration of ACCESS version 1.3 as well as a low-resolution NWP test. You can get more details on a model run by clicking on the notepad in the leftmost column. Open up the N48 experiment '''a''', either by double-clicking or selecting it and pressing the 'View...' button.

[[File:dev-vabga.png]]i

The left panel of the interface shows configuration sections, the right panel shows the available settings within each section. Feel free to take a look at what's available.

To submit a run you first need to create a set of job scripts, do this by pressing the '''Process''' button on the right. You need to do this every time you make a change in the settings. Once the processing is done press '''Submit''' to send the job to the supercomputer.

A UM model run happens in three stages. First the code is downloaded from subversion in the extract stage, then the model is compiled in the build stage, then the actual model run happens. The model is usually rebuilt for each experiment because the model has many different science sections, enabling or disabling these changes what code needs to be added to the model. Usually building the model takes about 20 min, the model's run time depends on how many processors you're using and how long a model it is.

To find the model's output you'll need to go back to Raijin. Check the run's status with <nowiki>nqstat</nowiki>, when it's done the output will be in the folder you created earlier:
<syntaxhighlight>
ls /short/$PROJECT/$USER/UM_ROUTDIR/$USER/vabga
</syntaxhighlight>

You can also find run logs in the folder ~/um_output.

NCI Guidelines

2017-11-29T04:18:00Z

ClaireCarouge: Imported from Wikispaces

=Centre’s CIs and supervisors=
----
You are expected to know these guidelines and to advise your ECRs and teams to consult these guidelines.
You are expected to contribute to the Centre with additional contributions, from NCMAS, your university share etc. where possible

=NCI Project Representative=
----
A project representative is either the Lead Chief Investigator listed on the project application or their delegate in charge of resource management.

Resource allocation at NCI happens quarterly, however, within the Centre we try to balance resource usage between our projects more frequently.

Project representatives are responsible for:
* Estimating quarterly SU requirements in collaboration with all project members, to ensure that there are available resources for all members
* If SU requests for a quarter are much larger than usual, the project representative is in charge to discuss with the Lead CI, the CMS team and eventually the RP Leads to ensure either the requests can be fulfilled or to set up priorities.
* Communicating with CMS the project's SU and storage requirements, so that resource usage can be balanced across the centre
* Reading and replying (when necessary) to usage update emails sent by CMS
* Reviewing project membership regularly
You will have a lower priority for access to Centre’s resources if there are underused resources in other projects you have access to.

=NCI Project Members=
----
All members of an NCI project have requirements for using Centre resources. Resources are shared by all members of the project, please make sure that other members are able to do their work.

==General==
* Everyone who intends to use more than 10% of the usual project allocation is responsible for communicating their compute time requirements to their project representative before the start of each quarter.
* Everyone is responsible to know their projects CIs ([[https://my.nci.org.au]]) and how their project is organised (who does what). Write an email to your project Lead CI if you don’t know.
* Everyone is responsible to plan their usage in compute time and storage.
* Keep your contact details up to date on [[https://my.nci.org.au]], even after leaving the Centre if you still need access - we will use these details when contacting project members.
* Before leaving the Centre you need to tidy up your space following the best practices guides (refer to [[Leaving the Centre guidelines | Leaving the Centre]]).

==Data Resources==
The ARC requires the Centre to abide to the Open Access policy. Existing CMS policies already comply with Open Access requirements for publishing data. Guidelines are online.

* Everyone is encouraged to fill in a data management plan to better keep track of your data and storage requirements
* Everyone is responsible to know the best practices for each file system at NCI. Please ensure any data in group spaces (/short, /g/data, massdata) are readable by your group:
** [[NCI Guidelines home | /home]]
** [[NCI Guidelines short | /short ]]
** [[NCI Guidelines gdata | /g/data]]
** [[NCI Guidelines MDSS | MDSS]]
* Check the central data store (data wiki page, NCI data catalogue, Python conda install, modules) and request your data or software package centrally if not already there. Do not download/install yourself before asking for central resources, we want to avoid duplicating data as much as possible

==Compute Resources==
* Large simulations should be approved by the Lead CI. Large is more than 60-70% of the usual quarterly time allocation for the project
* Very large simulations should be approved by the Research Program. Very large is more than 100% of the usual quarterly time allocation for the project (i.e. will cover multiple quarters or will require a substantial increase in allocation for the project)

=CMS Team=
----
The CMS team is here to provide support in managing project resources. We will:
* Send frequent usage updates to project representatives
* Provide resource usage estimates for commonly used model configurations
* Liaise with NCI to set quarterly compute resources, and modify resources within quarters if required
* Give generic and specific advice to users in managing their resources

[[include component="page" wikiName="climate-cms" page="nci_guidelines_index" editable="1"]]

NCI Guidelines Compute Allocations

2017-11-28T12:05:31Z

ClaireCarouge: Imported from Wikispaces

Jobs running on the login nodes or on the VDI do not use any compute time allocation. Jobs running on the login nodes have low limits on time and resources they can use. The VDI allows you to have multiprocessor jobs ongoing for up to 7 days without using any of the compute time allocation.

The compute time allocation is measured in SU. You only use compute time when running batch jobs. The amount of SU your job uses depends on the queue and the number of processors. The cost ranges from 1 SU per CPU-hour (walltime) to 3.75 SU per CPU-hour (walltime) depending on the queue the job is submitted. Details can be found on the [https://opus.nci.org.au/display/Help/Raijin+User+Guide#RaijinUserGuide-JobSubmission | NCI user guide].

Once the compute allocation is totally used, the project can use bonus time. In that case, all jobs are automatically moved to the normal queue and get a priority disadvantage.
At the Centre, we try to avoid the Centre's projects to run The CMS team can then ask NCI to shift SUs between projects.

[[include component="page" wikiName="climate-cms" page="nci_guidelines_index" ]]

NCI Guidelines home

2017-11-28T05:42:16Z

ClaireCarouge: Imported from Wikispaces

NCI's /home data space is your own space at NCI. It is small, only 2GB per user but it is backed up.

/home is intended for storage of small files that are difficult to recreate. It contains the system configuration files (.bashrc, .cshrc, .profile, .login). It is also good for source code. By default, this space is only accessible by you. You are free to change permissions a

Key features of /home are:
* Access from Raijin only (no access from VDI)
* Backups
* Small (2GB only)

[[include component="page" wikiName="climate-cms" page="nci_guidelines_index" editable="1"]]

NCI Guidelines Requesting Access

2017-11-28T04:46:37Z

ClaireCarouge: Imported from Wikispaces

Access to NCI facilities can be requested through the [https://my.nci.org.au/mancini/signup | NCI User Portal]

In order to access any resources or log into the supercomputer you will first need to join a [[NCI Guidelines Projects | Project]]. Your first project needs to be a project associated with compute time allocation.

Please read the [https://opus.nci.org.au/display/Help/How+to+create+a+user+account | NCI User Guide] for requirements. Please read the [https://opus.nci.org.au/display/Help/Raijin+User+Guide | Raijin User Guide] to familiarise yourself with the supercomputer environment.

[[include component="page" wikiName="climate-cms" page="nci_guidelines_index" editable="1"]]

River runoff regridding for MOM

2017-07-26T03:56:11Z

ClaireCarouge: Imported from Wikispaces

[https://github.com/OceansAus/runoff_tools | River runoff regridding tools for MOM] written by Russ Fiedler for COSIMA have been installed in the [https://github.com/coecms/easybuild-configs | EasyBuild software environment].

See this [https://github.com/aidanheerdegen/regrid_runoff/blob/master/regrid_runoff | example] for how they are used.
[[Category:mom]][[Category:runoff]][[Category:river]][[Category:regrid]]

MOM

2017-07-26T03:54:03Z

ClaireCarouge: Imported from Wikispaces

MOM (Modular Ocean Model) is an ocean climate model developed at GFDL, and is the ocean component of the ACCESS climate model system. The latest release of MOM is version 5.0.2.

=Guides=
* [[Acquiring and compiling MOM5]]
* [[Compiling MOM6 | Acquiring and compiling MOM6]]
* [[Running MOM]]
* [[Configuring MOM]]
* [[River runoff regridding for MOM | Regridding river runoff data]]
* [[MOM output diagnostics]]
* [[Designing a MOM experiment]]

=External resources=

These sites provide the most up-to-date information on MOM development

* [http://www.mom-ocean.org | MOM Development Lab]: Home page for MOM development
* [https://groups.google.com/forum/#!forum/mom-users | MOM Users Mailing List]: Hosted by Google Groups
* [https://github.com/BreakawayLabs/mom | MOM Sourcecode Repository]: Hosted by github

The sites listed below are less active, but contain useful information for running MOM5:

* [http://www.gfdl.noaa.gov/ocean-model | GFDL Ocean Circulation Models]: Summary of ocean modelling work at GFDL
* [http://data1.gfdl.noaa.gov/~arl/pubrel/r/mom4p1/src/mom4p1/doc/mom4_manual.html | MOM4p1 User Guide]: Brief summary of MOM4 usage
* [http://data1.gfdl.noaa.gov/~arl/pubrel/r/mom4p1/src/mom4p1/doc/guide4p1.pdf | Elements of MOM4p1]: Physical and numerical documentation of MOM4 (PDF document)
* [[ftp://ftp.gfdl.noaa.gov/perm/MOM4/mom4p1_pubrel_dec2009/exp/ | Sample MOM4 Experiments]]: Includes configuration and data input fields, and sample output (Configured for the MOM4p1 December 2009 release)
[[Category:mom mom4 ocean model]][[Category:Main Menu]]

MPAS

2017-02-14T23:45:06Z

ClaireCarouge: Imported from Wikispaces

The Model for Prediction Across Scales (MPAS) is a collaborative project for developing atmosphere, ocean and other earth-system simulation components for use in climate, regional climate and weather studies. [[http://mpas-dev.github.io/]]. The Atmospheric component of the model is now available for use on Raijin.

=MPAS-Atmosphere=
----
==Building==

===Pre-built Module===
A pre-built version of MPAS is available in the ACCESS modules:
<syntaxhighlight>
module use ~access/modules
module load mpas/5.0
</syntaxhighlight>

Once the module is loaded the data files are located in the directory $MPAS_DATA:
<syntaxhighlight>
$ ls $MPAS_DATA
CAM_ABS_DATA.DBL
CAM_AEROPT_DATA.DBL
GENPARM.TBL
LANDUSE.TBL
...
</syntaxhighlight>

===Building MPAS===
ARCCSS CMS maintains a branch of MPAS at https://github.com/coecms/MPAS-Release. This branch has some minor changes required to build the model on Raijin. To set up the build environment:
<syntaxhighlight>
$ git clone https://github.com/coecms/MPAS-Release
$ cd MPAS-Release
$ source environment.raijin
</syntaxhighlight>

You can then build individual cores with e.g. make ifort CORE=atmosphere, see the [http://www2.mmm.ucar.edu/projects/mpas/tutorial/mpas_atmosphere_users_guide_5.0.pdf | User's Guide] for full details

==Running==

See the [http://www2.mmm.ucar.edu/projects/mpas/tutorial/mpas_atmosphere_users_guide_5.0.pdf | User's Guide] for details on how to run the model. A number of idealised test cases as well as meshes for different resolutions are available from the [http://mpas-dev.github.io/atmosphere/atmosphere_download.html | MPAS download site]. The $MPAS_DATA directory has all of the TBL, DBL and namelist files required to run MPAS
<syntaxhighlight>
cp $MPAS_DATA myrun
cd myrun
# Edit configuration then qsub `run-atmosphere.pbs`
</syntaxhighlight>

==Resources==

[http://mpas-dev.github.io/ | MPAS Home Page]
[http://www2.mmm.ucar.edu/projects/mpas/tutorial/mpas_atmosphere_users_guide_5.0.pdf | Atmosphere User's Guide]

[https://github.com/coecms/MPAS-Release | ARCCSS MPAS Github]
[https://github.com/MPAS-Dev/MPAS-Release | Main MPAS Github]

MediaWiki:Sidebar

2017-02-14T03:19:49Z

ClaireCarouge: Imported from Wikispaces

NU-WRF run LIS offline

2017-02-14T01:08:52Z

ClaireCarouge: Imported from Wikispaces

You are now ready to start running your simulations. Go into your DECK_DIR as you have specified in the input_scripts.txt file before creating the decks. There you should find files named: run_LIS_YYYY_MM. For the testcase, there will only be run_LIS_1998_06 as the spin-up is only 1 year and we run the year in one go.
Before running you may want to check the deck. Make sure the PBS directives look right (check the email address!). Check the run directory as it appears in the first "cd" command. You may want to check you have a LIS executable in your LIS run directory. Once you are confident everything is setup correctly, you can either launch the run_LIS_1998_06 from the deck directory or link to it in your LIS run directory and start it from there. Note that if there are several decks, subsequent decks are automatically started in your LIS run directory.

To run the testcase simulation, simply type:
<syntaxhighlight>
./run_LIS_1998_06
</syntaxhighlight>
This should schedule 3 jobs to the PBS queues.

For longer simulations with several decks, the last PBS job (the post-processing job) will automatically fetch and launch the next deck until no next deck is found.

[[NU-WRF Prepare decks | Previous: NU-WRF Prepare decks]]
[[NU-WRF run WPS | Next: NU-WRF run WPS]]

NU-WRF Run WPS

2017-02-14T01:08:39Z

ClaireCarouge: Imported from Wikispaces

One of the main difference between running WRF stand-alone and WRF coupled with LIS is with creating the boundary conditions and initialisation files. When coupled to LIS, real.exe was modified to use information from a LIS output file to combine with the metgrid.exe outputs so that fields between the offline spin-up and the coupled run are coherent.
As such, '''one needs to have finished the LIS spin-up''' before creating the boundary conditions and initialisation files.

Once your spin-up is done, go into your DECK_DIR, then look for the file getbdy_grib_1999_06.pbs (for the testcase). You need to submit it to the queues:
<syntaxhighlight>
qsub getbdy_grib_1999_06.deck
</syntaxhighlight>

[[NU-WRF run LIS offline | Previous: NU-WRF run LIS offline]]
[[NU-WRF Run WRF | Next: NU-WRF Run WRF]]

NU-WRF Run WRF

2017-02-14T01:07:57Z

ClaireCarouge: Imported from Wikispaces

Once you have the boundary conditions and initial states, you can start running WRF. Again, you can run from the DECK_DIR or from your WRF running directory. The decks for WRF are named according to the format: runwrf_YYYY_MM_DD.deck. Note that all the WRF outputs will be in the WRF run directory and if there are several decks all subsequent decks will be run from the WRF run directory.

You might again want to check the PBS directives and maybe also the namelist.input options in the first deck before launching the simulation. To start the simulation for the testcase, simply type:
<syntaxhighlight>
./runwrf_1999_06_01.deck
</syntaxhighlight>

[[NU-WRF Run WPS | Previous: NU-WRF Run WPS]]

NU-WRF workflow

2017-02-14T00:51:03Z

ClaireCarouge: Imported from Wikispaces

First here is a schematic of the current workflow.
[[File:NU-WRF workflow schematic_2.jpg|720x102px|center]]
The CMS team has created scripts to automate the workflow as best as possible. You will first need to clone these to your own space on your machine. The scripts are [https://bitbucket.org/ccarouge/nu-wrf_scripts | here]. This is a public repository and you do not need any special permission to use it. Note that this guide is written for using the scripts on raijin but one can use the scripts on any machine.

The best way to explain the workflow is to look at a test case for a coupled NU-WRF simulation. The test case data is stored with the LIS dataset under /g/data1/w35/LIS/coupled_testcase.
# [[NU-WRF Creation of the domain | Creation of the domain]] and regridding the LIS input dataset.
# [[Prepare decks | Prepare decks]]
# [[NU-WRF run LIS offline | Run LIS offline]]
# [[NU-WRF Run WPS | Run WPS]]
# [[NU-WRF Run WRF | Run NU-WRF]]

NU-WRF Prepare decks

2017-02-14T00:36:47Z

ClaireCarouge: Imported from Wikispaces

Let's first start with a description of the files in NU-WRF_scripts/prepare_decks
{|
| MODEL_OUTPUT_LIST.TBL_d01 || LIS file to choose the output diagnostic and their format (units, averaging...) It is currently formatted to output all outputs required as input for WRF.
|-
| Makefile || This is to manage WPS with ERA-Interim GRIB files
|-
| getbdy_grib.deck || This is to manage the creation of WRF boundary condition and input files with GRIB files. Users may want to modify PBS options.
|-
| getbdy_inter.deck || This is to manage WPS and the creation of WRF boundary condition and input files starting with intermediate files. Users may want to modify the PBS options.
|-
| getbdy_mk35.deck || This is a historical file. It can give ideas on how to manage WPS and real.exe with inputs coming from a GCM.
|-
| input_scripts.txt || This file has the user inputs for the scripts
|-
| namelist.wps || This is the namelist.wps used for the testcase.
|-
| template_lis.config || This is a template lis.config file (configuration file for LIS). This is where users need to modify physic options
|-
| template_lis.config_2nests || This is an example configuration file for LIS with 2 nests.
|-
| template_run_LIS.csh || This is a template for the decks to run LIS offline. Users may want to modify the PBS options.
|-
| template_runwrf_raijin.deck || This is a template for the decks to run WRF or NU-WRF. Users may want to modify the PBS options.
|-
|}
All the files are setup to run the testcase. You simply need to modify the following in input_scripts.txt:
* the email address
* the project
* the paths to the codes: LIS, WPS and WRF
* the paths to store the decks and outputs: GEO_DIR, LISOUT_DIR, WPSOUT_DIR, WRFOUT_DIR, WRFRES_DIR, DECK_DIR. Note the user is responsible to create all these directories. Also you need to make sure the geo_em.d01 file created at the previous step is in the GEO_DIR directory.
All the codes are under NU-WRF_scripts/prepare_decks/codes. These are in python and you need to load a python module first. Any module will work. Then under NU-WRF_scripts/prepare_decks/codes, type:

<syntaxhighlight>
python prepare_LIS.py
python prepare_WPS.py
python prepare_WRF.py
</syntaxhighlight>
The first command prepares the decks for the spin-up of LIS offline. The second prepares the decks for WPS and the third prepares the decks for WRF. Note that the order of the commands matters. These commands should create a few files in DECK_DIR.

=Set CO2 concentration for LIS spin-up=
----
When coupled to WRF, CABLE will get the CO2 concentration used in the radiation scheme through LIS. But when offline, CABLE uses the CO2 concentration listed in the CABLE options in lis.config file. Considering the spin-up for CABLE can span a long period of time (30+ years) it can be interesting to use a time-varying CO2 concentration. Unfortunately this hasn't been integrated to the code creating the decks yet and it has to be handled separately. Also it is important to note the current code to update the CO2 concentration assumes LIS is going to be run month by month, which means you need to create the decks with the LIS_nmonths option in input_scripts.txt equals to 1 if you wish to update the CO2 concentration during the spin-up.

To update the CO2 concentration, you need to update the inputs in update_co2_in_lis_config.ncl in NU-WRF_scripts/prepare_decks/codes/. Then run it with
<syntaxhighlight>
ncl update_co2_in_lis_config.ncl
</syntaxhighlight>
Make sure to load an ncl module beforehand.

[[NU-WRF Creation of the domain | Previous: NU-WRF Creation of the domain]]
[[NU-WRF run LIS offline | Next: NU-WRF run LIS offline]]

NU-WRF Creation of the domain

2017-02-13T23:25:17Z

ClaireCarouge: Imported from Wikispaces

=Domain setup=
----
Since LIS and WRF need to run on the exact same grid, the first thing to setup for NU-WRF is the grid and corresponding input datasets. The creation of the domain and the grid is relatively simple as it is the same as for WRF alone. To get an in-depth explanation please refer to the [http://www2.mmm.ucar.edu/wrf/OnLineTutorial/Basics/GEOGRID/index.html | WRF tutorial]. The steps to follow are:
# Create a run directory for LIS anywhere you want. Then compile LIS only and copy the executable, LIS, from WRFV3/lis/make to the run directory (cp WRFV3/lis/make/LIS run_lis/.)
# Compile WRF and WPS. It's best to compile WRF with the lisreal option to enable the use of LIS else you'll need to recompile later.
# In WPS, update the namelist.wps to the domain you want. You may want to use util/plotgrids.ncl to help you design your grid.
# Run geogrid.exe
# Save the geo_em.dxx.nc files carefully as they are essential to share grid information between LIS and WRF.

Here are a few limitations and tips to keep in mind when designing your domain:
* You can not use a rotated lat-lon grid as LIS does not support those.
* Use the MAPFAC* variables output by WRF to ensure your grid is not too stretched in one or both directions. These should be close to 1 ideally. This is important for the code stability which it means being able to use larger time steps and hence to have faster simulations.
==Testcase==
To run the testcase, make sure the namelist.wps file is the same as in the script directory NU-WRF_scripts/prepare_decks/. in step 3.

=Input datasets=
----
All input data available for LIS is stored under /g/data1/w35/LIS/LIS_PARAMS
The global meteorological forcing data is under FORCING. It is advised to avoid using GDAS forcing over Australia.
There is some MERRA forcing available for Australia under SUPP_FORCING. This dataset has been bias-corrected for the precipitation.
All non-meteorological gridded inputs are under UMD. The highest resolution we have is 1 KM, more exactly it is on a lat-lon grid at 0.001° resolution (same naming convention for other resolutions).
All the *_parms directories contain the LSM's specific input (e.g. soil and vegetation parameter files).

=Interpolating the input dataset.=
----
To speed up simulation time, one has to interpolate the non-meteorological fields at the required resolution beforehand rather than during simulation. However, LIS has basic interpolation built-in so it is not necessary to interpolate to the exact grid of the simulation. But it is better to interpolate the input fields to the same resolution as the simulation domain or at least at a close enough resolution.
We are currently storing the input dataset in a few resolutions (including the 50KM resolution needed for the testcase) and we also provide the interpolation scripts to the users if they need to create new resolutions. If you think a new resolution of the dataset would be useful for several users, please contact [[mailto:climate_help@nci.org.au | climate_help]] in order to add it to our collection.

Once you have cloned the repository with the scripts. Go into NU-WRF_scripts/RegridLIS. You will need to open and modify the regrid_all.csh file. You need to at least give the resolution (res), the output directory (outdir) and the input directories (indir and indir100 for PRINCETON dataset that is at 100KM). The script is setup to interpolate all datasets one by one. To speed up the interpolation process, you may want to comment out the datasets that won't be used by your LSM(s).
Also, some interpolation programs accept command-line options which you might wish to change from the default. Refer to each interpolation program to know which options are available.

[[NU-WRF workflow | Previous: NU-WRF workflow]]
[[NU-WRF Prepare decks | Next: NU-WRF Prepare decks]]

NU-WRF

2017-02-13T04:30:32Z

ClaireCarouge: Imported from Wikispaces

NU-WRF stands for NASA Unified WRF. This is an augmented WRF code with additional software developed at NASA. This WRF version has been primarily chosen in order to run CABLE coupled to WRF through the LIS software. There are additional softwares distributed along WRF in NU-WRF:
* ARWpost
* GSDSU
* LVT
* MET
* RIP4
* UPP

These are still distributed on NCI machine but they have not been ported or tested by the CMS team. 

=Manuals=
----
Following are the links to the official relevant manuals for each software:
* [http://lis.gsfc.nasa.gov/LIS_documentation.php | LIS]
* [http://www2.mmm.ucar.edu/wrf/users/ | WRF]
* [https://trac.nci.org.au/trac/cable/wiki | CABLE]

=NU-WRF installation=
----
==Accessing NU-WRF==
LIS is under licence. To restrict access to the NU-WRF code, it was set as a private repository on [https://bitbucket.org/ | BitBucket]. To gain access to the repository, send an email to [[mailto:climate_help@nf.nci.org.au | climate_help]] to request the connection including the email address you want to link to BitBucket. You should then receive an invitation to join the NU-WRF repository.

==Installing NU-WRF==
===Clone the repository.===
Once you have access to the BitBucket repository, you need to clone the repository onto the NCI machine. There are two ways to clone from BitBucket, you can either use a HTTPS link or a SSH link. Both work with NCI machine. With HTTPS, you will need to provide your BitBucket password at each connection. For SSH, you will need to first create a SSH key but then you can setup the system to retrieve the key automatically without entering it (see [https://confluence.atlassian.com/display/BITBUCKET/Set+up+SSH+for+Git | BitBucket help]). Cloning the repository should create a local repository in your file system called nu-wrf.

===Compilation===
Since NU-WRF has been ported to NCI, you should be able to compile the WRF/WPS components of NU-WRF straight away. The code comes with the script nu-wrf/run_build. Please review the script to choose the appropriate compilation options (you can compile LIS alone '''or''' WRF alone '''or''' WRF coupled to LIS '''and/or''' WPS) and PBS options.
For a successful compilation, the output script should contain a line reading:
<syntaxhighlight>
Completed build of <component_name> component of NU-WRF!
</syntaxhighlight>
where <component_name> is replaced by the name of the compiled component. Note that if you ask for compiling two components at once (e.g. WRF and WPS), you will need to search for the "Completed" line for both components.

=Running WRF with LIS=
----
Normally to run a real simulation with WRF, the user has to create the boundary condition files with WPS then run WRF. Using LIS modifies the workflow in several ways (see [[NU-WRF workflow | graph]]):
* Interpolate the LIS input data to the grid resolution used for your WRF simulation.
* Run LIS offline (i.e. not coupled to WRF) on the exact same grid that will be used for the WRF simulation. You are required to save a certain number of fields that will be used to create the input conditions for WRF. This phase is called the spin-up.
* real.exe has been modified and now queries input from both WPS output and a LIS output file.
* wrf.exe will expect a LIS restart file and a LIS configuration file.
Scripts have been developed by the CMS team to facilitate the management of simulations with NU-WRF on NCI machines, including to run the spin-up and ensure the concordance of information between WRF and LIS configuration files. It is assumed that a user has access to NCI servers. However, since the workflow is based on template files, these are easily customisable to use on any machine. These also provide the possibility to store inputs and outputs on a different machine and to automatically move the files across machines for each run.
The scripts are in a public Git repository on Bitbucket: [https://bitbucket.org/ccarouge/nu-wrf_scripts | NU-WRF scripts].

See the [[NU-WRF workflow]] page for instructions about these scripts. These instructions will run you through testcases for CABLE through LIS only and CABLE-LIS-WRF.

MOM FAQ

2016-12-06T23:15:51Z

ClaireCarouge: Imported from Wikispaces

=Problem with collating outputs with payu=
----
Your simulation runs correctly but the collating of the outputs does not work and you have this error message in the log file:
<syntaxhighlight>
Traceback (most recent call last):
File "/jobfs/local/pbs/mom_priv/jobs/268316.r-man2.SC", line 9, in <module>
collate_cmd.runscript()
File "/apps/payu/0.6/lib/payu/subcommands/collate_cmd.py", line 93, in runscript
expt.collate()
File "/apps/payu/0.6/lib/payu/experiment.py", line 606, in collate
model.collate()
File "/apps/payu/0.6/lib/payu/models/fms.py", line 78, in collate
assert mppnc_path
AssertionError

</syntaxhighlight>
This probably means you don't have a mppnccombine executable in you mom/bin directory. If you need such a file please contact climate_help@nci.org.au to know where to find it. Include a copy of the error message with your email please.

WRF FAQ

2016-08-24T03:04:44Z

ClaireCarouge: Imported from Wikispaces

=Tips to reduce WRF outputs size=

* '''netCDF4 compression''': you are probably using netCDF v4.0 or newer. In that case, you can enable compression directly in WRF code so the output is compressed at creation time. If you ever want an output file in classic format, you can then use the namelist option ''use_netcdf_classic=.true.'' in the &time_control section. The choice of netcdf happens at the configuration step so if you already have compiled the code, clean it up with
> <syntaxhighlight>
clean -a
</syntaxhighlight>
> then follow these steps:
** Define the NETCDF4 environment variable:
>> for csh
>> <syntaxhighlight>
setenv NETCDF4 1
</syntaxhighlight>
>> for bash
>> <syntaxhighlight>
export NETCDF4=1
</syntaxhighlight>
** Configure and compile as usual.
* '''output a subset of variables:''' WRF comes with a built-in mechanism to choose which variables to output or not. So you don't have to output all the default variables (and you can add some non-default ones). The details can be found in the [http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3.8/users_guide_chap5.htm#runtimeio | WRF User's guide]
* '''clean up the output file:''' After creation, it can be useful to clean up the output file. For example, all variables have a Time dimension even if they are constant in time (latitudes and longitudes are usually constants unless you use a moving nest). So it can save storage space to remove this dimension. For example by using NCO:
> <syntaxhighlight>
ncwa -a Time -v XLONG wrfout_d01_2000-01-24_12\:00\:00 time0.nc
ncks -x -v XLONG wrfout_d01_2000-01-24_12\:00\:00 no_longitude.nc
ncks -A -v XLONG time0.nc no_longitude.nc
mv no_longitude.nc wrfout_d01_2000-01-24_12\:00\:00
</syntaxhighlight>
Obviously you can list more than 1 variable at once in the commands.

=What to do if WRF does not compile on Raijin?=
You need to make sure that you are using the WRF version that is stored on /projects/WRF on Raijin. Then you should first try to compile using the dmpar option. For this, clean (./clean -a) and run configure again and choose option #3 (dmpar). Then WRF should compile the code without problems. WRF has not yet been successfully compiled on Raijin using the other options.

=Which processor crashed in my WRF simulation?=
WRF output the standard output in rsl.out and the error in rsl.error. There is a pair of files for each processor. If you are running on a large number of processors, checking each file is very time-consuming.
The first thing to do when your simulation stops is to check the output file for your script. If your script to launch WRF is called script.pbs, then at the end of the job PBS will create a script.pbs.o1234567 where "1234567" is to be replaced by your job ID. Open this file and check the end. If you see a message like:
<syntaxhighlight>
--------------------------------------------------------------------------
mpirun has exited due to process rank 50 with PID 32659 on
node r2059 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

</syntaxhighlight>
Then it means the simulation did not finish normally. The first line of the message gives the processor number that finished abnormally: "due to '''process rank 50'''". So in this case, the problem happened on the processor 50 and the error message should then be located in rsl.error.0050. Note you might want to check rsl.out.0050 as well just in case a message was written to the output first.

=What to do if I get a segmentation fault error with WRF?=
If in rsl.error you get a message like this one:
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
wrf.exe 00000000017F59A1 Unknown Unknown Unknown
wrf.exe 00000000017F3655 Unknown Unknown Unknown
wrf.exe 0000000001ACFA07 Unknown Unknown Unknown
wrf.exe 0000000001387C8A Unknown Unknown Unknown
wrf.exe 0000000000EC4CDD Unknown Unknown Unknown
wrf.exe 0000000000DCCAB7 Unknown Unknown Unknown
you have a segmentation fault error. These can have multiple causes, the best way to find out what is causing the error is to recompile WRF using the debugging options and re-run.
# Clean the previous compilation in WRFV3/ with ./clean -a
# Copy configure.wrf.backup to configure.wrf
# Open configure.wrf and search for FCDEBUG. Uncomment the options by removing the #. If there are 2 #s signs, the best is to remove both unless you know what you are doing.
# Compile (DO NOT run configure) and run again. Check the error message at the end of the run and hopefully it should be more precise.

LIS and NUWRF FAQ

2015-12-10T22:34:34Z

ClaireCarouge: Imported from Wikispaces

=My LIS outputs are empty=
----
Please check what filename you have for the following option in lis.config:
<syntaxhighlight>
Model output attributes file
</syntaxhighlight>
Then make sure you have this file in your run directory. And run again.

NU-WRF set the environment

2015-12-03T03:51:57Z

ClaireCarouge: Imported from Wikispaces

[[toc | flat]]
=Access to the LIS dataset=
----
LIS comes with its own dataset for static data (topography, vegetation and soil types, etc) and some time-varying datasets (LAI, etc.). This dataset is stored at NCI but as LIS it is under license. In order to access it, you will need to ask access to it by emailing [[mailto:climate_help@nf.nci.org.au | climate_help]]. Access is granted to all staff or student working at one of the ARCCSS participating Universities and the ARCCSS partner organisations.

=Setup command restricted ssh-keys=
----
**If you are using a storage machine outside NCI''', you will need to transfer data to and from this machine automatically. '''If you are only using NCI machine''', please skip this part! The transfers will be done using the copyq queue of the scheduler on raijin which means that an ssh-agent forwarding can not be used (it is disabled for the queues). Hence to enable a password-less transfer, you will need to create a specific pair of ssh keys. This pair is special as it has an empty passphrase but is protected by limiting its usage to file transfers only. Note: this method requires that your storage machine has rrsync installed.

==Choose an archiving folder==
The rrsync command on raijin has been configured to refuse to move back up the folder tree on the remote host. That is you can not use "../" in the path to your data storage. You can also only give relative paths and not absolute paths. It is then important to carefully choose the default path for rrsync. If you do not give a path, rrsync will by default connect to your home directory on the remote machine which is usually small and not used for storage. So you want rrsync to connect to the root of your storage area. Once you have chosen the path, please add it as an environment variable named ${RSYNC_PATH} (do not change the name) on your storage machine. To do that:
* If you use bash shell, add to your ~/.bashrc file:
> <syntaxhighlight lang=bash>
export RSYNC_PATH=<path you have chosen>
</syntaxhighlight>
* If you use tcsh shell, add to your ~/.cshrc file:
> <syntaxhighlight lang=tcsh>
setenv RSYNC_PATH <path you have chosen>
</syntaxhighlight>
<path you have chosen> is to be replaced by the path you want. For UNSW, we suggest using /srv/ccrc. Then when using rsync you only give the path relative to ${RSYNC_PATH}.

==Setup the keys==
This is explained in details by NCI on [http://nci.org.au/services-support/getting-help/using-ssh-keys/ | this page]. On the documentation:
* "MyDesktop" refers to your storage machine
* /data/archive is to be replaced by ${RSYNC_PATH}.

==How to use the ssh keys==
===Transfer from raijin to storage machine===
Let's say your storage machine is called: maelstrom.ccrc.unsw.edu.au and your login on this machine is z3368490. You want to transfer the EXPDIR/ directory from raijin to maelstrom in the directory: ${RSYNC_PATH}/TEST/. You first have to create the ${RSYNC_PATH}/TEST/ directory on maelstrom. Then on raijin, type:
<syntaxhighlight lang=bash>
rsync -vrlpt ./EXPDIR -e “ssh -i $HOME/.ssh/id_rsa_file_transfer” z3368490@maelstrom.ccrc.unsw.edu.au:TEST/.
</syntaxhighlight>
Note the destination path is only TEST/ since ${RSYNC_PATH} is specified in the restricted command in authorized_keys file and there is no space between “TEST/” and “.”. You need a space before z3368490. The part -e “ssh -i $HOME/.ssh/id_rsa_file_transfer” has to be just before the remote path so the name of the directory on raijin is '''before''' the “-e” option! The “./”before EXPDIR is not requested but it is highly recommended to use it.
===Transfer from storage machine to raijin===
Now, if you want to get the ${RSYNC_PATH}/TEST/EXPDIR from maelstrom to raijin, you still need to be on raijin and type:
<syntaxhighlight lang=bash>
rsync -vrlpt -e "ssh -i $HOME/.ssh/id_rsa_file_transfer" z3368490@maelstrom.ccrc.unsw.edu.au:TEST/EXPDIR .
</syntaxhighlight>
Again there is a space before z3368490. Note the destination directory on raijin is at the end (the last “.”)

=Modules and environment variables=
----
To run NU-WRF you need to load, on raijin and the storage machine, the modules to access the Fortran compiler, OpenMPI (for parallelisation), ESMF v3.1.0rp3, SZIP and the netCDF libraries. These modules are loaded by default in the scripts provided for compiling and running the code. In addition, some scripts are written in Python (v2 not v3) and thus the latest Python 2 library should be loaded on the storage machine. The codes are under version control using Git so you will need to make sure Git is installed on your machine. It is accessible by default on raijin.

You also need to define a few environment variables for LIS and WRF. The following tables give the environment variables for bash and csh shells on different machines. If you are a member of this wiki, feel free to add information for an other machine.

Notes:
* NU-WRF has only been tested on raijin using the Intel Fortran compiler. If you plan on using an other compiler you are responsible for modifying the environment variables and scripts as necessary.
* If you are using raijin to run WPS as well as WRFV3, you need to load the additional modules listed on CCRC machines to raijin (Python and SZIP).

{|
|||!! On Raijin
|-
|''' bash shell, in file: .bashrc ||''' csh/tcsh shells, in file: .cshrc ||
|-
| export LIS_ARCH'''linux_ifc || setenv LIS_ARCH "linux_ifc" || This variable is for LIS. ifc stands for Intel Fortran Compiler 
|-
| ulimit -s unlimited || limit stacksize unlimited || For WRF 
|-
| export WRFIO_NCD_LARGE_FILE_SUPPORT'''1 || setenv WRFIO_NCD_LARGE_FILE_SUPPORT'''1 || Not necessary but highly recommended. Everything on one line. 
|-
| export WRF_ESMF_LIBS_MPI'''"-L/apps/esmf/3.1.0rp3-intel/lib/lib0/Linux.intel.64.openmpi.default -lesmf -lstdc++ -lrt" || setenv WRF_ESMF_LIBS_MPI "-L/apps/esmf/3.1.0rp3-intel/lib/lib0/Linux.intel.64.openmpi.default -lesmf -lstdc++ -lrt" || Note the path is machine dependant. Everything on one line. 
|-
|}

{|
|||!! On CCRC machines
|-
|''' bash shell, in file: .bashrc ||''' csh/tcsh shells, in file: .cshrc ||
|-
| export LIS_ARCH'''linux_ifc || setenv LIS_ARCH "linux_ifc" || This variable is for LIS. ifc stands for Intel Fortran Compiler
|-
| ulimit -s unlimited || limit stacksize unlimited || For WRF
|-
| export WRFIO_NCD_LARGE_FILE_SUPPORT'''1 || setenv WRFIO_NCD_LARGE_FILE_SUPPORT'''1 || Not necessary but highly recommended. Everything on one line.
|-
| export WRF_ESMF_LIBS_MPI'''"-L/share/apps/esmf/intel/ 3.1.0rp3/lib/lib0/Linux.intel.64.openmpi.default -lesmf -lstdc++ -lrt" || setenv WRF_ESMF_LIBS_MPI "-L/share/apps/esmf/intel/ 3.1.0rp3/lib/lib0/Linux.intel.64.openmpi.default -lesmf -lstdc++ -lrt" || Note the path is machine dependant. Everything on one line.
|-
|}

How to run WRF with inputs in netcdf format

2015-09-24T05:49:07Z

ClaireCarouge: Imported from Wikispaces

By default, metgrid.exe reads in only what is called intermediate format. WPS only comes with a conversion tool from GRIB to intermediate format.
In the climate community, we use the netCDF file format more than the GRIB file format. So some of you might have to write their own interpreter from netCDF to intermediate format.

You can find a good example of an interpreter in this git repository: [[https://github.com/coecms/Soil_Moisture_WRF]]
Note that this case was to create an intermediate file with only 1 variable in it and the configuration has to be done within the source file for now.

CABLE Data

2015-08-04T23:52:00Z

ClaireCarouge: Imported from Wikispaces

=Meteorological forcing datasets=
----
It is now possible to access input forcing datasets for CABLE offline simulations on raijin. 
These data is part of the RDSI climate datasets collection and can be accessed by members of the cable group (you become a member of the cable group when you register to have access to the source code). 
For details see the [https://trac.nci.org.au/trac/cable/wiki/CableData | CABLE wiki] 

One of the forcing dataset is GSWP3 (Global Soil Wetness Project Phase 3) for which you can find information [http://hydro.iis.u-tokyo.ac.jp/GSWP3/ | here]. 

=Evaluation / Benchmarking datasets=
----
We are in the process of preparing a collection of Evaluation/Benchmarking datasets, which will be available in the same way.

LDT

2015-07-17T05:47:20Z

ClaireCarouge: Imported from Wikispaces

LDT stands for LIS Data Tools. This is the pre-processing package for LIS. From LIS v7.1 it is necessary to pre-process the input parameters through LDT before running LIS.

=LDT is not an efficient regridding tool=
----
In LDT configuration file, you can give a series of parameters that will be used by the land surface model in LIS. Each parameter file can be on its own grid and LDT will interpolate the data to the correct grid. However, LDT is not suitable to work with data that is at a lot higher resolution than the target grid. For example, if you want to run LIS on a 100 km resolution grid, if your input file is at 1 km grid, you should not run LDT directly on this file. You are better off using the regridding routines developed at the Centre (email climate_help for information and access) to get the data to a generic 100 km grid. Then you use LDT to transfer the 100 km resolution input to the exact grid you want to run LIS on.

=LDT for CABLE=
----
Currently CABLE needs only the following gridded parameters:
* land cover types: IGBP
* soil fraction: FAO
* albedo
* LAI
Here is an example of a LDT config file to run with CABLE. Note that we are using AVHRR LAI. LIS comes with an option to run with MODIS LAI, this LAI is directly read in LIS as the data is inter-annual but it has not been tested with CABLE yet.

UM and ACCESS FAQ

2015-02-24T22:41:20Z

ClaireCarouge: Imported from Wikispaces

=Who can gain access to the UM and the ACCESS models?=

The full ACCESS model is made up of a number of sub-models for different processes - UM for the atmosphere, MOM for the ocean and CABLE for the land surface. Each of these components was created by different groups with different conditions of use.

The Met Office Unified Model is proprietary software released under a commercial licence. The Australian University community has a sub-licence to use the UM from CAWCR, managed by ARCCSS. Australian researchers outside of ARCCSS may request access to the model by emailing access_help@nf.nci.org.au. Note that our licence is restricted to running the model at NCI and does not permit sharing of model code outside of the UM partnership.

GFDL's Modular Ocean Model is open-source software available under the GPL. Any researcher is free to use, modify and share the model code.

CSIRO's CABLE model is proprietary software. See the [http://cawcr.gov.au/projects/access/cable/cable_licence_agreement.pdf | licence agreement] for conditions of use.

=What to do to gain access to the UM and the ACCESS models?=

ARCCSS researchers can get access to the UM and ACCESS models by contacting the CMS team at climate_help@nf.nci.org.au. Researchers outside of ARCCSS should contact the general ACCESS helpdesk at access_help@nf.nci.org.au. Using the models requires an account at [http://nci.org.au/ | NCI], the Australian national supercomputing centre.

=What to do to use the software in /projects/access/bin?=

Several tools useful for working with ACCESS are available at NCI under the path /projects/access/bin. These tools include output viewers and converters between the UM file format and standard formats like NetCDF and GRIB. To access these tools you must be registered as an ACCESS user (see above).

New UM User message

2014-09-17T00:54:45Z

ClaireCarouge: Imported from Wikispaces

Welcome to the ACCESS community!

The CoE for Climate Systems Science's Computational Modelling Systems team provides support with ACCESS models to the Australian university community. Email [[mailto:climate_help@nf.nci.org.au | climate_help@nf.nci.org.au]] to get in touch with us.

Model runs are configured on the server 'accessdev.nci.org.au', which you can access using your NCI account. This system is behind a firewall, to log on from your local workstation you will need a static IP address (talk to your local IT support to arrange this). Send this address to [[mailto:climate_help@nf.nci.org.au | climate_help@nf.nci.org.au]] to be allowed through the firewall. Note that the university community is only licensed to run the model on NCI computers.

Documentation for using the ACCESS model is available at the CMS Wiki: http://climate-cms.unsw.wikispaces.net. An account is not necessary to view the wiki, if you do apply for an account please let us know who it's for, either by adding a note to the account creation form or emailing the helpdesk.

The UK Met Office's Unified Model is the atmospheric component of ACCESS. The Met Office has a wiki at http://collab.metoffice.gov.uk, you can also join their mailing list by emailing [[mailto:UM_collaboration@metoffice.gov.uk | UM_collaboration@metoffice.gov.uk]].

NCI will add your account to the ACCESS group so that you can start using the model. Once this has been done follow the instructions at http://climate-cms.unsw.wikispaces.net/UM+Environment to get started.
[[Category:Unified Model]]