climate-cms wikis.unsw.edu.au - User contributions [en]

20CRv3

2023-09-28T02:50:40Z

P.petrelli:

'''NOAA-CIRES-DOE 20th Century Reanalysis V3'''

The 20CRv3 reanalysis uses a data assimilation system and surface pressure observations to generate a four-dimensional global atmospheric dataset of weather spanning 1836 to 2015.

Only surface observations of synoptic pressure is assimilated into the NOAA’s Global Forecast System,sea surface temperature and sea ice distribution are prescribed.  Temperature, pressure, winds, moisture, solar radiation and clouds, are estimated from the surface to the top of the atmosphere.

The temporal resolution of the main output is 3 hourly, however the data is also available as daily and monthly means.

The global grid used has a 75 km resolution.

More information is available from the [https://psl.noaa.gov/data/20thC_Rean/ PSL 20 Century reanalysis project ] and the  [https://portal.nersc.gov/project/20C_Reanalysis/ NERSC data portal]. The subdaily and member daily data was downloaded from the  [https://portal.nersc.gov/project/20C_Reanalysis/ NERSC data portal], all the other data from the [https://psl.noaa.gov/data/gridded/data.20thC_ReanV3.html PSL data portal].

The two portal published the data differently so there are inconsistencies in the way it is stored and named.

'''20CRV3 at NCI'''

The data is stored in

/g/data/ua8/LE_models/20CRv3/<frequency>/<variable>/<yr>

Available variables are:

'''subdaily: '''

*TMP2m - Temperature at 2 m 3 hourly by member
*HGT500 - Geopotential Height 3 hourly by member
*PRMSL -  Pressure at Mean Sea Level 3 hourly by member
*PRATE - Precipitation rate at surface 3 hourly by member

'''member_daily: '''

*PRMSL - Pressure at Mean Sea Level daily mean by member

'''member_monthly''':

*TMPS -  Surface air temperature monthly mean by member
*PRATE - Precipitation rate at surface monthly mean by member

'''mean_3hr:'''

*RH_2m - 3-hourly mean Relative humidity at 2m
*SPFH_2m - 3-hourly mean Specific humidity at 2m
*TMP_2m - 3-hourly mean Temperature at 2m
*UGRD_10m - 3-hourly mean U component of wind at 10m
*VGRD_10m - 3-hourly mean V component of wind at 10m
*ICEC_sfc - 3-hourly mean Sea ice area fraction
*PRES_sfc - 3-hourly mean Surface pressure
*TMP_sfc - 3-hourly mean Surface temperature
*HGT_pres -  3-hourly mean Geopotential Height on pressure Levels
*SPFH_pres - 3-hourly mean Specific humidity on pressure levels
*TMP_pres - 3-hourly mean Temperature on pressure levels
*UGRD_pres - 3-hourly mean U component of wind on pressure levels
*VGRD_pres - 3-hourly mean V component of wind on pressure levels
*SOILW_depth - 3-hourly mean Volumetric soil moisture content
*TSOIL_depth - 3-hourly mean Soil temperature

'''mean_daily:''' 

*prate - daily mean 3-hourly Precipitation Rate at surface
*prmsl - daily mean 3-hourly Pressure at Mean Sea Level
*tmax_2m -  daily 3-hourly Maximum Temperature at 2 m
*tmin_2m -  daily 3-hourly Minimum Temperature at 2 m
*apcp - daily mean 3-hourly accumulated convective precipitation at surface
*hgt - daily mean 3-hourly Geopotential Height on Pressure Levels
*uwind - daily mean 3-hourly U-wind on Pressure Levels
*vwind - daily mean 3-hourly V-wind on Pressure Levels

'''mean_monthly:''' 

*uwind - monthly mean 3-hourly U-wind on Pressure Levels
*vwind - monthly mean 3-hourly V-wind on Pressure Levels
*prate - monthly mean 3-hourly Precipitation Rate at surface (also as Long Term Mean)
*air - monthly mean 3-hourly Temperature on Pressure Levels

'''mean_monhtly/spread:'''

*air - monthly mean 3-hourly Temperature spread on Pressure Levels
*prate - monthly mean 3-hourly Precipitation Rate spread at surface

 

/g/data/ua8/LE_models/20CRv3/derived. contains some fields derived by Zak Bailie.

*HGT500 - daily mean derived from HGT500 sub-daily data
*TxTn - daily maximum of 2m air temperature, derived from TMP2m sub-daily data

'''Ackowledgement and references'''

Papers using the Twentieth Century Reanalysis Project datasets from the NESRC portal are requested to include the following text in their acknowledgments: "Support for the Twentieth Century Reanalysis Project dataset is provided by the U.S. Department of Energy, Office of Science Biological and Environmental Research (BER) program, by the National Oceanic and Atmospheric Administration Climate Program Office, and by the NOAA Physical Sciences Laboratory."

From the PSL portal:

"If you acquire 20th Century Reanalysis V3 data products from PSL, we ask that you acknowledge us in your use of the data. This may be done by including text such as ''20th Century Reanalysis V3 data provided by the NOAA/OAR/ESRL PSL, Boulder, Colorado, USA, from their Web site at /'' in any documents or publications using these data. We would also appreciate receiving a copy of the relevant publications. This will help PSL to justify keeping the 20th Century Reanalysis V3 data set freely available online in the future. Thank you!"

If you are using the data provided by Zak Bailie please ackowledge him.

== References ==

Compo, G.P., J.S. Whitaker, P.D. Sardeshmukh, N. Matsui, R.J. Allan, X. Yin, B.E. Gleason, R.S. Vose, G. Rutledge, P. Bessemoulin, S. Brönnimann, M. Brunet, R.I. Crouthamel, A.N. Grant, P.Y. Groisman, P.D. Jones, M. Kruk, A.C. Kruger, G.J. Marshall, M. Maugeri, H.Y. Mok, Ø. Nordli, T.F. Ross, R.M. Trigo, X.L. Wang, S.D. Woodruff, and S.J. Worley, 2011: [http://onlinelibrary.wiley.com/doi/10.1002/qj.776/abstract The Twentieth Century Reanalysis Project].'' Quarterly J. Roy. Meteorol. Soc.'', 137, 1-28. DOI: [http://onlinelibrary.wiley.com/doi/10.1002/qj.776/abstract 10.1002/qj.776.]

Slivinski, L.C., G. P. Compo, J. S. Whitaker, P. D. Sardeshmukh, B. S. Giese, C. McColl, R. Allan, X. Yin, R. Vose, H. Titchner, J. Kennedy, L. J. Spencer, L. Ashcroft, S. Brönnimann, M. Brunet, D. Camuffo, R. Cornes, T. A. Cram, R. Crouthamel, F. Domínguez-Castro, J. E. Freeman, J. Gergis, E. Hawkins, P. D. Jones, S. Jourdain, A. Kaplan, H. Kubota, F. Le Blancq, T.-C. Lee, A. Lorrey, J. Luterbacher, M. Maugeri, C. J. Mock, G.W. K. Moore, R. Przybylak, C. Pudmenzky, C. Reason, V. C. Slonosky, C. A. Smith, B. Tinz, B. Trewin, M. A. Valente, X. L. Wang, C. Wilkinson, K. Wood, and P. Wyszyński, 2019: [https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/qj.3598 Towards a more reliable historical reanalysis: Improvements for version 3 of the Twentieth Century Reanalysis system].'' Quarterly J. Roy. Meteorol. Soc.'', in press. DOI: [https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/qj.3598 10.1002/qj.3598.]

NOAA OISST

2023-03-07T01:42:20Z

P.petrelli:

The NOAA 1/4° daily Optimum Interpolation Sea Surface Temperature (or daily OISST) is an analysis constructed by combining observations from different platforms (satellites, ships, buoys) on a regular global grid. A spatially complete SST map is produced by interpolating to fill in gaps. More information from the [https://www.ncdc.noaa.gov/oisst NOAA website].

We have a local copy, updated daily of the AVHRR-Only OISST, which uses infrared satellite data from the Advanced Very High Resolution Radiometer (AVHRR).

The data covers from September 1981 till present, with daily temporal resoultion.

The scripts used to download and concatenate the data are available from the [https://github.com/coecms/CollectionsScripts/tree/master/OISST CMS github] .

NOAA has changed the delivery mode of this dataset in 2017, they also switched to netcdf4 format: "data from 1 April 2017 onward are in netCDF4 (with internal compression, so files will not be named *.gz) format. Older files are still netCDF3, until further notice." 

NB there is a potential bias introduced in data from 2017 onwards for v2.0, NOAA has recently released a new corrected version v2.1

[https://www.ncdc.noaa.gov/oisst/optimum-interpolation-sea-surface-temperature-oisst-v21 https://www.ncdc.noaa.gov/oisst/optimum-interpolation-sea-surface-temperature-oisst-v21]

Initially this version was available only from 2016 onwards, as of the 21st of May the all v2.1 series is available, from 1981 to present day.

Also, the initial release had a timestep issue so some files were off by 12 hours, this has been fixed int he current release. 

'''NB this dataset is now maintained as part of the [https://aus-ref-clim-data-nci.github.io/aus-ref-clim-data-nci/datasets/oisst.html Australian Community Reference Climate Data Collection] access will eventually be removed!'''

=== '''License''' ===

This dataset is freely available, there are no use limitations or accounts needed. Although there is an optional registration form that can be reached from the documentation page in the NOAA website under the "data access" session. Please consider register, so you'll receive updates on the data and as this also helps the data publishers to get an idea of the data usefulness and continue providing for public access. Use the following publications to cite the dataset:

Huang, B., C. Liu, V. Banzon, E. Freeman, G. Graham, W. Hankins, T. Smith, and H.-M. Zhang, 2020: Improvements of the Daily Optimum Sea Surface Temperature (DOISST) - Version 2.1. (J. Climate, submitted). (for v2.1)

Banzon, V., Smith, T. M., Chin, T. M., Liu, C., and Hankins, W., 2016: A long-term record of blended satellite and in situ sea-surface temperature for climate monitoring, modeling and environmental studies. ''Earth Syst. Sci. Data'', '''8''', 165–176, [http://www.earth-syst-sci-data.net/8/165/2016/essd-8-165-2016.html doi:10.5194/essd-8-165-2016(link is external)]. Reynolds, R. W., T. M. Smith, C. Liu, D. B. Chelton, K. S. Casey, and M. G. Schlax, 2007: Daily high-resolution-blended analyses for sea surface temperature. ''Journal of Climate'', '''20''', 5473–5496, [http://dx.doi.org/10.1175/2007JCLI1824.1 doi:10.1175/JCLI-D-14-00293.1(link is external)].

=== '''OISST on raijin''' ===

This dataset is available on raijin under the ua8 project (NB this might change in the near future because of the NCI data re-organisation). The original data is located in

/g/data/ua8/NOAA_OISST/AVHRR/raw/<year>/<files.gz>. for v2.0

/g/data/ua8/NOAA_OISST/AVHRR/v2-1/<year>/<files>

NB the v2-0 files are zipped as when downloaded.

The files are then concatenated for ease of use into yearly files, the resulting modified dataset is available under

/g/data/ua8/NOAA_OISST/AVHRR/v2-0_modified/<files>

/g/data/ua8/NOAA_OISST/AVHRR/v2-1_modified/<files> 

[[Category:Dataset]] [[Category:Clex-managed-data]]

Running Jupyter Notebook

2023-02-23T00:05:27Z

P.petrelli:

= On ARE =

Currently, the easiest way to run a Jupyter Notebook is on NCI's [https://opus.nci.org.au/display/Help/ARE+User+Guide Australian Research Environment] a Open-On-Demand service.  For a guide to set up and use the OOD, click [https://opus.nci.org.au/display/Help/3.1+Starting+JupyterLab+App here]. You can access all the gadi filesystems from the ARE environment and select different queues and cpus/memory options to suit your workflow. You can also select our hh5 conda environments.

Please follow the instructions to request the best sized cluster for your work.

 

= On Gadi =

NB as ARE has been released there is no advantage in using gadi_jupyter script. So we have stopped supporting it!

You can also run a notebook from Gadi's compute nodes using the [https://github.com/coecms/nci_scripts/blob/master/gadi_jupyter gadi_jupyter] script.

=== Pre-requisite ===

For this method to work, you must become part of the '''hh5''' project at NCI using [https://my.nci.org.au/mancini/project/hh5 my.nci.org.au].

This project will give you access to [[Conda|conda environments]] for Python managed by the CMS team.

=== Setting Up SSH ===

You will need to set up SSH keys to use gadi_jupyter

Create the file ~/.ssh/config with something like the following lines in:
<syntaxhighlight lang="text">
Host *.nci.org.au
User abc123 # Your NCI username here
</syntaxhighlight>

Create a SSH key by running 'ssh-keygen'. It will ask for a 'passphrase', make sure you enter one, it doesn't need to be the same as your NCI password. Use the default key file name.

Copy your SSH key to gadi. Some computers have a command that can do this automatically - 'ssh-copy-id gadi.nci.org.au'. If not, you need to add the contents of the file '~/.ssh/id_rsa.pub' on your local computer to the end of the file '~/.ssh/authorized_keys' on Gadi.

Enable your SSH key by running the command 'ssh-add', it will ask for the SSH passphrase you entered when you made the key (which may not be your NCI password). If it says 'Could not open a connection to your SSH agent', or '‘could not connect to authentication agent', run 'ssh-agent bash' and then 'ssh-add'.

You can now run './gadi_jupyter'

=== Notes ===

*You need to download this script to your own computer and execute there. The script handles the connection.
*This job requires an implementation of bash. This is default on MacOS and Linux, but not Windows.
*This will cost SU depending on the queue and the resources requested (default: 1 cores, express queue)
*You also won't have access to the internet to, for example, download data.

=== Running ===

You can download the script from the link above directly, or by running
<syntaxhighlight lang="bash">
git clone https://github.com/coecms/nci_scripts.git
</syntaxhighlight>

which will download the script into a directory called '''nci_scripts'''.

There are several options that you can use when running the script:
<syntaxhighlight lang="unknown">
General Options:
-h: Print help
-l: NCI username
-L: NCI login node (default 'gadi.nci.org.au')
-e: Conda environment

Queue Options:
-q QUEUE: Queue name
-n NCPU: Use NCPU cpus
-m MEM: Memory allocation (default 4*NCPU GB)
-t TIME: Walltime limit (default 1 hour)
-J JOBFS: Jobfs allocation (default 100 GB)
-P PROJ: Submit job under project PROJ
</syntaxhighlight>

=== Windows Desktops ===

To run gadi_jupyter on Windows you'll need Bash and some form of SSH available. This could be from

* git bash - [https://gitforwindows.org/ https://gitforwindows.org/]
* cygwin terminal - [https://www.cygwin.com/ https://www.cygwin.com/]

Install one of these, then within the terminal follow the 'Setting up SSH' instructions before running 'gadi_jupyter'. (The terminal should start in your 'My Documents' directory if you need to find the downloaded script)

[[Category:Python]]

Running Jupyter Notebook

2023-02-23T00:04:04Z

P.petrelli:

= On ARE =

Currently, the easiest way to run a Jupyter Notebook is on NCI's [https://opus.nci.org.au/display/Help/ARE+User+Guide Australian Research Environment] a Open-On-Demand service.  For a guide to set up and use the OOD, click [https://opus.nci.org.au/display/Help/3.1+Starting+JupyterLab+App here].

Please follow the instructions to request the best sized cluster for your work.

 

= On Gadi =

NB as ARE has been released there is no advantage in using gadi_jupyter script. So we have stopped supporting it!

You can also run a notebook from Gadi's compute nodes using the [https://github.com/coecms/nci_scripts/blob/master/gadi_jupyter gadi_jupyter] script.

=== Pre-requisite ===

For this method to work, you must become part of the '''hh5''' project at NCI using [https://my.nci.org.au/mancini/project/hh5 my.nci.org.au].

This project will give you access to [[Conda|conda environments]] for Python managed by the CMS team.

=== Setting Up SSH ===

You will need to set up SSH keys to use gadi_jupyter

Create the file ~/.ssh/config with something like the following lines in:
<syntaxhighlight lang="text">
Host *.nci.org.au
User abc123 # Your NCI username here
</syntaxhighlight>

Create a SSH key by running 'ssh-keygen'. It will ask for a 'passphrase', make sure you enter one, it doesn't need to be the same as your NCI password. Use the default key file name.

Copy your SSH key to gadi. Some computers have a command that can do this automatically - 'ssh-copy-id gadi.nci.org.au'. If not, you need to add the contents of the file '~/.ssh/id_rsa.pub' on your local computer to the end of the file '~/.ssh/authorized_keys' on Gadi.

Enable your SSH key by running the command 'ssh-add', it will ask for the SSH passphrase you entered when you made the key (which may not be your NCI password). If it says 'Could not open a connection to your SSH agent', or '‘could not connect to authentication agent', run 'ssh-agent bash' and then 'ssh-add'.

You can now run './gadi_jupyter'

=== Notes ===

*You need to download this script to your own computer and execute there. The script handles the connection.
*This job requires an implementation of bash. This is default on MacOS and Linux, but not Windows.
*This will cost SU depending on the queue and the resources requested (default: 1 cores, express queue)
*You also won't have access to the internet to, for example, download data.

=== Running ===

You can download the script from the link above directly, or by running
<syntaxhighlight lang="bash">
git clone https://github.com/coecms/nci_scripts.git
</syntaxhighlight>

which will download the script into a directory called '''nci_scripts'''.

There are several options that you can use when running the script:
<syntaxhighlight lang="unknown">
General Options:
-h: Print help
-l: NCI username
-L: NCI login node (default 'gadi.nci.org.au')
-e: Conda environment

Queue Options:
-q QUEUE: Queue name
-n NCPU: Use NCPU cpus
-m MEM: Memory allocation (default 4*NCPU GB)
-t TIME: Walltime limit (default 1 hour)
-J JOBFS: Jobfs allocation (default 100 GB)
-P PROJ: Submit job under project PROJ
</syntaxhighlight>

=== Windows Desktops ===

To run gadi_jupyter on Windows you'll need Bash and some form of SSH available. This could be from

* git bash - [https://gitforwindows.org/ https://gitforwindows.org/]
* cygwin terminal - [https://www.cygwin.com/ https://www.cygwin.com/]

Install one of these, then within the terminal follow the 'Setting up SSH' instructions before running 'gadi_jupyter'. (The terminal should start in your 'My Documents' directory if you need to find the downloaded script)

[[Category:Python]]

Running Jupyter Notebook

2023-02-23T00:02:33Z

P.petrelli:

= On ARE =

Currently, the easiest way to run a Jupyter Notebook is on NCI's [https://opus.nci.org.au/display/Help/ARE+User+Guide Australian Research Environment] a Open-On-Demand service.  For a guide to set up and use the OOD, click [https://opus.nci.org.au/display/Help/3.1+Starting+JupyterLab+App here].

Please follow the instructions to request the best sized cluster for your work.

 

= On Gadi =

You can also run a notebook from Gadi's compute nodes using the [https://github.com/coecms/nci_scripts/blob/master/gadi_jupyter gadi_jupyter] script.
=== Pre-requisite ===
For this method to work, you must become part of the '''hh5''' project at NCI using [https://my.nci.org.au/mancini/project/hh5 my.nci.org.au].

This project will give you access to [[Conda|conda environments]] for Python managed by the CMS team.

=== Setting Up SSH ===

You will need to set up SSH keys to use gadi_jupyter

Create the file ~/.ssh/config with something like the following lines in:
<syntaxhighlight lang="text">
Host *.nci.org.au
User abc123 # Your NCI username here
</syntaxhighlight>

Create a SSH key by running 'ssh-keygen'. It will ask for a 'passphrase', make sure you enter one, it doesn't need to be the same as your NCI password. Use the default key file name.

Copy your SSH key to gadi. Some computers have a command that can do this automatically - 'ssh-copy-id gadi.nci.org.au'. If not, you need to add the contents of the file '~/.ssh/id_rsa.pub' on your local computer to the end of the file '~/.ssh/authorized_keys' on Gadi.

Enable your SSH key by running the command 'ssh-add', it will ask for the SSH passphrase you entered when you made the key (which may not be your NCI password). If it says 'Could not open a connection to your SSH agent', or '‘could not connect to authentication agent', run 'ssh-agent bash' and then 'ssh-add'.

You can now run './gadi_jupyter'

=== Notes ===

*You need to download this script to your own computer and execute there. The script handles the connection.
*This job requires an implementation of bash. This is default on MacOS and Linux, but not Windows.
*This will cost SU depending on the queue and the resources requested (default: 1 cores, express queue)
*You also won't have access to the internet to, for example, download data.

=== Running ===

You can download the script from the link above directly, or by running
<syntaxhighlight lang="bash">
git clone https://github.com/coecms/nci_scripts.git
</syntaxhighlight>

which will download the script into a directory called '''nci_scripts'''.

There are several options that you can use when running the script:
<syntaxhighlight lang="unknown">
General Options:
-h: Print help
-l: NCI username
-L: NCI login node (default 'gadi.nci.org.au')
-e: Conda environment

Queue Options:
-q QUEUE: Queue name
-n NCPU: Use NCPU cpus
-m MEM: Memory allocation (default 4*NCPU GB)
-t TIME: Walltime limit (default 1 hour)
-J JOBFS: Jobfs allocation (default 100 GB)
-P PROJ: Submit job under project PROJ
</syntaxhighlight>

=== Windows Desktops ===

To run gadi_jupyter on Windows you'll need Bash and some form of SSH available. This could be from

* git bash - [https://gitforwindows.org/ https://gitforwindows.org/]
* cygwin terminal - [https://www.cygwin.com/ https://www.cygwin.com/]

Install one of these, then within the terminal follow the 'Setting up SSH' instructions before running 'gadi_jupyter'. (The terminal should start in your 'My Documents' directory if you need to find the downloaded script)

[[Category:Python]]

Getting Help

2023-02-20T21:54:48Z

P.petrelli:

== CMS help ==

The CMS team supports CLEX members with computational issues: 

*'''Running Models''': we support several models (see navigation bar on the left) and assist with others, subject to limitations of scope and team resources. 
*'''Data Analysis''': we assist with analysis and general computing tasks on NCI machines, and offer only limited support on other sites or home machines as we possess neither control over nor familiarity with such systems.
*'''Data Access and publishing''': we offer assistance in publishing data and accessing climate data available at NCI.

The CMS team can be contacted for support through: 

*The Climate Weather Science (CWS) helpdesk '''cws_help@nci.org.au'''
*[[One-on-one_calls|One-on-one_videocall]]
*In person meetings with local '''[http://climate-cms.wikis.unsw.edu.au/CMS CMS team members]'''
*CodeBreak

The CWS help desk is the preferred contact method as it allows the CMS team to prioritize queries so as to set aside time for other CMS duties, provides transparency of open requests to all CMS team members , and gives the CMS team flexibility to cover for team absences. Simply outline your issue in an email to '''cws_help@nci.org.au''' to open a ticket on our helpdesk.

== Guidelines for help requests ==

'''Before seeking assistance:'''

*Follow the '''[http://climate-cms.wikis.unsw.edu.au/Troubleshooting troubleshooting guide]'''
*Read relevant documentation on our wiki or '''[https://climate-cms.org/ blog]'''
*Google it !

'''When seeking assistance''':

*Reduce''' '''your query if possible to a minimal reproducible example '''('''Stack Overflow has a '''[https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means])'''
*Submit separate help requests for significantly new queries instead of appending to existing tickets
*Ensure required directories and/or files '''[http://climate-cms.wikis.unsw.edu.au/Tips:_Custom_file_permissions_at_creation have group permissions] '''that make them accessible to us. For example, NCI /home directories by default are not, so you will need to change their permissions, but typically we can access the NCI /scratch and /g/data for your group
*Include all relevant information in your help request:
**If a script/command generates an error, provide an exact reproduction of the script/command typed and error message
**What have you tried so far to troubleshoot
**Background info about your task (as we might be able to suggest instead a different approach) with links to any references invoked in your request
**Platform used for your task (e.g., your machine/OS, NCI Gadi, OOD, accessdev). If NCI, supply your username, and if Gadi indicate if the issue was in a login session or the PBS compute nodes
**Specify any environment variables set and modules loaded, and if loaded in your .bashrc or .bash_profile files (not recommended). For errors with the conda environment we support, include the environment version. 
**Upload any required code to '''[https://gist.github.com/ gist.github.com] '''or '''[https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository] '''for more complex examples. If barred by licensing issues, put it on NCI and provide the path. Never send copies of your code.
*Do not send code or error messages as screenshots or poorly formatted text

'''After seeking assistance''':

*You should receive a reply to your query within 24 hours during normal business hours
*We will provide a solution to your problem (if within scope), or a work-around, or an explanation for why it is currently unsolvable
*You should expect updates required for current tickets to take at most a week during non-holiday periods
*Consider if you need some '''[http://climate-cms.wikis.unsw.edu.au/Training training] '''to improve your skills

If any of the above expectations are not being met, please contact the CMS team.

Getting Help

2023-02-20T21:54:08Z

P.petrelli:

== CMS help ==

The CMS team supports CLEX members with computational issues: 

*'''Running Models''': we support several models (see navigation bar on the left) and assist with others, subject to limitations of scope and team resources. 
*'''Data Analysis''': we assist with analysis and general computing tasks on NCI machines, and offer only limited support on other sites or home machines as we possess neither control over nor familiarity with such systems.
*'''Data Access and publishing''': we offer assistance in publishing data and accessing climate data available at NCI.

The CMS team can be contacted for support through: 

*The Climate Weather Science (CWS) helpdesk '''cws_help@nci.org.au'''
*[[One-on-one_videocall]]
*In person meetings with local '''[http://climate-cms.wikis.unsw.edu.au/CMS CMS team members]'''
*CodeBreak

The CWS help desk is the preferred contact method as it allows the CMS team to prioritize queries so as to set aside time for other CMS duties, provides transparency of open requests to all CMS team members , and gives the CMS team flexibility to cover for team absences. Simply outline your issue in an email to '''cws_help@nci.org.au''' to open a ticket on our helpdesk.

== Guidelines for help requests ==

'''Before seeking assistance:'''

*Follow the '''[http://climate-cms.wikis.unsw.edu.au/Troubleshooting troubleshooting guide]'''
*Read relevant documentation on our wiki or '''[https://climate-cms.org/ blog]'''
*Google it !

'''When seeking assistance''':

*Reduce''' '''your query if possible to a minimal reproducible example '''('''Stack Overflow has a '''[https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means])'''
*Submit separate help requests for significantly new queries instead of appending to existing tickets
*Ensure required directories and/or files '''[http://climate-cms.wikis.unsw.edu.au/Tips:_Custom_file_permissions_at_creation have group permissions] '''that make them accessible to us. For example, NCI /home directories by default are not, so you will need to change their permissions, but typically we can access the NCI /scratch and /g/data for your group
*Include all relevant information in your help request:
**If a script/command generates an error, provide an exact reproduction of the script/command typed and error message
**What have you tried so far to troubleshoot
**Background info about your task (as we might be able to suggest instead a different approach) with links to any references invoked in your request
**Platform used for your task (e.g., your machine/OS, NCI Gadi, OOD, accessdev). If NCI, supply your username, and if Gadi indicate if the issue was in a login session or the PBS compute nodes
**Specify any environment variables set and modules loaded, and if loaded in your .bashrc or .bash_profile files (not recommended). For errors with the conda environment we support, include the environment version. 
**Upload any required code to '''[https://gist.github.com/ gist.github.com] '''or '''[https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository] '''for more complex examples. If barred by licensing issues, put it on NCI and provide the path. Never send copies of your code.
*Do not send code or error messages as screenshots or poorly formatted text

'''After seeking assistance''':

*You should receive a reply to your query within 24 hours during normal business hours
*We will provide a solution to your problem (if within scope), or a work-around, or an explanation for why it is currently unsolvable
*You should expect updates required for current tickets to take at most a week during non-holiday periods
*Consider if you need some '''[http://climate-cms.wikis.unsw.edu.au/Training training] '''to improve your skills

If any of the above expectations are not being met, please contact the CMS team.

One-on-one calls

2023-02-20T21:53:32Z

P.petrelli:

NB We discontinued using the appointlet app because of permission issues with accessing the calendars!!!

To book a 30 minutes one-on-one videocalls please open a ticket mailing the cws_help@nci.org.au. 

The email should include:

*a summary of what you would like to discuss
*preferred days/times
*who would you like to talk to if you have a preference

Please give us at least a working day notice, we will get back to you asap to arrange a suitable time.

If you have a specific issue to resolve please include as much relevant information as possible, like paths to codes, input files etc. This will give us a chance to be prepared for the meeting and make sure we can access all necessary information.

If you have a preference for a team member, include that in the email, we will try to accomodate your choice. However, we might occasionally suggest a different member based on availiability and expertise.

 

 

 

=   =

Getting Help

2023-02-20T21:38:26Z

P.petrelli:

== CMS help ==

The CMS team supports CLEX members with computational issues: 

*'''Running Models''': we support several models (see navigation bar on the left) and assist with others, subject to limitations of scope and team resources. 
*'''Data Analysis''': we assist with analysis and general computing tasks on NCI machines, and offer only limited support on other sites or home machines as we possess neither control over nor familiarity with such systems.
*'''Data Access and publishing''': we offer assistance in publishing data and accessing climate data available at NCI.

The CMS team can be contacted for support through: 

*The Climate Weather Science (CWS) helpdesk '''cws_help@nci.org.au'''
*'''One-on-one videocall'''
*In person meetings with local '''[http://climate-cms.wikis.unsw.edu.au/CMS CMS team members]'''
*CodeBreak

The CWS help desk is the preferred contact method as it allows the CMS team to prioritize queries so as to set aside time for other CMS duties, provides transparency of open requests to all CMS team members , and gives the CMS team flexibility to cover for team absences. Simply outline your issue in an email to '''cws_help@nci.org.au''' to open a ticket on our helpdesk.

For a one-on-one videocall open a ticket on the helpdesk, including:

*a summary of what you would like to discuss
*preferred days/times
*who would you like to talk to if you have a preference

Please give us at least a working day notice, we will get back to you asap to arrange a suitable time. If you express a preference for a specific member, depending on their current workload and/or the topic of discussion we might suggest someone else from the team as more suitable.

== Guidelines for help requests ==

'''Before seeking assistance:'''

*Follow the '''[http://climate-cms.wikis.unsw.edu.au/Troubleshooting troubleshooting guide]'''
*Read relevant documentation on our wiki or '''[https://climate-cms.org/ blog]'''
*Google it !

'''When seeking assistance''':

*Reduce''' '''your query if possible to a minimal reproducible example '''('''Stack Overflow has a '''[https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means])'''
*Submit separate help requests for significantly new queries instead of appending to existing tickets
*Ensure required directories and/or files '''[http://climate-cms.wikis.unsw.edu.au/Tips:_Custom_file_permissions_at_creation have group permissions] '''that make them accessible to us. For example, NCI /home directories by default are not, so you will need to change their permissions, but typically we can access the NCI /scratch and /g/data for your group
*Include all relevant information in your help request:
**If a script/command generates an error, provide an exact reproduction of the script/command typed and error message
**What have you tried so far to troubleshoot
**Background info about your task (as we might be able to suggest instead a different approach) with links to any references invoked in your request
**Platform used for your task (e.g., your machine/OS, NCI Gadi, OOD, accessdev). If NCI, supply your username, and if Gadi indicate if the issue was in a login session or the PBS compute nodes
**Specify any environment variables set and modules loaded, and if loaded in your .bashrc or .bash_profile files (not recommended). For errors with the conda environment we support, include the environment version. 
**Upload any required code to '''[https://gist.github.com/ gist.github.com] '''or '''[https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository] '''for more complex examples. If barred by licensing issues, put it on NCI and provide the path. Never send copies of your code.
*Do not send code or error messages as screenshots or poorly formatted text

'''After seeking assistance''':

*You should receive a reply to your query within 24 hours during normal business hours
*We will provide a solution to your problem (if within scope), or a work-around, or an explanation for why it is currently unsolvable
*You should expect updates required for current tickets to take at most a week during non-holiday periods
*Consider if you need some '''[http://climate-cms.wikis.unsw.edu.au/Training training] '''to improve your skills

If any of the above expectations are not being met, please contact the CMS team.

Open access license

2022-12-08T02:27:49Z

P.petrelli:

== Why a license ==

If you are sharing your data or code freely, you might be wondering why you should be using a license, but making your data freely available to others doesn't mean that you shouldn't establish in which way it can be used. This is where open access licenses come in handy and this is why you can have different flavours of the same open-access license.

Licenses are really useful for both the "data" creator and the user, it helps a potential user to work out quickly if the data is suitable for their intended use and helps them citing and using the data in the way the creator intended.

 

== Commonly used licenses ==

Here we summarise the licenses we commonly use. For a much more detailed view of these licenses, ARDC (the Australian Research Data Commons) has a [https://ardc.edu.au/resource/research-data-rights-management-guide/ comprehensive guide] which covers also '''copyright'''.

The important thing to understand about copyright is that it is usually held by or shared with your own institution, even if you are publishing with CLEX. This is because CLEX is not a legal entity and your institution has the Intellectual Property of your work (see below). When you or CLEX applies a license to your data, you are doing so on behalf of your institution. Most institutions have an open-access policy and would be fine with an open-access license unless there are particular circumstances around your data, for example, if it is already covered by an agreement or is of a sensitive nature. If you are in doubt, check with your institution research office if they are fine with the license you want to use.

=== '''Common license rules''' ===

Before looking at the different flavours of licenses, we need to understand what they are covering. First of all, these are all open-access licenses so by applying one of them, you are usually giving free access to your data which means that you are not charging for using your data. However, you can regulate how a potential user can access and use your data. Usually, a license will cover some or all of the following points:

*'''Attribution:''' a user is required to cite and/or acknowledge your data, this is usually included in any license. You should make sure you are sharing the information on how to cite the data.
*'''Commercial use and/or research use:''' you can limit the use of your data to research only or exclude any commercial use of your data.
*'''Distribution vs private use''': if a user can distribute your data/software or can only use it for their own work
*'''License and copyright notice:''' a copy of the license and copyright has to be included with the data or software.
*'''Modification''': a user can modify, augment or transform your data to create a new dataset or software
*'''Same license:''' the dataset or software resulting from modification of yours has to be distributed under the same license, so they cannot restrict the use of their derived product.
*'''No-Derivatives:''' if a user modifies, augments or transforms the data in any way they cannot redistribute it. This can be changed of course if they get in contact with the creator and the creator allows an exception. In fact, this clause is often added only so the creator can be kept informed on how the data is used rather than a strict restriction on derivatives.

The following two points apply only to software:

*'''Disclose source:''' source code must be made available when derived product is distributed
*'''State changes: ''' any changes made must be documented

Finally licenses usually contain a '''disclaimer''' to cover warranty and liability, i.e. that the data or software is provided "as is" and that the copyright owner cannot be considered responsible for any damages derived by using the product.

As you can see there are a lot of different flavours possible. Usually, any license will cover "attribution" which is the most common reason to apply a license: to get your work recognised!

 

=== '''Datasets license''' ===

While we consider both datasets and code as a form of data, from a licensing point of view they are treated differently. For datasets we suggest the [https://creativecommons.org/licenses/ Creative Commons] licenses (CC). As well as the international version, there is also an [https://creativecommons.org.au Australian Creative Commons]. These licenses are simple to use, they offer 6 different flavours, which cover most use cases and an [https://creativecommons.org/choose/ online tool] to help you choose between them. They have a human-readable version as well as the legal text and finally, the International version was created to cover your data independently of the potential user's country of origin.

Creative Commons use abbreviation for each of the options:

*BY - for attribution which is always present, the right to distribute is also implicit in any CC license
*NC - for not commercial
*SA - for share alike
*ND - for not derivatives

Check this 5 minutes video to learn more on the available combinations:

[https://creativecommons.org/about/videos/creative-commons-kiwi/ https://creativecommons.org/about/videos/creative-commons-kiwi/]

If you are unsure which license to use, CLEX advises using the [https://creativecommons.org/licenses/by/4.0/legalcode International CC-BY 4.0] license.

We used to recommend the non-commercial version but this can have unexpected ramifications. For example, if you use some data or an image for a presentation at a conference that people have to pay to attend, this could be considered Commercial use.

=== '''Software licenses''' ===

For software, you can refer to the [https://opensource.org/licenses Open Source initiative] and the [https://software.ac.uk/resources/guides/adopting-open-source-licence?_ga=2.23847290.1503282406.1562136302-1313042888.1562136302 Software Sustainability Institute], SSI. They both offer a good introduction, SSI also lists several other websites covering available licenses in a human-readable way and useful comparison tools. Among these:

from github: [https://choosealicense.com/licenses/ https://choosealicense.com/licenses/]

from TLDR Legal: [https://tldrlegal.com/licenses/browse https://tldrlegal.com/licenses/browse]

They both summarise licenses by their "rules".

We usually apply the [https://www.apache.org/licenses/LICENSE-2.0 Apache 2.0 license] to the code we produce. Creative Commons are not suitable for licensing software, as stated on the Creative Commons website:

" ... Unlike software-specific licenses, CC licenses do not contain specific terms about the distribution of source code, which is often important to ensuring the free reuse and modifiability of software. Many software licenses also address patent rights, which are important to software but may not be applicable to other copyrightable works. Additionally, our licenses are currently not compatible with the major software licenses, so it would be difficult to integrate CC-licensed work with other free software. Existing software licenses were designed specifically for use with software and offer a similar set of rights to the Creative Commons licenses. "

Most of the differences are due to the collaborative nature of most open access code.

Apache 2.0 applies the following

{| border="1" cellpadding="1" cellspacing="1" style="width: 500px;"
|-
! Permissions
! Conditions
! Limitations
|-
|
*Commercial use
*Distribution
*Modification
*Patent use
*Private use

|
*License and copyright notice
*State changes

|
*Liability
*Trademark use
*Warranty

|}

Summarised this means that commercial and private use, distribution and modification are all allowed. A user can also use parts of the code covered by a patent, should there be any, without paying royalties. A user has to attach the original license and copyright notices to any derived product they want to distribute, as well as stating any changes made. Finally, the software creator cannot be held liable for damages and they share the product "as is" with no warranty and the license does not grant trademark rights.

A good insight is again given by [https://opensource.com/article/18/2/how-make-sense-apache-2-patent-license this article] from the [https://opensource.com opensource.com] website.

 

== '''How to apply a license''' ==

Once you have decided which license to apply it is usually fairly easy to apply it to your dataset or code.

When publishing a dataset with NCI, for example, we create a license.txt file which contains the chosen license legal text.

At the top of the file, we specify the dataset title ( referred to as "licensed material") and the owner of the work we are licensing (referred to as "licensor"). Please note that in the example below CLEX is acting as licensor on behalf of the author and/or their institution.
<blockquote><pre>You can see the license online at

http://creativecommons.org/licenses/by/4.0/legalcode

Licensor - ARC Centre of Excellence for Climate Extremes (CLEX) - Level 4, Mathews Building, University of New South Wales, Sydney, NSW, Australia, 2052, Phone: +61 2 9385 9393, e-mail coecss@unsw.edu.au
Author: "author name" <"author-email">

Licensed Material - <dataset title> (version 1.0)

Creative Commons Attribution 4.0 International Public License

By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons ....</pre>
</blockquote>
The license.txt file then will be available both in the dataset local directory and on the NCI dataset catalogue together with the files. Also, we will use the appropriate license link to the Creative Commons website in the files global attributes and in the metadata records we produce.

Other publishers usually have similar requirements.

When publishing or even simply sharing our software, the license is both available in the code repository as a license.txt file and included in the actual files.
<blockquote><pre>#!/usr/bin/env python
# Copyright 2019 ARC Centre of Excellence for Climate Extremes
# author: "author name" <"author-email">
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.</pre>
</blockquote>
It is important that whichever license you choose the owner, author and title of the work are clear, including the source in case of a derived work. This is the information that anyone who wants to use your work should be using to generate a correct attribution. Even better, you should be providing the preferred citation text yourself.

== '''More on intellectual property and copyrights by institution''' ==

[https://services.anu.edu.au/research-support/technology-transfer/commercialising-your-research/intellectual-property ANU]

[https://research.unimelb.edu.au/partner/technology-licensing/for-researchers/intellectual-property-for-researchers/staff-ownership-of-intellectual-property Melbourne Uni] (need to login to view)

[https://www.monash.edu/library/researchers/researchdata/guidelines/ownership Monash]

[https://www.arc.unsw.edu.au/help/university-advice/intellectual-property UNSW]

[https://www.utas.edu.au/research/partnering/technology-transfer-and-commercialisation/intellectual-property UTAS ]

== '''Q&A''' ==

*Does a license last forever?
**Technically a license is valid until you hold the copyright, which usually in Australia expires 70 years after creation. You can revoke a license and apply a different one anytime 
*If I move from one license to a different one, would the new one apply in retrospect?
**No, a license is an agreement you enter in with someone else at the moment in which they obtain your data/code, so for them the old license will still apply. The new license will apply to anyone who download your data/code from when the license enters in use. If you update your data or code and a "old" user downloads the updates though the new license will apply for them from that moment on.
*What are my options if someone breach the license?
**The license is enforceable in court, clearly that's an extreme step, usually people breach the license "accidentally" you should first make contact with them and point out they're breaching the licensing terms and what should they do to amend it. If the breach is of a serious nature contacting the research office of your institution and seek legal advice on the steps to be taken is probably the best course of action.
*How can my license be valid if CLEX or myself act as licensor when the copyright belongs to my institution?
** If you are the creator of the data/code then you can apply a license on behalf of your institution, they won't mind as long as the license you are using is in line with their recommendations. Most of Australian universities and the ARC, which funds most projects, require open access for any research product (unless there is a valid reason not to).

[https://creativecommons.org/faq/ Creative Commons FAQ]

[http://www.apache.org/foundation/license-faq.html Apache 2.0 FAQ]

 

[[Category:Data induction]]

Getting Help

2022-11-23T23:02:40Z

P.petrelli:

== CMS help ==

The CMS team supports CLEX members with computational issues: 

*'''Running Models''': we support several models (see navigation bar on the left) and assist with others, subject to limitations of scope and team resources. 
*'''Data Analysis''': we assist with analysis and general computing tasks on NCI machines, and offer only limited support on other sites or home machines as we possess neither control over nor familiarity with such systems.
*'''Data Access and publishing''': we offer assistance in publishing data and accessing climate data available at NCI.

The CMS team can be contacted for support through: 

*The Climate Weather Science (CWS) helpdesk '''cws_help@nci.org.au'''
*'''[http://climate-cms.wikis.unsw.edu.au/One-on-one_calls One-on-one videocall]'''
*In person meetings with local '''[http://climate-cms.wikis.unsw.edu.au/CMS CMS team members]'''
*CodeBreak

The CWS help desk is the preferred contact method as it allows the CMS team to prioritize queries so as to set aside time for other CMS duties, provides transparency of open requests to all CMS team members , and gives the CMS team flexibility to cover for team absences. Simply outline your issue in an email to '''cws_help@nci.org.au''' to open a ticket on our helpdesk.

== Guidelines for help requests ==

'''Before seeking assistance:'''

*Follow the '''[http://climate-cms.wikis.unsw.edu.au/Troubleshooting troubleshooting guide]'''
*Read relevant documentation on our wiki or '''[https://climate-cms.org/ blog]'''
*Google it !

'''When seeking assistance''':

*Reduce''' '''your query if possible to a minimal reproducible example '''('''Stack Overflow has a '''[https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means])'''
*Submit separate help requests for significantly new queries instead of appending to existing tickets
*Ensure required directories and/or files '''[http://climate-cms.wikis.unsw.edu.au/Tips:_Custom_file_permissions_at_creation have group permissions] '''that make them accessible to us. For example, NCI /home directories by default are not, so you will need to change their permissions, but typically we can access the NCI /scratch and /g/data for your group
*Include all relevant information in your help request:
**If a script/command generates an error, provide an exact reproduction of the script/command typed and error message
**What have you tried so far to troubleshoot
**Background info about your task (as we might be able to suggest instead a different approach) with links to any references invoked in your request
**Platform used for your task (e.g., your machine/OS, NCI Gadi, OOD, accessdev). If NCI, supply your username, and if Gadi indicate if the issue was in a login session or the PBS compute nodes
**Specify any environment variables set and modules loaded, and if loaded in your .bashrc or .bash_profile files (not recommended). For errors with the conda environment we support, include the environment version. 
**Upload any required code to '''[https://gist.github.com/ gist.github.com] '''or '''[https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository] '''for more complex examples. If barred by licensing issues, put it on NCI and provide the path. Never send copies of your code.
*Do not send code or error messages as screenshots or poorly formatted text

'''After seeking assistance''':

*You should receive a reply to your query within 24 hours during normal business hours
*We will provide a solution to your problem (if within scope), or a work-around, or an explanation for why it is currently unsolvable
*You should expect updates required for current tickets to take at most a week during non-holiday periods
*Consider if you need some '''[http://climate-cms.wikis.unsw.edu.au/Training training] '''to improve your skills

If any of the above expectations are not being met, please contact the CMS team.

NASA-TRMM

2022-11-06T22:04:24Z

P.petrelli:

'''Tropical Rainfall Measuring Mission - 3 hours Level 3 product'''

'''Important update:'''this dataset is now superseded by [[GPM-IMERG]].

TRMM-TMPA is a 3 hours Level 3 product from the Tropical Rainfall Measuring Mission (TRMM 3B42) providing a rainfall estimate.

This dataset is the output from the TMPA (TRMM Multi-satellite Precipitation) Algorithm, and provides precipitation estimates in the TRMM regions that have the (nearly-zero) bias of the ”TRMM Combined Instrument” precipitation estimate and the dense sampling of high-quality microwave data with fill-in using microwave-calibrated infrared estimates.

The spatial resolution is 0.25ºx0.25º latitude-longitude (on a Cylindrical Equal Distance grid) , covering -180E to 180W and -50S to 50N degrees. Current version is V7 and the dataset is published by Goddard Earth Sciences Data and Information Services Center (GES DISC). A summary for the dataset is available on the [https://disc.gsfc.nasa.gov/uui/datasets/TRMM_3B42_7/summary GESC DISC website]. A full technical description of this and other TRMM products can be found in the [https://pmm.nasa.gov/sites/default/files/document_files/3B4XRT_doc_V7_4_19_17.pdf Real-Time TRMM Multi-Satellite Precipitation Analysis Data Set] documentation. From this document:

"... *3B42RT* is the official PPS identifier of the HQ+VAR data set. The identifier indicates that it is a level 3 (gridded) product with input from multiple sensors ("B") using non-TRMM data ("40"-series), running in Real Time. The *units of the TMPA-RT estimates* as stored in the data files is 0.01 mm/h for both precipitation and random error. To recover the original floating-point values in mm/h, divide by 100. The precipitation values are based on satellite snapshots. ... The *period of record* for the TMPA-RT is 1 March 2000 through the present. The start is based on the start of the CPC merged 4 Km IR Tb data set. In contrast, the Version 7 TRMM product 3B42 provides after-real-time processing of the TMPA from 1 January 1998 to the delayed present. ..."

=== '''License''' ===

Tropical Rainfall Measuring Mission (TRMM) (2011), TRMM (TMPA) Rainfall Estimate L3 3 hour 0.25 degree x 0.25 degree V7, Greenbelt, MD, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] [[https://disc.gsfc.nasa.gov/datacollection/TRMM_3B42_7.html 1]]]

Access to GES DISC data requires all users to be registered with the Earthdata Login system, even if we downloaded the data for you you should register. It is important for the data providers to have an estimate of users to justify the work that goes with providing the data services. Complete terms of use and information on how to sign up are available on the [https://wiki.earthdata.nasa.gov/display/EL/Earthdata+Login+Overview+and+Policy,+v1.2 GEOS-DISC data policy webpage] .

=== '''TRMM 3B42 at NCI''' ===

Directory structure is:

/g/data/ua8/Precipitation/TRMM/3B42/<format>/<YEAR>/<files>

Available formats are hdf and netcdf4

==== '''HDF-EOS''' ====

Filenames are:

3B42.<YYYYMMDD>.<HH>.<version>.HDF for example: 3B42.19980203.21.7.HDF

The files are in HDF4-EOS format. To access this dataset using python you need to use the pyhdf module, other hdf modules cannot deal with the HDF-EOS format. 

We are using the CRC32 checksum to check the files integrity, the script to download, update and check the files is available in the CMS github:

[https://github.com/coecms/CollectionsScripts/blob/master/NASA_TRMM/trmm-download-checks.py trmm-download-checks.py]

Usually new files are produced with a 3 months delay from the current date.

==== '''Netcdf4''' ====

We are currently downloading the same version in netcdf format.

Each year has been concatenated in one file with the exclusion of the current year 

which would be in the <year> subdirectory.

[[Category:Dataset]] [[Category:Clex-managed-data]]

Finding datasets

2022-11-06T21:55:10Z

P.petrelli:

== '''Finding climate data at NCI''' ==

There are a lot of climate data resources which are available at NCI. Only some of them have been listed in the NCI data catalogue, and even then there are a few tips that can help you working out if the dataset you are looking for is already available or not.

 

=== '''Check the [https://geonetwork.nci.org.au/geonetwork/srv/eng/catalog.search#/home NCI data catalogue]''' ===

NCI uses a geonetwork catalogue to list its collections, this covers most of the bigger data collections hosted by NCI. To find a dataset you can run a text search or select from a list of available attributes.

'''Free text search'''

Geonetwork free text search will look for an exact match in the record title and description. For example, if I am looking for "precipitation" datasets I need to type the entire word, if I type only "precip" it will return only one dataset containing this sentence in its description:

"Proportion of days per month with precip > 0.2mm: ..."

Likewise typing "precipitation" will return 55 datasets including any record that has the exact word in their title or description.

'''Selection by topic and other attributes'''

NCI created some categories dividing the datasets based on "topic", they are shown on the catalogue main page.  Once you click on one of them, you will see a panel on the left side of the screen showing all the available attributes you can use to select datasets. The same is true if you run a text search, you could then refine your selection using this panel. Probably the most useful attributes are the keywords, they are based on keywords set by the data manager themselves or on [http://www.abs.gov.au/Ausstats/abs@.nsf/Latestproducts/6BB427AB9696C225CA2574180004463E?opendocument Field Of Research (FOR) standards].

The dataset descriptions and attributes are provided by the owners of the data, and sometimes overruled by NCI, so there is a big variance in what is included and how keywords are chosen or interpreted. For example often the FOR codes "0401 Atmospheric Sciences" can also appear as  "0401", "Atmospheric Sciences" and "ATMOSPERIC SCIENCES" or one of its sub-categories "040107 - Meteorology". 

Another limitation is that a dataset might be available as part of a data collection but not having its own record. If the collection is well described, then at least you should be able to locate it,  sometimes the record descriptions are very generic and don't necessarily list all the datasets included.

NCI is working on an improved and more user-friendly interface and is also working with all the data managers to improve the quality of the records. How the data projects will be re-organised it is not yet defined but [[Data_projects_update|this page]] has some information on how this is likely to impact climate datasets. Feel free to send them feedback or tell us if you prefer and we will pass it on.

 

=== '''Check the CMS wiki''' ===

The CMS wiki [http://climate-cms.wikis.unsw.edu.au/Category:Clex-managed-data lists all the datasets] we download and manage for our researchers. Some of these are also listed in geonetwork, mostly are not because in some cases we have downloaded only a small subset.

 

 

=== '''Ask the helpdesk:  [mailto:cws_help@nci.org.au cws_help@nci.org.au] ''' ===

If you still cannot locate the datasets you were looking for, or you find the dataset but the description was not sufficient to work out if it is what you need, then feel free to e-mail us on the helpdesk.

As both geonetwork and the wiki are potentially incomplete or some of the records might be out of date, it is always a good idea to double check with us. We also might know about other data which is available on the NCI server, but not necessarily listed or enquire on your behalf to our partners to help you locate a specific product.

If what you are looking for it is not available yet, we can help you downloading the data. When we receive a request to download data, we try to get back to you as fast as possible with an answer and a timeframe. We are usually able to download the data for you and put in a shared environment where others can also access it. If the dataset requires a lot of storage, the download is time consuming or needs ongoing maintenance we have to check if it is in the Centre objectives with the Infrastructure Committee before going ahead.

It is rare that we have to say no to someone, and we do not do it without a fairly strong reason, because we prefer to download and manage the dataset for you. Having the data shared in a central location, where others can also access it, is a better use of disk storage.

This [[Requesting_data|wiki page]] covers how to request data in more detail.

== '''Accessing the data''' ==

Once you know where the dataset is located, you need to join the relevant project, as for any other NCI project you go to

[https://my.nci.org.au https://my.nci.org.au] , search for the project and put in a request.

The lead CI for that project will receive an e-mail and either approve your request or contact you for further information. While this might feel frustrating there are very good reasons why a data manager might want to know how you will use the data. They might want to be sure you are aware of the dataset limitations and not using it improperly. They might want to know which subset of the data users are actually interested into and they usually need to justify the time and effort that goes into maintaining a dataset. This is becoming increasingly important for NCI, too. They have to prove that their funding is spent in useful ways. NCI is currently reviewing all data projects, their aim is to have a separate project for each dataset, if possible, and to make sure every single user has to request access to the data project as opposed to have world readable files.

 

==== '''Datasets hosted on NCI servers and managed by CLEX''' ====

*'''[[:Category:Clex-managed-data|Replicas of external datasets]]''' 
*'''[[:Category:Published_Data|ARCCSS and CLEX published datasets]]''' 

==== '''Other datasets hosted by NCI''' ====

*CMIP3 - Coupled Model Intercomparison Project Phase 3 
*[[CMIP|CMIP5]] - Coupled Model Intercomparison Project Phase 5 
*[[CMIP|CMIP6]] - Coupled Model Intercomparison Project Phase 6 
*[https://cordex.org CORDEX] - Coordinated Regional Climate Downscaling Experiment
*[http://climate-cms.wikis.unsw.edu.au/ERA5 ERA5] - ECMWF latest reanalysis

'''NB''' we have a tool [https://clef.readthedocs.io/en/latest/readme.html CleF - climate finder] to help you locate the CMIP and CORDEX data on gadi as well as request for new CMIP and CORDEX data to be downloaded. All the datasets listed above are also managed by NCI.

*[[AGCD|AGCD - Australian Gridded Climate Data]]
*[http://openradar.io Australian Operational Weather Radar Archive]
*[https://geonetwork.nci.org.au/geonetwork/srv/eng/catalog.search#/home NCI geonetwork records] - to see the metadata record
*[https://dapds00.nci.org.au/thredds/catalog.html NCI Thredds catalog] - to access the actual files

 

==== '''ARCCSS and CLEX datasets and software published on Research Data Australia (RDA)''' ====

Since the start of ARCCSS we published  our datasets on Research Data Australia (RDA), the Australian Research Data Commons (ARDC) metadata repository. This is a metadata catalogue so it doesn't provide direct access to the data itself. However, it allows us to extend our data description, and have records also for research programs, authors and software all in one place. The first datasets to be published were from the Climate Model Downscaling Data for Impacts Research (CliMDDIR), then the ACCESS CMIP5 simulations.

*[[ACCESS_CoE_simulations|ACCESS]] - CMIP5 simulations
*[http://climddir.org/ CliMDDIR] - Climate Model Downscaling Data for Impacts Research
*[[ARCCSS_collection_datasets|CLEX and ARCCSS collections]] - CLEX and ARCCSS datasets from the NCI Data Catalogue
*[https://researchdata.ands.org.au/atmospheric-access13-historical-sub-project/645160 C20C+ ACCESS] - Atmospheric ACCESS1.3 historical all forcing model output for the Climate of the 20th Century Plus (C20C+) Detection and Attribution sub-project
*[https://researchdata.ands.org.au/marine-heatwaves-detection-code/814983 MarineHeatWaves] - Marine heatwaves detection code

 

==== '''CLEX Code Collection on Zenodo''' ====

In 2020 we started a Zenodo community to collect and publish software, Zenodo allows us to get a DOI for the code records. Publishing the code associated to a paper is now often required when submitting an article to a journal. It also allows us to publish software created by researchers and students that could be useful to other and by the CMS team itself. Zenodo is an international repository and as such can reach a wider audience. If you would like to contribute to this collection just let us know.

''' '''[https://zenodo.org/communities/arc-coe-clex/?page=1&size=20 CLEX Code Collection Zenodo community]

 

== '''External data resources''' ==

If you have not yet found a dataset that suits your needs on NCI, there are still a lot of external data resources, you can browse online and sometimes easily access data from.

Rather than going straight to a specific dataset, you should first make sure the data you want to use is fit for purpose. There are sites that offer reviews of climate datasets that can help you doing just that. It is always important to know how the data was generated before using it even if a dataset was recommended by someone else. There might be new better alternative datasets or others might have discovered and shared issues with the data. Doing a bit of research before using the data might save you a lot of time and stress. 

These two are a very good place to start.

*[https://climatedataguide.ucar.edu NCAR Climate data guide] 
*[https://reanalyses.org Reanalysis.org] 

=== '''Australian data portals''' ===

*[https://researchdata.edu.au Research Data Australia] - metadata repository for all Australian research datasets 
*[http://www.bom.gov.au/climate/data/ BoM Climate data services]
*[http://www.bom.gov.au/cyclone/history/tracks/ BoM Tropical Cyclone data services]
*[http://dl.tpac.org.au TPAC][http://dl.tpac.org.au/thredds/catalog.html   Climate and Ocean data portal]
*[https://portal.aodn.org.au AODN] - Australian Ocean Data Network (includes IMOS observation data)

=== '''Global data portals''' ===

*[https://www.ncdc.noaa.gov/data-access/paleoclimatology-data/datasets NOAA paleoclimatology data repository ] - I am highlighting the paleo data because we are also publishing our paleoclimate datasets in this repository, but NOAA data collection includes also much more ocean and atmospheric data, you can access them also starting from the same link.
*[https://ooi-website.whoi.edu/data-portal/ OOI - Ocean Observatory Initiative]
*[https://podaac.jpl.nasa.gov/ NASA JPL data portal ] - including visualisation tool
*[https://earthdata.nasa.gov/ NASA earthdata portal]

A relatively recent option are remote desktops or other cloud resources with integrated data access

*[https://cds.climate.copernicus.eu/cdsapp#!/toolbox Copernicus Climate Change Service Toolbox]
*[https://giovanni.gsfc.nasa.gov/giovanni/ GIOVANNI] - GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure

Interdisciplinary data repositories that include climate related dataset

*[https://research.jcu.edu.au/data/default/rdmp/home Research Data JCU (James Cook University)]
*[https://aurin.org.au AURIN] - Australian Urban Research Infrastructure Network 

Google has developed a [https://toolbox.google.com/datasetsearch dataset search toolbox], currently is only a beta version, you can find here an [https://www.blog.google/products/search/making-it-easier-discover-datasets/ overview] . All the datasets published via the centre are also discoverable with this tool.

 

=== '''Accessing remote data: OPeNDAP''' ===

Many of these repositories will allow you to access the data remotely without downloading it. Some will have developed their own tools as we saw above for the cloud and remote analysis web tools, but most will do that by making their data available through OPeNDAP a web-based software that allows users to access datasets remotely. Many software used for analysis recognise an OPeNDAP url as a filename. An OPeNDAP url is usually constituted by the remote address of the file followed by optional constraints.

Usually any data repository that uses a thredds server, including the NCI data collection, makes the data available via OPeNDAP.

We have a blog that demonstrates [https://climate-cms.org/2019/01/18/using-opendap.html how to build and use and OPeNDAP url], other more in depth information on OPeNDAP is available from their [https://www.opendap.org/ website], including a list of software that understand this protocol.

Using OPeNDAP is particularly useful when you need only a subset of a dataset, like a limited region or timeseries of the data. It is also a good idea if you are accessing a dataset that gets updated often so you can always get the latest available version of the data.

==== '''Climate related repositories using OPeNDAP''' ====

[http://dap.nci.org.au/thredds/catalog.html NCI thredds catalogue]

[https://rda.ucar.edu/thredds/catalog/catalog.html NCAR thredds catalogue]

[http://dl.tpac.org.au TPAC][http://dl.tpac.org.au/thredds/catalog.html   Climate and Ocean data portal]

[https://erddap-uncabled.oceanobservatories.org/uncabled/erddap/index.html OOI ERDDAP catalogue] - Ocean Observatory Initiative

[https://podaac-opendap.jpl.nasa.gov/opendap/hyrax/allData/ NASA JPL PO.DDAC repository] - Physical Oceanography Distributed Active Archive Center

[https://search.earthdata.nasa.gov/search NASA Earth Observing System Data and Information System (EOSDIS) opendap servers] this includes also a [https://developer.earthdata.nasa.gov/opendap/resources/eosdis-opendap-servers tutorial] to use OPeNDAP

[http://thredds.ucar.edu/thredds/catalog.html UCAR thredds catalogue]

[https://thredds.daac.ornl.gov/thredds/catalogs/ornldaac/ornldaac.html NASA ORNL DDAC repository] - Oak Ridge National Laboratory Distributed Active Archive Center  for biogeochemical dynamics

[https://disc.gsfc.nasa.gov/information/tools?title=OPeNDAP%20and%20GDS NASA GES DISC repository] - atmospheric composition, water & energy cycles and climate variability 

 

[[Category:Data induction]]

CLEx induction

2022-10-10T02:57:23Z

P.petrelli:

The CMS team is here to help you with the technical aspects of your work. A lot of you (probably all) will have to work with computer code and software, usually in a Linux/Unix environment. You might also need to access and/or share large amounts of data, work on different servers, etc.

Some of you will be somewhat familiar with these tasks whereas they will be new to others. We are here to help whatever your technical knowledge is. We are happy to answer any question, we can't necessarily do everything but feel free to ask as we might have ideas or we might know who you should contact and what to exactly ask.

The members of the [[CMS|CMS]] team are:

{| class="wikitable"
|-
! Name
! Sitting at
! Email
! CMS "local" support person for
|-
| Paola Petrelli
| U Tasmania
| paola.petrelli@utas.edu.au
| U Tasmania
|-
| Holger Wolff
| Monash
| holger.wolff@monash.edu
| Monash
|-
| Ramzi Kutteh
| UNSW
| r.kutteh@unsw.edu.au
| UNSW
|-
| Sam Green
| UNSW
| s.green@unsw.edu.au
| UNSW
|-
| Dale Roberts
| ANU
| dale.roberts@unimelb.edu.au
| UniMelb
|}

The "local" support person will be the CMS person welcoming you to the Centre and possibly giving you an induction. But you are more than welcome to contact any of the CMS team members for support.

Although we won't ignore emails sent to our individual addresses, our preferred way of contact is through our help desk email address: [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Contacting the help desk will make sure your email is seen by one of us in case the person at your institution is away for example. Don't worry about the help desk email being an NCI account, you can still ask any question there.

[[Getting_Help|Here]] are some guidelines on formulating your requests to CMS

 

= Set up your computer =

We have put instructions for the setup we recommend for each operating system on the following pages:

*[[Getting_started_on_Windows|Windows]]
*[[Getting_started_on_Mac|Mac]]
*[[Getting_started_on_Linux|Linux]]

= Step 1: Get an account with NCI =

NCI operates the computers that you will be doing most of your work on. NCI provides services to the Centre, they are not part of the Centre of Excellence.

To get access to NCI servers, you need to do three things:

1. You need to find out which project(s) you need to be connected to.

Your project decides who gets billed for what you do on the NCI Servers. Your supervisor should be able to tell you which project(s) you should get connected to. Be mindful that there are two types of projects: projects for computation and projects for datasets. The mapping of the computational projects for CLEX can be found [[CLEX_projects_at_NCI|here]]. We recommend to join all the projects relevant to your research project but to choose 1 default project to work from. The same page with the list of computational projects explains how to set your default project. The projects for datasets do not have any computational time added to them so you need to make sure you ask for at least one computation project. Your supervisor should know. Note if you will run the ACCESS model or the UM model, you need to ask a connection to the "access" project in addition to at least one computation project.

2. You need to register with NCI.

This can be done on [https://my.nci.org.au/mancini/signup/0 NCI registration page] You will need to supply some information, and read and accept their policies. In order to be added to the system, you need to request connection to at least one project and be accepted on the project. Note that this can take a bit of time until the Lead chief investigator of the project has received and confirmed your request via email. Note that this is an automatic email process, you don't need to send an email to the lead chief investigator. Although, you might want to do so if you think you need to introduce yourself to him/her. Note that you should be able to request connection to several projects while getting signed up.

3. You might want to ask your supervisor whether you also should be connected to any of the following data projects.

These don’t provide any computing allowances, but they give access to certain datasets or models that you might need.

{| class="wikitable"
|-
! Project
! Purpose
! Administrator
|-
| '''ua8'''
|  GSWP3, NCEP Polar sst, OISST, CMIP5 ocean processing, model ensemble data + others
| CMS
|-
| '''rq7'''
| [[YOTC|ECMWF Year of Tropical Convection re-analysis]]
| CMS
|-
| '''ks32'''
| ARCCSS and CLEX published data
| CMS
|-
| '''ub4'''
| [[ERA_INTERIM|ECMWF ERA Interim re-analysis 6 hrs data]] and
| CMS
|-
| '''cb20'''
| [[CMIP|CMIP3]] replica data
| NCI
|-
| '''al33'''
| [[CMIP|CMIP5]] replica data
| NCI
|-
| '''rr3'''
| [[CMIP|CMIP5]] published data
| NCI
|-
| '''oi10'''
| [[CMIP|CMIP6]] replica data
| NCI
|-
| '''fs38'''
| [[CMIP|CMIP6]] published data
| NCI
|-
| '''rr7'''
| Use only for JRA55, MERRA2 the other datasets have been removed
| BoM
|-
| '''cable'''
| Give access to the [https://trac.nci.org.au/trac/cable/wiki CABLE model] and [[CABLE_Data|CABLE benchmarking dataset]]. Please read the [https://trac.nci.org.au/trac/cable/wiki/CableRegistration CABLE registration] information.
| CSIRO
|-
| '''access'''
| Give access to the [[Unified_Model|Unified Model]] and ACCESS tools
| CMS - BoM - CSIRO
|-
| '''rt52'''
| [[ERA5|ECMWF ERA5 re-analysis]] single and pressure levels
| NCI
|-
| '''zz93'''
| [[ERA5|ECMWF ERA5 re-analysis]] Land data
| NCI
|-
| '''sx70'''
| Shared input data for WRF
| CMS
|-
| '''ik11'''
| Shared data for COSIMA. Inputs and outputs
| COSIMA
|-
| '''ia89'''
| 400m resolution ACCESS limited-area simulation over Australia
| CMS
|-
| '''gy85'''
| Himawari-8 GeoCat 1.0.3 Australian Domain
| CMS
|}

 

= Step 2: Set up your Connection =

The standard method of connecting to the NCI systems is SSH. We want you to use a passphrase protected private key authorisation method with an ssh-agent and agent forwarding for convenience. If you understood every part of that sentence, go ahead and set it up. If not, the next part describes what to do.

The instructions describe how to set up your connection to both gadi.nci.org.au and accessdev.nci.org.au. You '''can not''' set up a connection to accessdev if you are not going to use the ACCESS or UM models. So please only follow the instructions you need!

 

== Linux and MacOS: ==

#'''Create the ssh directory''' on your computer <syntaxhighlight lang="bash">$ mkdir -p ~/.ssh</syntaxhighlight>

 

#'''Create a config file''' <syntaxhighlight lang="bash">$ nano ~/.ssh/config </syntaxhighlight>

Enter the following contents, replacing xx0000 with your NCI user name.
<syntaxhighlight lang="text">
Host gadi gadi.nci.org.au
HostName gadi.nci.org.au
User xx0000
ForwardX11 yes
ForwardX11Trusted yes
Host access accessdev.nci.org.au
HostName accessdev.nci.org.au
User xx0000
ForwardX11 yes
ForwardX11Trusted yes
ForwardAgent yes
</syntaxhighlight>
If you use a Mac, and you want to store the key in your keychain so that it gets automatically loaded when you log in, you can add this section to your config file: <syntaxhighlight lang="text">
Host *
AddKeysToAgent yes
UseKeychain yes
</syntaxhighlight>

#'''Create a key pair.''' It is imperative that you select a strong passphrase when asked for it. <syntaxhighlight lang="bash">$ ssh-keygen -t rsa</syntaxhighlight>

 

#'''Distribute the key pair.''' On most systems, there is a useful script to distribute the key pairs called '''ssh-copy-id''': <syntaxhighlight lang="bash">
$ ssh-copy-id gadi
$ ssh-copy-id access
</syntaxhighlight>

If that script fails, you have to distribute the keys manually with these commands:
<syntaxhighlight lang="bash">
$ cat ~/.ssh/id_rsa.pub | ssh gadi "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"
$ cat ~/.ssh/id_rsa.pub | ssh access "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"
</syntaxhighlight>

#'''Test whether you have an agent.''' If you have an agent already running, this command will ask for the passphrase for the just-created key pair: <syntaxhighlight lang="bash">$ ssh-add</syntaxhighlight>

If the last command told you that it couldn’t open a connection, then you don’t have an agent. Come talk to someone of the CMS team, we will help you set it up.

 

== Windows ==

Since Windows has a different heritage to Mac and Linux some extra setup work is required to use SSH. See the [[Getting_started_on_Windows|Getting started on Windows]] page for details.

If you're using [[Getting_started_on_Windows#Windows_Subsystem_for_Linux_.28WSL.29|WSL]] or [[Getting_started_on_Windows#Cygwin|Cygwin]] use the [[#Linux_and_MacOS:|SSH instructions for Mac and Linux]] above.

If using Power Shell, generate your ssh key with (Power Shell doesn't support 'rsa' keys)
<syntaxhighlight lang="unknown">
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa
</syntaxhighlight>
The contents of C:\Users\USER\.ssh\id_ecdsa.pub then need to be copied into ~/.ssh/authorized_keys on Gadi manually, as Power Shell doesn't have the ssh-copy-id command. The Power Shell SSH config file is located in your user directory, e.g. C:\Users\USER\.ssh\config
== SSH-keys for file transfers ==

Additionally, some groups transfer data from NCI to outside NCI, for example for storage on University maintained servers. For these transfers, the best way to proceed is to create a restricted ssh key pair used only for file transfer. The setup is explained in [https://opus.nci.org.au/display/Help/Using+SSH+keys Using restricted commands for transferring files]. Most of the instructions are still valid, except for step 3. The restricted command prefix should now be:
<syntaxhighlight lang="text">
from="gadi-dm*.nci.org.au,gadi*.nci.org.au",command="~/bin/rrsync /data/archive",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa
</syntaxhighlight>

= Step 3: Basics of the NCI system =

We are mainly using 3 systems of the NCI: gadi, accessdev and VDI. Gadi is the supercomputer that we run our models on. Accessdev is a virtual server that we have software installed that are required to allow the running the models. VDI is a virtual desktop designed for interactive analysis of your results. All these servers run Linux as operating system.

== Read the manual! ==

Considering you will be using systems that are provided by the NCI, you should take the time to read the documentation provided by them about their systems. That is everything on their [https://opus.nci.org.au/display/Help/NCI+Help help knowledge base].

== Additional notes ==

=== Modules: ===

The servers have a lot of software installed, some of which they have multiple versions of. If you need to use a software, you usually will need to load a certain module. A full help of the module command is accessible through the man pages on gadi. Here are the most important commands regarding the modules:

{|
|-
| $ module help
| Show help about the module system
|-
| $ module avail
| List all available modules (a lot!)
|-
| $ module list
| List the currently loaded modules
|-
| $ module load intel-compiler
| Load the default version of the intel compiler
|-
| $ module load intel-compiler/2019.3.199
| Load a specific version of the intel compiler
|-
| $ module switch intel-compiler/2019.3.199
| Switch to a different version of the intel compiler (unload and load)
|-
| $ module unload intel-compiler/2019.3.199
| Unload the specified version of the intel compiler
|-
| $ module use /g/data/hh5/public/modules
| Add our Centre's curated modules to the list of available modules
|}

Some modules conflict with one another, for example different versions of the same software. You might have to unload one module first to load a different one.

=== PBS: ===

The other important system you need to understand in order to work on gadi is the scheduler PBS. This is the system for running software on dedicated nodes inside gadi. Basically, a user tells PBS that they want to run this job on this many CPUs with these resources for at least so long, and PBS places it in a queue, and when the required resources are available, it will assign this job to them.

Again, NCI has a very good documentation about how to run this [https://opus.nci.org.au/display/Help/How+to+submit+a+job here]

The most important commands are:

*'''qsub''' – Submit a job to the queue
*'''qstat''' – Display the status of the queued jobs
*'''qdel''' – Remove a job from the queue

Of these, qsub is the most important, and there are so many parameters that we won’t tell them all here.

In general, the fewer resources you request, the earlier your job will run. But if your program exceeds the requested resources at any time, PBS will kill the job, and you have to start over.

More specifically:

*Gadi is comprised of nodes with 48 CPUs each. If you request less than 48 CPUs, you will get the number of CPUs that you request. If you request more, you will need to use a multiple of 48 CPUs.
*Nodes come with 192GB of memory to share within the 48 CPUs. So if your job requires 48 or more CPUs, you might as well request all the memory, for example 384GB for a 96-cpu job.

Running a job on the queue costs so-called '''SU'''. On the old raijin, one SU was equivalent to one core for one hour. On the much faster new gadi, one core for one hour costs 2 SU. Note that on gadi, the SU cost is calculated using the higher of the number of cores and the memory usage. So if you request 5 cores but only 2GB memory, its SU cost is based on the 5 cores. If you request 1 core but 96GB of memory (half of the whole node), the cost will be based on 24 cores (half the whole node).

= Step 4: How to run models =

A lot of information to gain access to models or run the models can be found on the ARCCSS CMS wiki (see link at the end).

A very important message from the CMS team: Often you will work with source code. Please ensure that your version of the source code is under version control! We use SVN and git (usually hosted on github) for the model codes and smaller utilities.'''DO NOT JUST COPY SOURCE CODE! CHECK IT OUT, OR CHECK IT IN!'''

== UM ==

The Unified model is currently transitioning from the old user environment umui to the new rose/cylc user interface. Either way, you need to log into accessdev.nci.org.au.

=== umui(x) ===

The old user interface has to be used for versions up to 7.x, and can be used for 8.x. umui was developed by the UM, umuix is very similar, but more convenient developed by CSIRO. An introduction into its use is on our wiki: [http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI]

=== rose/cylc ===

The new user interface can be used from version 8.x, and has to be used for versions 9.0 and later.

== WRF ==

The CMS team ports the WRF model at NCI. Information on using the model is on [[WRF|the CMS wiki]]. Users should also be aware that we do not reproduce the information from the [http://www2.mmm.ucar.edu/wrf/users/ WRF model users website] which stays a very important resource for all users. In particular, every new user is strongly encouraged to run through the official tutorials (found under User Support on the WRF model users site). As found on the WRF pages on this wiki, only the compilation steps differ from the NCAR tutorial.

== CABLE ==

CABLE is under licence so you need to request access to the code. All the information you need to get access and use CABLE is on the [https://trac.nci.org.au/trac/cable CABLE trac site]

= Data management =

Most of the information on how the data is managed is stored in this wiki in the [[Data|Data Services]] page. Here you can find information on the datasets managed by the Center as well the relevant policies in regard to data. Take some time to get familiar with it, here are some suggestions:

#We now offer a 1-day Data Induction training, if you have an opportunity to participate do so, particularly if you are just starting. If you missed out, all the [[Data_induction|course materials]] are stored on the wiki and we tried to create a path for you touching all the relevant wiki pages. There are 3 parts to the data induction, the first covers the policies, the second focuses on publishing and the third the data and code management best practices. 
#We strongly suggest to have a [[Data_Management_Plan|data management plan]] from the start of your project. You can use the [https://clex.dmponline.cloud.edu.au/ CLEX DMPonline tool] to do this. Because this tool stores your plan into a database, you can save it and access it as many times as you want, which means you can build your data management plan as you progress with your research. It is good though to start as soon as you can, DMPonline is structured as to teach about data practices and the resources available to you, it also informs the CMS team on which data, software and/or infrastructure you are planning to use, so we can better tailor our efforts to your needs.
#Get familiar with [[Publishing_options|publishing options]], while this might seem faraway, it is a good idea to keep track from now of what you are doing in regard to data and metadata, as it is often much more difficult to retrieve this information at the end of your project. There are also some technical times involved in getting data properly published.
#Consider creating a [[Persistent_research_identifiers|researcher-ID]] to uniquely identify your work.
#Make sure you browse through all the available data sources before you download data, this wiki page cover [[Finding_datasets|how to find out if a dataset]] is available at NCI. You could be wasting time and storage downloading something which might be already available, or that we could download and manage for you
#Become familiar with the different filesystems available on any server you are using, make sure you know which are meant to be working spaces, where you can archive your data, how to share data with collaborators and best practices to transfer data within and out of the system. If you are using gadi on NCI you can find this information on their [http://nci.org.au/nci-systems/national-facility/data-storage/filesystems/ user support ]and training sections.
#You can explore a lot of data related information in the [https://ardc.edu.au Australian Research Data Commons] website
#Don't hesitate in contacting cws_help<at>nci.org.au for any question you might have, or also e-mail directly paola.petrelli<at>utas.edu.au, if you have a more general question or you are confused about data management. We are always available to discuss with you the details of your project and give you advice.

= How to learn more =

== General setup: ==

a blog post by a UNSW's PhD student on a few things to know and learn: [http://web.maths.unsw.edu.au/~oangelil/blog/blog2/page_blog2.html Oliver Angélil's post]

== CMS: ==

*E-Mail: [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*Chat with us on [https://arccss.slack.com Slack] (Use your university email in your sign up.)
*This [[Home|Wiki]]
*In person.

== NCI: ==

[https://opus.nci.org.au/display/Help/NCI+Help Guides and reference pages]

[http://nci.org.au/services-support/training/ Training material] and [https://training.nci.org.au/ this]

== Version Control with git: ==

(Note: git might also be very useful for your thesis!) [https://www.youtube.com/playlist?list=PLg7s6cbtAD165JTRsXh8ofwRw0PqUnkVH Git video tutorial by GitHub]

== Data management: ==

[[Data_induction|Data induction]]

= Getting help =

Your first port of call for help should be the CMS team via the help-desk: [mailto:cws_help@nci.org.au cws_help@nci.org.au]. This help desk is followed by all the members of the CMS team and by some staff from NCI. [[Getting_Help|Here]] are some guidelines on formulating your request

[[Category:Training]]

Scratch file expiry

2022-10-09T23:34:33Z

P.petrelli:

'''On 17th May 2022''', NCI is introducing an automatic system to purge unused data from the scratch filesystem.   Below we are giving some steps on how best to prepare for that change. We also give some information on some useful tools to prepare a good data workflow going forward.

= Preparation =

Your best course of action is to start preparing now to avoid having any useful data going into quarantine on the 17th May. Below are the steps we recommend you to follow:

*[https://opus.nci.org.au/pages/viewpage.action?pageId=156434436 '''Read''' '''the information'''] provided by NCI
*'''Clean up /g/data''': delete what you can, archive to tape or outside NCI what you can.
*'''Clean up /scratch''': delete what you can, move to /g/data or tape or outside NCI if you need long term storage without accessing it.
*Run “'''nci-file-expiry list-warnings -p <project> > expiry_warning_<project>.txt'''” for all the projects you are a member.
*'''Check the output''' of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Run “'''nci-file-expiry list-warnings'''” : this will catch any file you own in any project you may have forgotten about and decide what to do with anything that appear there. From October 2022 gadi shows, on login,  a summary of files which are about to expire or that are quarantined.
*'''Rethink your data pipeline''': do not leave data you are not using anymore in /scratch. Decide what you need to do with it when you stop using it: delete, move to /g/data or outside NCI or archive to tape.
*'''Manage your data under /g/data regularly''': review your data, archive to tape or outside NCI as necessary.

= Additional information =

Below is a list of resources you might find useful to prepare for the automatic file expiry and build a good data workflow for the future.

[[Archiving_data|Archiving data at NCI]]

[https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html Blog on building a sustainable data workflow]

[[Storage|Description of the various filesystems at NCI]]

 

 

= Long term strategy =

#'''Keep using /scratch'''. There is not enough disk space if nobody is using /scratch. Your data is safe on /scratch as long as you access it, i.e. read, modify, create. Only the data you won't use for some time needs to be managed.
#'''Managing your data does not mean storing everything on /g/data. '''You need to think about what future use you have for that data. If you don't need it anymore and it can be reproduced, delete it. If you will need to publish the data, publish it now. It is a lot easier to publish a new version if you need a small change than to publish the initial version. For information, on what data needs to be published see the [[CLEX_Data_policy|CLEX Data Policy]] and [[Which_data_should_I_publish|Which_data_should_I_publish]] page. If you can't publish now but won't need it for a long time, put the data on tape. If you won't need this data for a short time but you know you will get back to it soon, put it on /g/data
#'''Learn how to use the tape system'''. See [[How_to_use_MDSS_tape_storage_at_NCI|how to use the MDSS tape system]] or  [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html our blog].
#Try and '''avoid your data getting into quarantine'''. Make managing your data '''a periodic, frequent task''' in your calendar. This includes all data: /scratch, /g/data and tape.
#'''Be careful when going on leave!''' Make sure to check your data holdings under /scratch before your leave.
#'''For extended leave''' (e.g. ship cruises, parental leave, sabbatical), '''do not leave any important data under /scratch.'''

Scratch file expiry

2022-10-09T23:33:59Z

P.petrelli:

'''On 17th May 2022''', NCI is introducing an automatic system to purge unused data from the scratch filesystem.   Below we are giving some steps on how best to prepare for that change. We also give some information on some useful tools to prepare a good data workflow going forward.

= Preparation =

Your best course of action is to start preparing now to avoid having any useful data going into quarantine on the 17th May. Below are the steps we recommend you to follow:

*[https://opus.nci.org.au/pages/viewpage.action?pageId=156434436 '''Read''' '''the information'''] provided by NCI
*'''Clean up /g/data''': delete what you can, archive to tape or outside NCI what you can.
*'''Clean up /scratch''': delete what you can, move to /g/data or tape or outside NCI if you need long term storage without accessing it.
*Run “'''nci-file-expiry list-warnings -p <project> > expiry_warning_<project>.txt'''” for all the projects you are a member.
*'''Check the output''' of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Run “'''nci-file-expiry list-warnings'''” : this will catch any file you own in any project you may have forgotten about and decide what to do with anything that appear there. From October 2022 gadi will show a summary of files which are about to expire or that are quarantined on login.
*'''Rethink your data pipeline''': do not leave data you are not using anymore in /scratch. Decide what you need to do with it when you stop using it: delete, move to /g/data or outside NCI or archive to tape.
*'''Manage your data under /g/data regularly''': review your data, archive to tape or outside NCI as necessary.

= Additional information =

Below is a list of resources you might find useful to prepare for the automatic file expiry and build a good data workflow for the future.

[[Archiving_data|Archiving data at NCI]]

[https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html Blog on building a sustainable data workflow]

[[Storage|Description of the various filesystems at NCI]]

 

 

= Long term strategy =

#'''Keep using /scratch'''. There is not enough disk space if nobody is using /scratch. Your data is safe on /scratch as long as you access it, i.e. read, modify, create. Only the data you won't use for some time needs to be managed.
#'''Managing your data does not mean storing everything on /g/data. '''You need to think about what future use you have for that data. If you don't need it anymore and it can be reproduced, delete it. If you will need to publish the data, publish it now. It is a lot easier to publish a new version if you need a small change than to publish the initial version. For information, on what data needs to be published see the [[CLEX_Data_policy|CLEX Data Policy]] and [[Which_data_should_I_publish|Which_data_should_I_publish]] page. If you can't publish now but won't need it for a long time, put the data on tape. If you won't need this data for a short time but you know you will get back to it soon, put it on /g/data
#'''Learn how to use the tape system'''. See [[How_to_use_MDSS_tape_storage_at_NCI|how to use the MDSS tape system]] or  [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html our blog].
#Try and '''avoid your data getting into quarantine'''. Make managing your data '''a periodic, frequent task''' in your calendar. This includes all data: /scratch, /g/data and tape.
#'''Be careful when going on leave!''' Make sure to check your data holdings under /scratch before your leave.
#'''For extended leave''' (e.g. ship cruises, parental leave, sabbatical), '''do not leave any important data under /scratch.'''

ERA INTERIM

2022-08-26T03:30:19Z

P.petrelli:

A subset of the ERA Interim re-analysis data ranging from 1979 to the 31st of August, when ERA Interim production ceased is replicated on NCI server. We downloaded the original grib data from the [http://apps.ecmwf.int/datasets/data/interim_full_daily ECMWF server] , specific variables were then extracted and converted to monthly netcdf files. For more information on the original ECMWF collection visit their website: [https://confluence.ecmwf.int/display/CKB/What+is+ERA-Interim ERA-Interim page]

=== '''Known issues''' ===

There are several known quality issues for the ERA Interim data, here is a list of web resources:

[http://old.ecmwf.int/research/era/do/get/index/QualityIssues ERA Interim quality issues from ECMWF website]

[https://climatedataguide.ucar.edu/climate-data/era-interim Climate Data Guide page on ERAI]

[http://reanalyses.org/atmosphere/overview-current-reanalyses Overview and re-analysis comparison from reanalyses.org]

Some of these analysis are described more in depth in the following article: [http://onlinelibrary.wiley.com/doi/10.1002/qj.828/full The ERA-Interim reanalysis: configuration and performance of the data assimilation system]

We are listing all the issues we are aware of in this [[ERAI-issues|ERAI known issues page]], please feel free to contribute, we just provide the data as it is downloaded from the ECMWF server, we do checks to make sure that our data is the same as the original but we cannot run checks for the original data.

We found very high values for temperature (~330-350 K) near the lower troposphere in the following periods: 26-31 Dec 2002 and 21-25 Dec 2003. Other variables might be similarly affected.

=== '''License''' ===

To comply with the ECMWF licensing you need to be registered with the ECMWF to use this data. To register with the ECMWF you have to sign the [http://apps.ecmwf.int/datasets/data/interim_full_daily/licence/ ERA Interim license and agreement].  

To acknowledge the ERA-Interim data, please refer to the ERA-Interim [https://apps.ecmwf.int/datasets/data/interim-full-daily/licence/ licence] for details on the wording to use.

To cite the source of the data, you may use the following data citation (as part of the bibliography):

European Centre for Medium-range Weather Forecast (ECMWF) (2011): The ERA-Interim reanalysis dataset, Copernicus Climate Change Service (C3S) (accessed ''<insert date of access here''>), available from [https://www.ecmwf.int/en/forecasts/datasets/archive-datasets/reanalysis-datasets/era-interim https://www.ecmwf.int/en/forecasts/datasets/archive-datasets/reanalysis-datasets/era-interim] 

If no specific advice is given by the journals, it is usually recommended that the above data citation is put in the acknowledgements section.

 You also have to add the following paper to your references: [https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.828 https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.828]

=== '''Data access''' ===

You can then request to join the ub4 project by using your NCI account to access [https://my.nci.org.au/ My NCI portal]. Remember you have to register with ECMWF before. Be aware that by requesting to be part of the ub4 project you are automatically agreeing to the ECMWF license agreement terms'''.'''

 

The data is available on gadi under /g/data/ub4/erai/grib for the grib version and /g/data/ub4/erai/netcdf for the netcdf version.

*[https://docs.google.com/spreadsheets/d/1qnQC_Ki5IAwZPD9viV79tfenemPGoYWKfDa5vEwDl90/pubhtml Data Inventory]''': what's available for each level'''

NB  that we download all the available surface fields, both analysis and forecast, but we don't convert all of them to netcdf, unless they are specifically requested.

The netcdf version is organized in monthly field files, while the grib files have all the fields in one file.

The current netcdf version was released in April 2015, the previous version hosted on ua8 is not anymore available. If you requested access before we moved the data to ub4, you are already part of ub4 and don't need to request access again to use the data.

Monthly averaged fields are available in ua8 as part of the CREATE-IP temporary copy hosted in

/g/data/ua8/synda/CREATE-IP/reanalysis/ECMWF/ERA-Interim/

This is a temporary replica , eventually NCI should provide the CREATE-IP dataset in a different project, see this [[Data_projects_update|page]] for details.

If you are having trouble getting access or the data you are looking for is not available there, let us know by e-mailing cws_help<at>nci.org.au .

=== '''Current netcdf version: v1.0''' ===

The main differences with the previous version, which is not anymore available, are:

1) netcdf4 format with compression, which allowed us to get rid of scale and offset

2) new filenames of the form <varname>_<frequency>_ERAI_historical_<level>_<from-date>_<to-date>.nc

Ex. : ta_6hrs_ERAI_historical_an-ml_20110101_20110131.nc

3) some of the variable names have changed to the correspondent CMIP5 standard name, all names are now lower cases

4) new directory structure /g/data/ub4/erai/netcdf/<frequency>/<realm>/<level>/<version>/<variable>/files…

Ex. : /g/data/ub4/erai/netcdf/6hr/atmos/oper_an_ml/v01/ta/

5) there is more metadata information in the files, in particular we adopted CF conventions and added standard_names wherever possible. We also added an attribute MD5 checksum for each variable, this can be use to check for data corruption by using ncks or nco

Most of the data will be in the "atmos" realm, with the exception of ERAI land which is is "land" and few ocean fields. And most files have "6hr" frequency, with the exception of the forecast data which has "3hr" frequency.

The scripts used to convert the files from grib to netcdf are available in the checks github. [https://github.com/coecms/CollectionsScripts https://github.com/coecms/CollectionsScripts]

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

Coding best practices

2022-08-22T00:06:49Z

P.petrelli:

= Introduction =
There are code standards and conventions available depending on the language you are using and sometimes also conventions adopted for specific collaborative projects. These can be quite complex and out of scope if you are writing a code for your analysis, however there are a few things you can do to make your code much more readable and safer from bugs which are quite simple. In the video linked below, kindly provided by DataTAS, the presenter gives some useful tips which can be applied to any language:

"[https://www.facebook.com/1354624204600925/videos/138577474926767 Reproducible research how to write code that is built to last]" 

It is worth watching the video (the actual presentation is about half of the video ~35 minutes) to understand fully how valuable these tips are and also to get a perspective from someone who went from a science background to a commercial software engineering position.

Below is a list of best practices discussed in the video.

=== Naming ===

*Use descriptive names for variables and functions
*Use consistent naming across the code
*Avoid hard-coding values
*Initialising variables

=== Code structure ===

*Indents
*Comments
*Use functions to organise your code 
*Don't Repeat Yourself (DRY) code
*One statement per line
*Write explicit code
*Keep your files a reasonable length
*Clear flow: try to have only one exit point in a function
*Test important parts of your code  

 

= Writing Reusable Code =

There are many definitions of reusable code, and the details often depend on the use to which the code is being put. From [https://en.wikipedia.org/wiki/Code_reuse wikipedia]
<blockquote>
The key idea in reuse is that parts of a computer program written at one time can be or should be used in the construction of other programs written at a later time.
</blockquote>
 

=== Why ===

Why write reusable code? This excellent [https://www.frontiersin.org/articles/10.3389/fninf.2017.00069/full article] articulates what it takes to transform code into a fully fledged scientific contribution, and reusability is a key component:
<blockquote>
Making your program reusable means it can be easily used, and modified, by you and other people, inside and outside your lab
</blockquote>
The process of making your code reusable will also make it better, less error prone, saving time and increasing productivity. 

=== How ===

How do you write reusable code? There are levels of reusability, from re-using your own code to a fully published library/module for others to use. Start with the basics and with experience add more reproducible practices. If you're already doing the basics, try some of the intermediate or advanced ideas.

 

'''Basic'''

The basics of code reusability involve easily readable code and DRY (don't repeat yourself) principles, so using functions/subroutines/procedures to avoid copying blocks of code and modifying each block. The tips in the introduction can all help with code reusability as well as those listed below.

Ten tips for writing readable code (based on PHP but principles are universal):

[https://dzone.com/articles/10-tips-how-to-improve-the-readability-of-your-sof https://dzone.com/articles/10-tips-how-to-improve-the-readability-of-your-sof] 

A good (python specific) section of a course from Software Carpentry about writing functions:

[https://swcarpentry.github.io/python-novice-inflammation/08-func/index.html https://swcarpentry.github.io/python-novice-inflammation/08-func/index.html]

For the beginner python programmer it can be difficult to know exactly '''how''' to go about reusing code, how to organise it and import it into your notebooks or programs. This is a short, clear article about the python specific details on reusing your code:

[https://towardsdatascience.com/creating-reusable-code-for-data-science-projects-740391ec7bad https://towardsdatascience.com/creating-reusable-code-for-data-science-projects-740391ec7bad]

In a similar vein (and also python specific), how and when to use a <tt>main</tt> function in python:

[https://realpython.com/python-main-function/ https://realpython.com/python-main-function/]

 

'''Intermediate'''

Once code style, readability and DRY principles hace been mastered the next step is improving what you're already doing and using the more advanced language features.

This is a really nice and clear (FORTRAN specific) explanation of how to move from a purely procedural approach, explaining progressively more advanced features of FORTRAN functions and subroutines, and finishing with a real-world scientific example program:

[https://livebook.manning.com/book/modern-fortran/chapter-3/62 https://livebook.manning.com/book/modern-fortran/chapter-3/62]

Unfortunately the above link is to a book (Modern Fortran) only part of which is freely viewable. Ideally the book may be available institutionally, but if there is an equivalent freely available link let us know.

 

'''Advanced'''

To see the advice above put into practice, you can [https://youtu.be/8bZh5LMaSmE watch this video]. It is a bit long (18min). It is under the Advanced resources because the code it is using is written in Rails. Not many of you will have experience in Rails but the purpose of the video is to talk about the principles and not the specificity of the code. It also involves some more advanced concepts such as classes and modules.

Documentation plays a critical role for code reusability. Any effort to document code is worthwhile and will improve reusability, but it is likely a large effort in code documentation will only be made in advanced code reuse scenarios, like a published module or library. In that scenario this is an excellent introduction primarily about  taking into account the audiences for different aspects of code documentation:

[https://documentation.divio.com/introduction/ https://documentation.divio.com/introduction/]

 

=== '''Style guides''' ===

Python: 

*[https://www.python.org/dev/peps/pep-0008/ pep8]   Python Enhancement Proposal
*[https://docs.python-guide.org/writing/style/ https://docs.python-guide.org/writing/style/]
*[https://realpython.com/python-pep8/ https://realpython.com/python-pep8/]
*[https://realpython.com/lessons/reserved-keywords/ Python reserved keywords]

Julia:

*[https://docs.julialang.org/en/v1/manual/style-guide/ style guide]
*[https://docs.julialang.org/en/v1/base/base/#Keywords reserved keywords]

R:

*[https://style.tidyverse.org R style guide] from tidyverse
*[https://www.datamentor.io/r-programming/reserved-words/ R reserved keywords]

 

[[Category:Data induction]]

Coding best practices

2022-08-22T00:01:22Z

P.petrelli:

= Introduction =
There are code standards and conventions available depending on the language you are using and sometimes also conventions adopted for specific collaborative projects. These can be quite complex and out of scope if you are writing a code for your analysis, however there are a few things you can do to make your code much more readable and safer from bugs which are quite simple. In the video linked below, kindly provided by DataTAS, the presenter gives some useful tips which can be applied to any language:

"[https://www.facebook.com/1354624204600925/videos/138577474926767 Reproducible research how to write code that is built to last]" 

It is worth watching the video (the actual presentation is about half of the video ~35 minutes) to understand fully how valuable these tips are and also to get a perspective from someone who went from a science background to a commercial software engineering position.

Below is a list of best practices discussed in the video.

=== Naming ===

*Use descriptive names for variables and functions
*Use consistent naming across the code
*Avoid hard-coding values
*Initialising variables

=== Code structure ===

*Indents
*Comments
*Use functions to organise your code 
*Don't Repeat Yourself (DRY) code
*One statement per line
*Write explicit code
*Keep your files a reasonable length
*Clear flow: try to have only one exit point in a function
*Test important parts of your code  

 

= Writing Reusable Code =

There are many definitions of reusable code, and the details often depend on the use to which the code is being put. From [https://en.wikipedia.org/wiki/Code_reuse wikipedia]
<blockquote>
The key idea in reuse is that parts of a computer program written at one time can be or should be used in the construction of other programs written at a later time.
</blockquote>
 

=== Why ===

Why write reusable code? This excellent [https://www.frontiersin.org/articles/10.3389/fninf.2017.00069/full article] articulates what it takes to transform code into a fully fledged scientific contribution, and reusability is a key component:
<blockquote>
Making your program reusable means it can be easily used, and modified, by you and other people, inside and outside your lab
</blockquote>
The process of making your code reusable will also make it better, less error prone, saving time and increasing productivity. 

=== How ===

How do you write reusable code? There are levels of reusability, from re-using your own code to a fully published library/module for others to use. Start with the basics and with experience add more reproducible practices. If you're already doing the basics, try some of the intermediate or advanced ideas.

 

'''Basic'''

The basics of code reusability involve easily readable code and DRY (don't repeat yourself) principles, so using functions/subroutines/procedures to avoid copying blocks of code and modifying each block. The tips in the introduction can all help with code reusability as well as those listed below.

Ten tips for writing readable code (based on PHP but principles are universal):

[https://dzone.com/articles/10-tips-how-to-improve-the-readability-of-your-sof https://dzone.com/articles/10-tips-how-to-improve-the-readability-of-your-sof] 

A good (python specific) section of a course from Software Carpentry about writing functions:

[https://swcarpentry.github.io/python-novice-inflammation/08-func/index.html https://swcarpentry.github.io/python-novice-inflammation/08-func/index.html]

For the beginner python programmer it can be difficult to know exactly '''how''' to go about reusing code, how to organise it and import it into your notebooks or programs. This is a short, clear article about the python specific details on reusing your code:

[https://towardsdatascience.com/creating-reusable-code-for-data-science-projects-740391ec7bad https://towardsdatascience.com/creating-reusable-code-for-data-science-projects-740391ec7bad]

In a similar vein (and also python specific), how and when to use a <tt>main</tt> function in python:

[https://realpython.com/python-main-function/ https://realpython.com/python-main-function/]

 

'''Intermediate'''

Once code style, readability and DRY principles hace been mastered the next step is improving what you're already doing and using the more advanced language features.

This is a really nice and clear (FORTRAN specific) explanation of how to move from a purely procedural approach, explaining progressively more advanced features of FORTRAN functions and subroutines, and finishing with a real-world scientific example program:

[https://livebook.manning.com/book/modern-fortran/chapter-3/62 https://livebook.manning.com/book/modern-fortran/chapter-3/62]

Unfortunately the above link is to a book (Modern Fortran) only part of which is freely viewable. Ideally the book may be available institutionally, but if there is an equivalent freely available link let us know.

 

'''Advanced'''

To see the advice above put into practice, you can [https://youtu.be/8bZh5LMaSmE watch this video]. It is a bit long (18min). It is under the Advanced resources because the code it is using is written in Rails. Not many of you will have experience in Rails but the purpose of the video is to talk about the principles and not the specificity of the code. It also involves some more advanced concepts such as classes and modules.

Documentation plays a critical role for code reusability. Any effort to document code is worthwhile and will improve reusability, but it is likely a large effort in code documentation will only be made in advanced code reuse scenarios, like a published module or library. In that scenario this is an excellent introduction primarily about  taking into account the audiences for different aspects of code documentation:

[https://documentation.divio.com/introduction/ https://documentation.divio.com/introduction/]

 

=== '''Style guides''' ===

*[https://www.python.org/dev/peps/pep-0008/ Python: pep8]   Python Enhancement Proposal
**[https://docs.python-guide.org/writing/style/ https://docs.python-guide.org/writing/style/]
**[https://realpython.com/python-pep8/ https://realpython.com/python-pep8/]
*[https://realpython.com/lessons/reserved-keywords/ Python reserved keywords]
*[https://docs.julialang.org/en/v1/manual/style-guide/ Julia: style guide]
*[https://style.tidyverse.org R style guide] from tidyverse
*[https://www.datamentor.io/r-programming/reserved-words/ R reserved keywords]

 

[[Category:Data induction]]

JRA55

2022-08-08T01:08:28Z

P.petrelli:

The Japan Meteorological Agency (JMA) conducted JRA-55, the second Japanese global atmospheric reanalysis project. It covers 55 years, extending back to 1958, coinciding with the establishment of the global radiosonde observing system. Compared to its predecessor, JRA-25, JRA-55 is based on a new data assimilation and prediction system (DA) that improves many deficiencies found in the first Japanese reanalysis. These improvements have come about by implementing higher spatial resolution (TL319L60), a new radiation scheme, four-dimensional variational data assimilation (4D-Var) with Variational Bias Correction (VarBC) for satellite radiances, and introduction of greenhouse gases with time varying concentrations. The entire JRA-55 production was completed in 2013, and thereafter will be continued on a real time basis. (This information was obtained from the NCAR website [http://rda.ucar.edu/datasets/ds628.0/ http://rda.ucar.edu/datasets/ds628.0/])

The [https://esgf.nccs.nasa.gov/projects/create-ip/ CREATE-IP] project offers now a version of this dataset which comes already as netcdf, from now on we are going to download this version in the ua8 project (see below for details). We won't be updating anymore the RDA version available in rr7.

 

=== '''License''' ===

You need to register by filling the [http://jra.kishou.go.jp/comm/application_en.html JRA-55 project application form] to use this dataset. A copy of their agreement is available here as [[:File:Application_JRA_Data_Access.pdf]].

More information, the JRA55 user manuals for pressure and model levels and quality issues with the data are listed both in the [http://jra.kishou.go.jp/JRA-55/index_en.html#about JRA-55 project website] and the NCAR [https://climatedataguide.ucar.edu/climate-data/jra-55 Climate Data Guide] and [http://rda.ucar.edu/#!lfd?nb=y&b=proj&v=JMA%20Japanese%2055-year%20Reanalysis JRA-55 dataset webpages]

To cite:

Japan Meteorological Agency/Japan (2013), JRA-55: Japanese 55-year Reanalysis, Daily 3-Hourly and 6-Hourly Data, [https://doi.org/10.5065/D6HH6H41 https://doi.org/10.5065/D6HH6H41], Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory, Boulder, Colo. (Updated monthly.) Accessed<font style="font-family: Poppins, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 245); text-decoration-style: initial; text-decoration-color: initial;">† dd mmm yyyy.   <small style="font-size:13px; color:#000000; font-family:Poppins, sans-serif; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-transform:none; white-space:normal; widows:2; word-spacing:0px; -webkit-text-stroke-width:0px; background-color:#fffff5; text-decoration-style:initial; text-decoration-color:initial">†Please fill in the "Accessed" date with the day, month, and year (e.g. - 5 Aug 2011) you last accessed the data.

 

=== '''Known issues''' ===

29-01-2020 - Some tropical cyclones occurring over the Northeast Pacific and the North Atlantic from 1959 to 1987 were erroneously represented as anti-cyclonic vortices in the Japanese 55-year Reanalysis (JRA-55) dataset. A detailed report is available at the JRA-55 website:

#Issue with tropical cyclone analysis in JRA-55

[https://protect-au.mimecast.com/s/ocz8CANZy7iKkkn2UY1gbM?domain=jra.kishou.go.jp https://jra.kishou.go.jp/JRA-55/index_en.html#quality]

=== '''JRA55 at NCI''' ===

''CREATE-IP version:''

This version is available in ua8 at

/g/data/ua8/synda/CREATE-IP/reanalysis/JMA/JRA-55/JRA-55/atmos/

6 hourly data:  6hr/v20200612

variables: tas, uas, vas, ps, psl, zg (pressure levels)

monthly data:  mon/v20200612

variables: clt, evspsbl, hur, hus, pr, prw, ps, psl, rlds, rldscs, rlus, rlut, rlutcs, rsds, rsdscs, rsdt, rsus, rsuscs, rsut, rsutcs, sfcWind, ta, tas, tro3, ts, ua, uas, va, vas, zg  (3D variables are on pressure levels)

==== ''Discontinued rr7 version:'' ====

'''NB this hasn't been updated for a long time so we are not giving access to it anymore!!!'''

Data is located in

*pressure levels: /g/data/rr7/JRA55/6hr/atmos/<varname>/v1/<filenames>
*model levels: /g/data/rr7/JRA55/6hr/atmos/reg-tl319/<varname>/v1/<filenames>

Filenames are: <var>_<mip>_JRA55_<grid>_<YYYYMMDDHH-YYYMMDDHH>.nc</var> <grid> information is currently present only for model level data and it is "reg-tl319" <mip> is 6hrPlev and 6hrLev for pressure and model levels respectively

==== '''List of variables''' ====

Currently available variables on model levels are

*ta - atmospheric temperature
*ua - eastward wind component
*va - northward wind component
*hus - specific humidity
*zg - geopotential
*ps - surface air pressure

Currently available variables on pressure levels are

*tas- surface atmospheric temperature
*uas - surface eastward wind component
*vas - surface northward wind component
*hurs - surface relative humidity
*huss - surface specific humidity
*zg - geopotential
*ps - surface air pressure
*psl - air pressure at sea level
*dptas - Near-Surface Dew Point Depression

==== '''Download & conversion process''' ====

We downloaded the model analysis level from the NCAR website and the pressure level data from the JMA Data Dissemination System

==== '''Feedback''' ====

You can add comment on this collection directly here or on the dedicated [https://opus.nci.org.au/display/CMIP/Observational+and+Reanalysis+Data Observational and Reanalysis Data page] on NCI confluence For questions use the [https://opus.nci.org.au/display/CC/questions/all Climate Community confluence questions] .

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

FAQs

2022-07-28T02:32:44Z

P.petrelli:

This page lists FAQs for the different model supported by the CMS team, as well as FAQs around accessing and publishing data.

= FAQ pages =

*[[UM_and_ACCESS_FAQ|UM and ACCESS FAQ]]
*[[Um-errors|UM error messages]]
*[[WRF_FAQ|WRF FAQ]]
*[[LIS_and_NUWRF_FAQ|LIS and NUWRF FAQ]]
*[[MOM_FAQ|MOM FAQ]]
*[[Data_FAQ|Data FAQ]]
*[[CESM_FAQ|CESM FAQ]]

= FAQ miscellaneous =

== How to debug issues with gadi_jupyter? ==

The script submitted to PBS and the PBS logs are saved under the directory tmp/runjp directory under the user's /scratch directory for the group used to run the script. These will often have more information if the script was started alright but you can't seem to be able to connect to the Jupyer Notebooks. This usually happens when the job started alright but exited due to an error in the script.

For example, let's say I (ccc561) requests gadi_jupyter to run using the project w35. The information will then be stored under '''/scratch/w35/ccc561/tmp/runjp'''

If I then run gadi_jupyter again but this time running on w40, the information will be stored under '''/scratch/w40/ccc561/tmp/runjp'''

== How do I run independent jobs in parallel? ==

If you have to run the same script with a lot of independent sets of inputs, or you have lots of similar scripts to run at NCI. And these scripts run on 1 processor. You could submit each of your jobs independently via PBS but then small jobs are not efficient at NCI. It would probably be better for you to have a master script that dispatches your jobs to available processors. You then submit this master script to PBS and this script requires a lot more processors and hence might be more efficient with the scheduler. There is an example of such a master script [[Embarrassingly_parallel_job|on this page]].

== How to transform 1D lat/lon arrays into 2D lat/lon arrays with Python? ==

If using numpy or masked arrays, one can use the resize() method. Assuming slat and slon are 1D latitude and longitude arrays, do this:
<syntaxhighlight lang="python">
tmp = numpy.resize(slat, (slon.size, slat.size))
slat2D = numpy.transpose(tmp)
slon2D = numpy.resize(slon, (slat.size, slon.size))

</syntaxhighlight>
If you want masked arrays, replace "numpy." by "numpy.ma."  
== How do I push my local code changes on gadi back to github using ssh keys? ==

You have an account on [http://github.com | github]. You have cloned a repository to raijin and made changes to it. Now you want to push those changes back to the github repository but you get errors like this (where <username> is your github username):
<syntaxhighlight lang="text">
error: The requested URL returned error: 403 Forbidden while accessing https://github.com/<username>/test.git/info/refs

fatal: HTTP request failed
</syntaxhighlight>
Firstly, you need to [https://help.github.com/articles/generating-ssh-keys/ | make an ssh key and add the public part of the key to your account on github]. Check the url to which you are trying to push your changes:   <syntaxhighlight lang="text">
git remote -v
origin https://github.com/<username>/test.git (fetch)
origin https://github.com/<username>/test.git (push)
</syntaxhighlight>

If it looks like the output above you need to alter the url and put in a login name in front of the address. For ssh access the login name is always git. The command to set the remote url is:
<syntaxhighlight lang="text">
git remote set-url origin ssh://git@github.com/<username>/test.git

</syntaxhighlight>
but replace <username> with your github username.
== Why do I get segfaults? ==

A segmentation fault means the program you are using has tried to read or write a memory location it does not have access to. This can mean there is an error in the code, or there is a system limit set. If the program runs for others, but not you, it may well be a system limit issue. This seems to be quite common when trying to write netcdf files. In some cases this can be overcome by setting your stack size to unlimited

For tcsh/csh add this to your .cshrc:
<syntaxhighlight lang="text">
limit stacksize unlimited
</syntaxhighlight>

For bash add this to your .bashrc:
<syntaxhighlight lang="text">
ulimit -s unlimited
</syntaxhighlight>

Note: you must log out and log back in again for this change to take effect.

 

== No files found or non-existent path error while running a PBS job ==

When submitting a job on gadi you need to declare explicitly what gdata and/or scratch projects your script need to access.

This is done by setting the storage flag in the job you are submitting:

PBS -l storage=gdata/hh5+gdata/ua8+gdata/e14

In the above case I want to use input data from gdata/ua8 save my results to gdata/e14 and use the conda modules which are in gdata/hh5

== I cannot see a project folder under /g/data and/or /scratch ==

You need to be a member of a project to access or even just list the project folder. So if you cannot see a project folder, it doesn't mean that it doesn't exist or it was removed, it simply means that you are not a member of that project. Go to my.nci.org.au if you want to request access.

== Weird errors for a script that was previously working  ==

Make sure you have loaded the right modules and only them! Avoid to load modules directly in your .bash_profile, .bashrc files . They will be loaded every time and occasionally interfere with the modules you are loading for a specific task. 

It is useful and safe to add instructions as

module use /g/data/hh5/public/modules

as this is making the modules available without loading any.

Sometimes you need to install a python module in your user space ( $HOME/.local/... ) as it is not available in a managed conda environment. Again, occasionally modules stored in your local user space can interfere with other loaded modules, for example if they have different dependencies. If you find this is causing issues you can exclude the local environment by setting this environment variable

PYTHONNOUSERSITE=x (  =True and =1 should also work as values)

 

== Running out of memory when running a relatively small timeseries analysis using xarray/dask ==

Xarray loads data lazily, which means that it will only load the data values when you need them. However, the data is stored in the files in chunks and all values in a chunk will be loaded as a block. The most common pattern for chunks in netcdf files is to have a grid stored in the same chunk. So, if you run a calculation for a specific timestep, you will only load one chunk. This means that often timesteps are on separate chunks, often `time` has a chunk size of 1, where all timesteps for a particular grid point are stored on different chunks. If you don’t change the file chunking and try to run a calculation that needs all the timeseries for a single grid point, you will be effectively loading the entire file in memory.

If you want to rechunk your data to have all timesteps on the same chunks you can run:
<syntaxhighlight lang="text">
array = array.chunk(chunks={‘time’=-1})
</syntaxhighlight>

-1 means that the ‘time’ dimension size will be used as chunk size. If you have 100 timesteps, it would be equivalent to set the new chunks as {‘time’=100}

 

== Issues installing a R module on gadi ==

First of all, you might not need to install the module, as NCI is now managing some R environments which might include already what you need: [https://opus.nci.org.au/display/DAE/Getting+Started [https://opus.nci.org.au/display/DAE/Getting+Started]].

If this approach doesn't work for you and you need to install a module, make sure you have all the necessary modules loaded. As outlined in the [https://opus.nci.org.au/display/Help/R Gadi user guide] you need to load the same intel module that was used to install the R module you are using. If you still have issues, contacting the NCI Helpdesk, rather than us, is more efficient as they have more experience with R.

== I want to use Matlab on gadi ==

You will need to setup your institutional Matlab license. We cannot help you troubleshooting any of this, just follow the [https://opus.nci.org.au/display/Help/MATLAB NCI instructions] and contact their Helpdesk if you run into any issue.

[[Category:FAQ]]

American Geophycal Union

2022-07-27T22:05:20Z

P.petrelli:

'''Journals'''

*Geophysical Research Letters
*Journal of Geophysical Research: Atmospheres
*Journal of Geophysical Research: Oceans
*Water Resources Research

'''Open data policy'''

 [https://www.agu.org/Publish-with-AGU/Publish/Author-Resources/Policies/Data-policy AGU Data Policy]

'''Guidelines'''

[https://www.agu.org/Publish-with-AGU/Publish/Author-Resources/Data-and-Software-for-Authors Data guidelines from author resources]

[http://www.copdess.org/enabling-fair-data-project/author-guidelines/ COPDESS author guidelines]

Some of the AGU journals also have their specific data guidance:

*[https://www.agu.org/-/media/Files/Publications/JGR-Atmospheres-Data-Sharing-Guidance.pdf JGR Atmospheres Data Sharing Guidance]
*[https://www.agu.org/-/media/Files/Publications/WRR-Data-Sharing-Guidance.pdf Water Resources Research Data Sharing Guidance]

'''Requirements'''

*No supplement is required, can be produced if data is new and not described in paper
*DOI is not explicitly required
*Software is required

Center publishing procedure is fully compliant with the requirements

[[Category:Publisher Policies]]

Wiley Blackwell

2022-07-27T22:03:22Z

P.petrelli:

'''Journals'''

*Global Change Biology

 

'''Open data policy'''

[https://authorservices.wiley.com/author-resources/Journal-Authors/open-access/data-sharing-citation/data-sharing-policy.html Open Access data sharing policy]

This details 4 different levels of data sharing:

#Encourages Data Sharing
#Expects Data Sharing
#Mandates Data Sharing
#Mandates Data Sharing and Peer Reviews Data

Information on which level is adopted is provided for each journal in the "Author guidelines: Editorial policies and Ethical Consideration" section.

The following is an example for Global Change Biology journal, which Mandates data sharing:

[https://authorservices.wiley.com/author-resources/Journal-Authors/open-access/data-sharing-citation/data-sharing-policy.html Global Change Biology data sharing policy]

'''Guidelines'''

[https://www.nature.com/documents/nr-data-availability-statements-data-citations.pdf Data policy guidance for authors]

[https://www.nature.com/sdata/policies/repositories#envgeo Recommended repositories]

 

'''Requirements'''

*DOI is not explicitly required
*Software is explicitly required in some of the policies
*Data Availiability Statement is required in some of the policies

Centre publishing procedure is fully compliant with the requirements

[[Category:Publisher Policies]]

Royal Meteorological Society

2022-07-27T22:01:40Z

P.petrelli:

'''Journals'''

*International Journal of Climatology
*Quarterly Journal of the Royal Meteorological Society

'''Open data policy'''

[https://rmets.onlinelibrary.wiley.com/pb-assets/hub-assets/rmets/Policy_on_availability_of_data-1508421522713.pdf Rmets Data Policy]

 

'''Guidelines'''

NA

 

'''Requirements'''

*No supplement is required
*DOI is not explicitly required
*model code or software is seen as metadata rather than data
*Data should be deposited in repositories that are permanent or have plans for succession

Center publishing procedure is compliant with all requirements but the permanent repository

[[Category:Publisher Policies]]

Data Management Plan

2022-07-21T04:12:47Z

P.petrelli:

A Data Management Plan (DMP) is a document describing what will be used and generated by a project and how this will be managed and shared. The [https://ardc.edu.au/resources/aboutdata/data-management-plans/ definition] and actual form of a data management plan can vary depending on the use the plan is created for. What you might include also depends on the project field.

A DMP can be required:

*to apply for a grant
*by your institution for PhD and/or major projects
*when you want to publish your data in a repository

and it is also useful:

*to keep track of data provenance
*to manage storage and computing resources at CLEX level
*whenever you share data, even if you do not intend to publish yet

=== '''Creating a Data Management Plan''' ===

CLEX encourage you to create a DMP as early as you can in your project. We provide [[CLEX_Roadmap|CLEX_Roadmap]] a DMP webtool which you can use to create, store and share DMPs, as well as for guidance on how to structure one. We distinguish between three data management levels: personal, group and public. 

==== '''Personal''' ====

 This applies to any project exploring new ideas and/or procedures. A single researcher mostly conducts the work and hence there are no specific sharing requirements.

At this level a formal data management plan is not necessary, however one can still be useful to start collecting information in case the project becomes bigger and to help plan how the research will be conducted.

At this stage a plan can help design the experiment:

*which data will be used, if it is locally available (see [[Finding_datasets|Finding datasets page]]) and if can be used (see [[Open_access_license|licenses]])
*basic workflow description
*bulk estimate of data volume and compute resources
*software and support (including training) necessary to run experiment and/or post-process data

It is also never too early to start to work on your project [[Provenance|provenance]]:

*applying where possible relevant [[Conventions|metadata conventions]] to any data produced, it is easier when done from the first analysis steps rather than having to update the metadata subsequently.
*adopting a version control for your code
*building references to data/code sources

==== '''Group''' ====

 This applies to any project where data will be shared. Hence this applies to any group activity but also to single researcher projects whose output could be of interest to other researchers (this is often true of model experiments).

When you share data with others it is important for the data to be accompanied by its provenance. This is so others can verify the validity of the data and also use the data in the correct way.

A DMP should also include:

*list of researchers responsible for the data and all the interested parties
*all information previously entered in the DMP should be reviewed to make sure it is still applicable
*directory structure and filenames should be consistent and follows conventions wherever possible, CF standard should be adopted if not already in use.
*data reduction plan: does all model data need to be saved after analysis?
*data retention plan: when can data be deleted?
*[[Backup_strategy|backup strategy]]

If research is part of a collaboration project:

*a [[Data_Collaboration_Agreement|data agreement]] on how data will be shared/published should be in place from the start
*any data policy and/or standard adopted by the project should be followed

'''Public'''

 This applies to the final output of the project when data is ready for publication. In addition to the requirements of the group level:

*a descriptive title and acronym
*a version, including a versioning strategy for future updates
*a plain English abstract
*links to relevant documentation and code
*license
*keywords
*long-term storage strategy
*any other specific requirements that the chosen repository might have, must be addressed.

=== '''Tips''' ===

As a DMP is documenting and describing your data output as a paper documents and describes your research output, some tips about writing research papers can be applied to DMP, too. We collected a few we found online below.
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">...Pitfalls include using complicated jargon, including unnecessary details, and writing for your highly specialized colleagues instead of a wider audience. ...</div> <div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">Only abbreviations firmly established in the field are eligible, … avoiding those which are not broadly used ...</div> <div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">… when looking for keywords, avoid words with a broad meaning and words already included in the title.</div> <div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">… you need to include detailed information so a knowledgeable reader can reproduce the experiment. However, do not repeat the details of established methods; use References and Supporting Materials to indicate the previously published procedures.</div> <div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">… indicate uses and extensions if appropriate. Moreover, you can suggest future experiments and point out those that are underway ...</div> <div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">Effective research articles are interesting and useful to a broad audience, including scientists in other fields.</div> <div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">use standard systems … conventions</div>
 

[[Category:Data induction]]

MERRA2

2022-06-24T03:20:41Z

P.petrelli:

The Modern Era Retrospective-Analysis for Research and Applications (MERRA) is a NASA reanalysis integrating satellite observations using a major new version of the Goddard Earth Observing System Data Assimilation System Version 5 (GEOS-5).

MERRA focuses on historical analyses of the hydrological cycle on a broad range of weather and climate time scales covering the modern satellite era from 1979 to present. The state-of-the-art GEOS-5 data assimilation system includes many modern observing systems (such as EOS) in a climate framework Specifically, the GEOS-DAS Version 5 implements Incremental Analysis Updates (IAU) to slowly adjust the model states toward the observed state.

MERRA-2 is produced with the GMAO/GEOS-5 Data Assimilation System Version 5.12.4 and its native resolution is 0.5° x 0.625° x 72 hybrid sigma/pressure levels. Hourly data intervals are used for two-dimensional products, while 3-hourly intervals are used for three-dimensional products. These may be on the model’s native 72-layer vertical grid or at 42 pressure surfaces extending to 0.1 hPa. MERRA-2 spans from 1979 to present (with about 5 months lag).

The 42 pressure levels are: 1000, 975, 950, 925, 900, 875, 850, 825, 800, 775, 750, 725, 700, 650, 600, 550, 500, 450, 400, 350, 300, 250, 200, 150,100, 70, 50, 40, 30, 20,10, 7, 5, 4, 3, 2, 1,0.7, 0.5, 0.4, 0.3, 0.1 hPa The 72 native grid levels are:

Information retrieved from [https://climatedataguide.ucar.edu/climate-data/nasa-merra NCAR climate re-analysis guide] and [http://wiki.seas.harvard.edu/geos-chem/index.php/MERRA-2 GEOS-CHEM MERRA-2 wiki]. The data was downloaded from the [https://disc.sci.gsfc.nasa.gov/information/glossary/581b59c65fa71421a68e64fa/merra-2?tags=mission GES DISC data store] where information on all the MERRA-2 products is available.

 

== '''License and Acknowledgements''' ==

Access to GES DISC data requires all users to be registered with the Earthdata Login system, even if we downloaded the data for you you should register. It is important for the data providers to have an estimate of users to justify the work that goes with providing the data services. Complete terms of use and information on how to sign up are available on the [https://wiki.earthdata.nasa.gov/display/EL/Earthdata+Login+Overview+and+Policy,+v1.2 GEOS-DISC data policy webpage] . Example of citation including the DOI, each product has its own (see below). Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 inst6_3d_ana_Np: 3d,6-Hourly,Instantaneous,Pressure-Level,Analysis,Analyzed Meteorological Fields V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/A7S6XP56VZWS

 

== '''MERRA2 at NCI''' ==

We are hosting a copy of some of the MERRA-2 products on gadi in the rr7 project. The data is available in /g/data/rr7/MERRA2/raw/<MERRA-product>/<year>/<files> Filenames are <runid>.<collection>.<timestamp>.nc4

Since the rr7 project is going to be removed and the datasets relocated elsehwere, from 2020 we started downloading newly requested products to the ua8 project:

/g/data/ua8/MERRA2/<frequency>/<product>/

=== '''Hosted in rr7''' ===

==== '''tavg1_2d_rad_Nx''' ====

*MERRA 2D,1-hourly,time-averaged,single-level,assimilation,radiation diagnostics
*rr7/MERRA2/raw/M2T1NXRAD.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_rad_Nx.yyyymmdd.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly
*doi: 10.5067/Q9QMY5PBNV1T
*file size: 209 Mb
*citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_rad_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Radiation Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/Q9QMY5PBNV1T

 

==== '''tavg1_2d_flx_Nx''' ====

*MERRA2 2D 1-hourly,time-averaged,single-level,assimilation,surface flux diagnostics
*rr7/MERRA2/raw/M2T1NXFLX.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_flx_Nx.yyyymmdd.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly
*doi: 10.5067/7MCPBJ41Y0K6
*file size: 380 Mb
*citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_flx_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Surface Flux Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/7MCPBJ41Y0K6

 

==== '''tavg1_2d_int_Nx''' ====

*MERRA 2D ,1-hourly,time-averaged,single-level,assimilation,vertically integrated diagnostics
*rr7/MERRA2/raw//M2T1NXINT.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_int_Nx.yyyymmdd.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly
*doi: 10.5067/Q5GVUVUIVGO7
*file size: 1.3 Gb
*citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_int_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Vertically Integrated Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/Q5GVUVUIVGO7

 

==== '''inst6d_3d_ana_Np''' ====

*MERRA 3D,6-hourly,instantaneous,pressure-level,analysis,analyzed meteorological fields
*rr7/MERRA2/raw/M2I1NPANA.5.12.4/yyyy/mm/MERRA2_100.inst6_3d_ana_Np.yyyymmdd.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 6 hourly Levels: 42
*doi: 10.5067/A7S6XP56VZWS
*file size: 511 Mb
*citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 inst6_3d_ana_Np: 3d,6-Hourly,Instantaneous,Pressure-Level,Analysis,Analyzed Meteorological Fields V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/A7S6XP56VZWS



==== '''tavg1_2d_slv_Nx''' ====

*MERRA 2D,1-hourly,Time-Averaged,Single-Level,Assimilation,Single-Level Diagnostics
*rr7/MERRA2/raw//M2T1NXSLV.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_slv_Nx.yyyymmdd.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly
*doi: 10.5067/VJAFPLI1CSIV
*file size: 400 Mb
*citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/VJAFPLI1CSIV

 

==== '''inst3d_3d_asm_Np''' ====

*MERRA 3D,3-hourly,Instantaneous,Pressure-Level,Assimilation,Assimilated Meteorological Fields
*/rr7/raw/MERRA2/M2I3NPASM.5.12.4/yyyy/mm/MERRA2_100.inst3_3d_asm_Np.yyyymmdd.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 3 hourly Levels: 42
*doi: 10.5067/QBZ6MG944HW0
*file size: 1.2 Gb
*citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 inst3_3d_asm_Np: 3d,3-Hourly,Instantaneous,Pressure-Level,Assimilation,Assimilated Meteorological Fields V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/QBZ6MG944HW0

=== '''Hosted in ua8''' ===

==== '''inst1_2d_asm_Nx''' ====

*MERRA-2 inst1_2d_asm_Nx: 2d,1-Hourly,Instantaneous,Single-Level,Assimilation,Single-Level Diagnostics V5.12.4
*ua8/MERRA2/1hr/M2I1NXASM.5.12.4/<yyyy>/<mm>/MERRA2_100.inst1_2d_asm_Nx.<yyyymmdd>.nc4
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly 
*doi: 10.5067/3Z173KIE2TPD
*file size: 198 MB
*citation: Global Modeling and Assimilation Office (GMAO) (2015), MERRA-2 inst1_2d_asm_Nx: 2d,1-Hourly,Instantaneous,Single-Level,Assimilation,Single-Level Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed: '''[''Data Access Date'']''', [https://doi.org/10.5067/3Z173KIE2TPD 10.5067/3Z173KIE2TPD] 



==== '''inst3_2d_gas_Nx''' ====

*MERRA-2 inst3_2d_gas_Nx: 2d,3-Hourly,Instantaneous,Single-Level,Assimilation,Aerosol Optical Depth Analysis V5.12.4
*ua8/MERRA2/3hr/M2I3NXGAS.5.12.4/<yyyy>/<mm>/MERRA2_100.inst3_2d_gas_Nx.<yyyymmdd>.nc4
*doi: 10.5067/HNGA0EWW0R09
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: 3 hourly
*file size: 9.2 MB
*citation: Global Modeling and Assimilation Office (GMAO) (2015), MERRA-2 inst3_2d_gas_Nx: 2d,3-Hourly,Instantaneous,Single-Level,Assimilation,Aerosol Optical Depth Analysis V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed: '''[''Data Access Date'']''', [https://doi.org/10.5067/HNGA0EWW0R09 10.5067/HNGA0EWW0R09]



==== '''instM_2d_gas_Nx''' ====

*MERRA-2 instM_2d_gas_Nx: 2d,Monthly mean,Instantaneous,Single-Level,Assimilation,Aerosol Optical Depth Analysis V5.12.4
*ua8/MERRA2/monthly/M2IMNXGAS.5.12.4/<yyyy>/MERRA2_100.instM_2d_gas_Nx.<yyyymm>.nc4
*doi: 10.5067/XOGNBQEPLUC5
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: monthly
*file size: 9.2 MB
*citation: Global Modeling and Assimilation Office (GMAO) (2015), MERRA-2 instM_2d_gas_Nx: 2d,Monthly mean,Instantaneous,Single-Level,Assimilation,Aerosol Optical Depth Analysis V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed: '''[''Data Access Date'']''', [https://doi.org/10.5067/XOGNBQEPLUC5 10.5067/XOGNBQEPLUC5]

 

==== '''tavgM_2d_aer_Nx''' ====

*''''''MERRA-2 tavgM_2d_aer_Nx: 2d,Monthly mean,Time-averaged,Single-Level,Assimilation,Aerosol Diagnostics V5.12.4
*ua8/MERRA2/monthly/M2TMNXAER.5.12.4/<yyyy>/MERRA2_100.tavgM_2d_aer_Nx.<yyyymm>.nc4
*doi: 10.5067/FH9A0MLJPC7N
*resolution: lat/lon - 0.5 /0.625 degrees Timestep: monthly
*file size: 481 MB
*citation: Global Modeling and Assimilation Office (GMAO) (2015), MERRA-2 tavgM_2d_aer_Nx: 2d,Monthly mean,Time-averaged,Single-Level,Assimilation,Aerosol Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed: '''[''Data Access Date'']''', [https://doi.org/10.5067/FH9A0MLJPC7N 10.5067/FH9A0MLJPC7N]

 

[https://docs.google.com/spreadsheets/d/1ZF3A-miws1yifBzH3d4XfXXg0YNEDpFLej0yd6uLW3E/edit?usp=sharing '''List of variables available for each product''']

 

{{#iDisplay:https://docs.google.com/spreadsheets/d/1rkCZegrJoY5fJV50uxXlb3AsywcvzJKEk56ZVigpQ_U/pubhtml?widget=true&headers=false|1000|500}}

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

FROGs

2022-06-22T01:29:10Z

P.petrelli:

'''Frequent Rainfall Observations on GridS (FROGS) database'''

FROGs is a collection of daily precipitation products on a common 1×1 deg grid. The collection includes satellite, ground-based and reanalysis products. Most of the satellite products were calibrated using rain gauges data, for these, where available, the non adjusted version is also provided. The temporal range included starts from 1901 to 2019 depedning on the data product. Where it is possible the database will be extended in future updates of FROGs. The data products spatial coverage is also dependent on the original sources.

For more information please refer to the [https://frogs.prod.lamp.cnrs.fr FROGs website] and in particular to their [https://frogs.prod.lamp.cnrs.fr/news/ news page] for updates.

=== '''FROGs at NCI''' ===

The data is available in the ua8 project as part of the Precipitation dataset collection

/g/data/ua8/Precipitation/FROGs/1DD_V1/<dataset>/<files>

=== '''Dataset list''' ===

Below, the datasets are listed depending on the source data type and spatial coverage

==== '''Satellite based quasi global''' ====

*3B42 v7.0
*3B42 v7.0 IR
*3B42 v7.0 MW
*3B42 RT v7.0
*3B42 RT v7.0 uncalibrated
*CMORPH V1.0, RAW
*CMORPH V1.0, CRT
*GPCP 1DD CDR v1.3, not-enforced
*GPCP 1DD CDR v1.3, enforced
*GPCP IP
*GSMAP-RNL-gauges v6.0
*GSMAP-RNL-no gauges v6.0
*GSMAP-NRT-gauges v6.0
*GSMAP-NRT-no gauges v6.0
*IMERG v6 early uncalibrated
*IMERG v6 late uncalibrated
*IMERG v6 final uncalibrated
*IMERG v6 final calibrated
*PERSIANN CCS CDR 
*PERSIANN CDR v1 r1

==== '''Land only''' ====

*CHIRP v1
*CHIRP v2
*SM2RAIN-CCI

==== '''Ocean only''' ====

*HOAPS v4

==== '''Satellite based regional''' ====

*ARC v2
*COSH
*TAMSAT v2
*TAMSAT v3
*TAPEER v1.5

==== '''Rain gauges based''' ====

*CPC
*GPCC First Guess v1
*GPCC Full Daily v1
*GPCC Full Daily v2018
*GPCC Full Daily v2020
*REGEN Long Term Stations v1-2019
*REGEN All Stations v1-2019

==== '''Atmospheric reanalysis''' ====

*CFSR
*ERA5
*ERA Interim
*GSWP3 Rain & Snow
*JRA55
*MERRA1
*MERRA2

=== '''Citation''' ===

Roca, R., Alexander, L. V., Potter, G., Bador, M., Jucá, R., Contractor, S., Bosilovich, M. G., and Cloché, S.: FROGs: a daily 1∘×1∘gridded precipitation database of rain gauge, satellite and reanalysis products, [https://doi.org/10.14768/06337394-73A9-407C-9997-0E380DAC5598 https://doi.org/10.14768/06337394-73A9-407C-9997-0E380DAC5598], 2019, accesed on <date>

As this is an ongoing collection more data is added without chnages to the doi, so please specify date of access in your citation.

Please note that this is a collection of datasets and the original sources should be cited too wherever possible.

=== '''References''' ===

Roca, R., Alexander, L. V., Potter, G., Bador, M., Jucá, R., Contractor, S., Bosilovich, M. G., and Cloché, S.: FROGS: a daily 1° × 1° gridded precipitation database of rain gauge, satellite and reanalysis products, Earth Syst. Sci. Data, 11, 1017-1035, [https://doi.org/10.5194/essd-11-1017-2019 https://doi.org/10.5194/essd-11-1017-2019], 2019

 

[[Category:Clex-managed-data]] [[Category:Dataset]]

GLEAM

2022-06-20T23:05:04Z

P.petrelli:

GLEAM (Global Land Evaporation Amsterdam Model) is a set of algorithms that separately estimate the different components of land evaporation (or 'evapotranspiration’): transpiration, bare-soil evaporation, interception loss, open-water evaporation and sublimation. Additionally, GLEAM provides surface and root-zone soil moisture, potential evaporation and evaporative stress conditions.
The latest available version is v3.6, this includes two datasets v3.6a and v3.6b
There are 10 variables included:

*Actual Evaporation (E) 
*Soil Evaporation (Eb)  
*Interception Loss (Ei)
*Potential Evaporation (Ep) 
*Snow Sublimation (Es)
*Transpiration (Et)
*Open-water Evaporation (Ew)
*Evaporative Stress (S)
*Root-zone Soil Moisture (SMroot)
*Surface Soil Moisture (SMsurf)

These two datasets differ only in their forcing and temporal coverage:

*'''GLEAM v3.6a''': a global dataset spanning the 42-year period from 1980 (January 1st) to 2021 (December 31st). The dataset is based on satellite and reanalysis data (MSWX net radiation and air temperature).
*'''GLEAM v3.6b''': a global dataset spanning the 19-year period from 2003 (January 1st) to 2021 (December 31st). The dataset is based on satellite data.

'''Key differences''' between v3.6 and the previous (v3.5) release:

*Both v3.6a and v3.6b use the latest version of MSWEP precipitation (v2.8), ESA-CCI soil moisture (v6.2), and VODCA VOD.
*The v3.6a dataset now uses the new MSWX dataset for radiation and temperature.
*The dimension order in both v3.6a and v3.6b files has been changed from [time, longitude, latitude] to [time, latitude, longitude] following the requests received by most users.

For more information refer to the [https://www.gleam.eu GLEAM website], the readme file and the [https://www.gleam.eu/#faq FAQ].
Initially we replicated GLEAM V3 as part of the CABLE benchmarking collection (project wd9). The v3.5 version is hosted in the ua8 project, not in wd9 as previous versions.
== '''Previous versions''' ==
The development of V3 started in 2017, since then 3 subversions were made available:  v3.1, v3.2, v3.3 and v3.5, each one composed of two datasets, usually referred to as "a" and "b". The key differences between v3.5 and the previous (v3.3) release are:
*The v3.5 datasets are produced using a new Python version of GLEAM.
*Both v3.5a and v3.5b use the latest version of MSWEP precipitation (v2.8), ESA-CCI soil moisture (v5.3), and VODCA VOD.
*The v3.5b dataset uses the latest version of CERES radiation (v4.1) and AIRS temperature (v7.0).
*The vegetation fractional cover in v3.5b comes from MOD44B instead of the MEaSUREs dataset (still used in v3.5a).
*The variable potential evaporation is now released including interception loss.
*v3.5a and v35.b cover respectively  between 1980-2020 and 2003-2020

The GLEAM v3 includes:

#A new data assimilation scheme that has been validated for Australia (Martens et al., 2016) and that has been optimised to work at the global scale.
#An updated water balance module that describes the infiltration rates as a function of the vertical gradient in soil moisture.
#Updated evaporative stress functions that combine the vegetation optical depth and the root-zone soil moisture estimates.

This version is described in detail by Martens et al. (2017, GMD), listed in the acknowledgements section.
The most recent is v3.3, the key differences between the new datasets and v3.2 are:
#The v3.3a dataset is now produced using surface radiation and near-surface air temperature from the latest reanalysis of ECMWF, ERA5 (as opposed to ERA-Interim in v3.2a).
#Both GLEAM datasets are now produced using dynamic land cover information based on the MEaSUREs Vegetation Continuous Fields dataset (as opposed to a static map based on MODD44B v52 in the v3.2 datasets).
#All forcing datasets have been updated to their last versions and extended until the end of 2018. Due to the latency in CERES radiation data, GLEAM v3.3b only runs until September 2018.
#4. Next to the daily-resolution data, both monthly and yearly datasets are available as well.
 
== '''GLEAM at NCI''' ==

The latest v3.5 version is available in ua8

/g/data/ua8/GLEAM_v3-5/v3-5{a/b}/<frequency>/<year>/<files>

where frequency is daily/monthly/yearly.

The GLEAM_v3-5 folder also contain the original documentation as pdf.

The older versions are part of the CABLE benchmarking collection and are part of the wd9 project, for each version # files are in:

/g/data/wd9/BenchMarking/GLEAM_v3-#/v3-#{a/b}

where the specific datasets are called "a" or "b" and in case of v1 also "c" .

To access this data currently you need to be part of the cable group, access for the cable group can be requested as for other group on [https://my.nci.org.au https://my.nci.org.au], you might be asked though also to agreed to the cable group terms and conditions.

Access for the wd9 collection will change on the next few months following NCI re-organization of dataset projects.

== '''License''' ==

'''Acknowledgements'''

Whenever GLEAM datasets are used in a scientific publication, the following references should be cited:

#Martens, B., Miralles, D.G., Lievens, H., van der Schalie, R., de Jeu, R.A.M., Fernández-Prieto, D., Beck, H.E., Dorigo, W.A., and Verhoest, N.E.C.: [http://www.geosci-model-dev.net/10/1903/2017/ GLEAM v3: satellite-based land evaporation and root-zone soil moisture], '''''Geoscientific Model Development''''', '''10''', 1903–1925, doi: 10.5194/gmd-10-1903-2017, 2017.
#Miralles, D.G., Holmes, T.R.H., de Jeu, R.A.M., Gash, J.H., Meesters, A.G.C.A., Dolman, A.J.: [http://www.hydrol-earth-syst-sci.net/15/453/2011/hess-15-453-2011.html Global land-surface evaporation estimated from satellite-based observations], '''''Hydrology and Earth System Sciences''''', '''15''', 453–469, doi: 10.5194/hess-15-453-2011, 2011.

The current v3.5 datasets have been produced by [https://www.ugent.be/bw/environment/en/research/h-cel/staff/akash.htm Dr. Akash Koppa] and [https://www.ugent.be/bw/environment/en/research/h-cel/staff/dominik.htm Dr. Dominik Rains].

'''Scientific use'''

GLEAM datasets will not be used for commercial purposes.

'''Feedback'''

Any feedback about the datasets and/or website are highly appreciatedand can be sent to [mailto:info@gleam.eu info@gleam.eu].

 

 

 [[Category:Dataset]][[Category:Clex-managed-data]]

GPCP

2022-06-20T05:01:17Z

P.petrelli:

The Global Precipitation Climatology Project, produced by the NASA Mesoscale Atmospheric Processes Laboratory, comprises threee products: One-Degree Daily (1DD), monthly product and Pentad data. We downloaded the following products:

- daily v1.2  this is limited to 2015 and currently not distributed anymore;

- daily v1.3 this is available up to a three months from present date and it is the recommended daily version;

- monthly v2.3 which is the recommended monthly version, goes up to present date, contains some preliminary files.

The GPCP One-Degree Daily Precipitation Data Set (1DD Data Set) provides daily, global 1x1-deg gridded fields of precipitation totals for October 1996 through the delayed present. The 1DD draws upon several different data sources, surface and satellite measurements, covering different areas of the globe. This product is updated sporadically, about once a month. We downloaded v1.3 daily and v2-3 monthly from the [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/ NCEI server], and v1.2 from the [https://rda.ucar.edu/datasets/ds728.3/#!description NCAR Research Data Archive] . More information is available from the [https://precip.gsfc.nasa.gov/ GPCP webpage] .

And detailed documentation is available from the NCEI data server for [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/doc/ v1-3 daily] and [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-monthly/doc/ v2.3 monthly]. This documents are also available on the filesystem in the dataset folder.

=== '''License''' ===

Access is free but users are requested to add the official citation as below and email the citation to george.j.huffman@nasa.gov or david.t.bolvin@nasa.gov.

=== '''Citation''' ===

v1.2 Huffman, G. J., D. T. Bolvin, and R. F. Adler. 2016. GPCP Version 1.2 One-Degree Daily Precipitation Data Set. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. [https://doi.org/10.5065/D6D50K46 https://doi.org/10.5065/D6D50K46]. Accessed† 30 Jan 2018.

v1.3 daily - Adler, Robert; Wang, Jian-Jian; Sapiano, Mathew; Huffman, George; Bolvin, David; Nelkin, Eric; and NOAA CDR Program (2017). Global Precipitation Climatology Project (GPCP) Climate Data Record (CDR), Version 1.3 (Daily) [Indicate subset used.]. NOAA National Centers for Environmental Information. doi:10.7289/V5RX998Z [access date]

v2.3 monthly - Adler, Robert; Wang, Jian-Jian; Sapiano, Matthew; Huffman, George; Chiu, Long; Xie, Ping Ping; Ferraro, Ralph; Schneider, Udo; Becker, Andreas; Bolvin, David; Nelkin, Eric; Gu, Guojun; and NOAA CDR Program (2016). Global Precipitation Climatology Project (GPCP) Climate Data Record (CDR), Version 2.3 (Monthly). National Centers for Environmental Information. doi:10.7289/V56971M6 [access date]

=== '''References''' ===

Publications using this dataset should also cite the following journal article:

Huffman, G. J., R. F. Adler, M. M. Morrissey, D. T. Bolvin, S. Curtis, R. Joyce, B. McGavock, and J. Susskind, 2001: [https://doi.org/10.1175/1525-7541(2001)002 <0036:GPAODD>2.0.CO;2 Global precipitation at one-degree daily resolution from multisatellite observations]. J. Hydrometeor., 2, 36-50 (DOI: 10.1175/1525-7541(2001)002<0036:GPAODD>2.0.CO;2).

 

=== '''GPCP on raijin''' ===

Data is available in the ua8 project

*/g/data/ua8/Precipitation/GPCP/<frequency>/<version>/<YYYY>/<files>

       where frequency is '''mon''' or '''day''' and versions are '''v1-2''' and '''v1-3 '''for daily data and '''v2-3''' for monthly

Files are in netcdf4 format and filenames are respectively

*gpcp_1dd_v1.2_p1d.YYYYMMDD.nc   for v1.2 daily

where the date is the represented day, for example:  gpcp_1dd_v1.2_p1d.199610.nc

*gpcp_v01r03_daily_dYYYYMMDD_cYYYYMMDD for v1.3 daily

where the first date is the represented day and the second is the creation date for the file, for example: gpcp_v01r03_daily_d20170323_c20171211.nc

* 

gpcp_v02r03_monthly_dYYYYMMDD_cYYYYMMDD.nc

same conventions as for daily v1.3

If preliminary files are present (currently applies only to v2.3 monthly) then this is indicated in the filenames as in this example:

*gpcp_v02r03-preliminary_monthly_d201908_c20190910.nc

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

GPCP

2022-06-20T04:58:30Z

P.petrelli:

The Global Precipitation Climatology Project, produced by the NASA Mesoscale Atmospheric Processes Laboratory, comprises threee products: One-Degree Daily (1DD), monthly product and Pentad data. We downloaded the following products:

- daily v1.2  this is limited to 2015 and currently not distributed anymore;

- daily v1.3 this is available up to a three months from present date and it is the recommended daily version;

- monthly v2.3 which is the recommended monthly version, goes up to present date, contains some preliminary files.

The GPCP One-Degree Daily Precipitation Data Set (1DD Data Set) provides daily, global 1x1-deg gridded fields of precipitation totals for October 1996 through the delayed present. The 1DD draws upon several different data sources, surface and satellite measurements, covering different areas of the globe. This product is updated sporadically, about once a month. We downloaded v1.3 daily and v2-3 monthly from the [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/ NCEI server], and v1.2 from the [https://rda.ucar.edu/datasets/ds728.3/#!description NCAR Research Data Archive] . More information is available from the [https://precip.gsfc.nasa.gov/ GPCP webpage] .

And detailed documentation is available from the NCEI data server for [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/doc/ v1-3 daily] and [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-monthly/doc/ v2.3 monthly]. This documents are also available on the filesystem in the dataset folder.

=== '''License''' ===

Access is free but users are requested to add the official citation as below and email the citation to george.j.huffman@nasa.gov or david.t.bolvin@nasa.gov.

=== '''Citation''' ===

v1.2 Huffman, G. J., D. T. Bolvin, and R. F. Adler. 2016. GPCP Version 1.2 One-Degree Daily Precipitation Data Set. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. [https://doi.org/10.5065/D6D50K46 https://doi.org/10.5065/D6D50K46]. Accessed† 30 Jan 2018.

v1.3 daily - Adler, Robert; Wang, Jian-Jian; Sapiano, Mathew; Huffman, George; Bolvin, David; Nelkin, Eric; and NOAA CDR Program (2017). Global Precipitation Climatology Project (GPCP) Climate Data Record (CDR), Version 1.3 (Daily) [Indicate subset used.]. NOAA National Centers for Environmental Information. doi:10.7289/V5RX998Z [access date]

v2.3 monthly - Adler, Robert; Wang, Jian-Jian; Sapiano, Matthew; Huffman, George; Chiu, Long; Xie, Ping Ping; Ferraro, Ralph; Schneider, Udo; Becker, Andreas; Bolvin, David; Nelkin, Eric; Gu, Guojun; and NOAA CDR Program (2016). Global Precipitation Climatology Project (GPCP) Climate Data Record (CDR), Version 2.3 (Monthly). National Centers for Environmental Information. doi:10.7289/V56971M6 [access date]

=== '''References''' ===

Huffman, G. J., R. F. Adler, M. M. Morrissey, D. T. Bolvin, S. Curtis, R. Joyce, B. McGavock, and J. Susskind, 2001: [https://doi.org/10.1175/1525-7541(2001)002 <0036:GPAODD>2.0.CO;2 Global precipitation at one-degree daily resolution from multisatellite observations]. J. Hydrometeor., 2, 36-50 (DOI: 10.1175/1525-7541(2001)002<0036:GPAODD>2.0.CO;2).

 

=== '''GPCP on raijin''' ===

Data is available in the ua8 project

*/g/data/ua8/Precipitation/GPCP/<frequency>/<version>/<YYYY>/<files>

       where frequency is '''mon''' or '''day''' and versions are '''v1-2''' and '''v1-3 '''for daily data and '''v2-3''' for monthly

Files are in netcdf4 format and filenames are respectively

*gpcp_1dd_v1.2_p1d.YYYYMMDD.nc   for v1.2 daily

where the date is the represented day, for example:  gpcp_1dd_v1.2_p1d.199610.nc

*gpcp_v01r03_daily_dYYYYMMDD_cYYYYMMDD for v1.3 daily

where the first date is the represented day and the second is the creation date for the file, for example: gpcp_v01r03_daily_d20170323_c20171211.nc

* 

gpcp_v02r03_monthly_dYYYYMMDD_cYYYYMMDD.nc

same conventions as for daily v1.3

If preliminary files are present (currently applies only to v2.3 monthly) then this is indicated in the filenames as in this example:

*gpcp_v02r03-preliminary_monthly_d201908_c20190910.nc

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

GPCP

2022-06-20T04:41:56Z

P.petrelli:

The Global Precipitation Climatology Project, produced by the NASA Mesoscale Atmospheric Processes Laboratory, comprises threee products: One-Degree Daily (1DD), monthly product and Pentad data. We downloaded the following products:

- daily v1.2  this is limited to 2015 and currently not distributed anymore;

- daily v1.3 this is available up to a three months from present date and it is the recommended daily version;

- monthly v2.3 which is the recommended monthly version, goes up to present date, contains some preliminary files.

The GPCP One-Degree Daily Precipitation Data Set (1DD Data Set) provides daily, global 1x1-deg gridded fields of precipitation totals for October 1996 through the delayed present. The 1DD draws upon several different data sources, surface and satellite measurements, covering different areas of the globe. This product is updated sporadically, about once a month. We downloaded v1.3 daily and v2-3 monthly from the [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/ NCEI server], and v1.2 from the [https://rda.ucar.edu/datasets/ds728.3/#!description NCAR Research Data Archive] . More information is available from the [https://precip.gsfc.nasa.gov/ GPCP webpage] .

And detailed documentation is available from the NCEI data server for [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/doc/ v1-3 daily] and [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-monthly/doc/ v2.3 monthly]. This documents are also available on the filesystem in the dataset folder.

=== '''License''' ===

Access is free but users are requested to add the official citation as below and email the citation to george.j.huffman@nasa.gov or david.t.bolvin@nasa.gov.

=== '''Citation''' ===

v1.2 Huffman, G. J., D. T. Bolvin, and R. F. Adler. 2016. GPCP Version 1.2 One-Degree Daily Precipitation Data Set. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. [https://doi.org/10.5065/D6D50K46 https://doi.org/10.5065/D6D50K46]. Accessed† 30 Jan 2018.

v1.3 Simply use references, for the moment no specific indication has been given.

=== '''References''' ===

Huffman, G. J., R. F. Adler, M. M. Morrissey, D. T. Bolvin, S. Curtis, R. Joyce, B. McGavock, and J. Susskind, 2001: [https://doi.org/10.1175/1525-7541(2001)002 <0036:GPAODD>2.0.CO;2 Global precipitation at one-degree daily resolution from multisatellite observations]. J. Hydrometeor., 2, 36-50 (DOI: 10.1175/1525-7541(2001)002<0036:GPAODD>2.0.CO;2).

 

=== '''GPCP on raijin''' ===

Data is available in the ua8 project

*/g/data/ua8/Precipitation/GPCP/<frequency>/<version>/<YYYY>/<files>

       where frequency is '''mon''' or '''day''' and versions are '''v1-2''' and '''v1-3 '''for daily data and '''v2-3''' for monthly

Files are in netcdf4 format and filenames are respectively

*gpcp_1dd_v1.2_p1d.YYYYMMDD.nc   for v1.2 daily

where the date is the represented day, for example:  gpcp_1dd_v1.2_p1d.199610.nc

*gpcp_v01r03_daily_dYYYYMMDD_cYYYYMMDD for v1.3 daily

where the first date is the represented day and the second is the creation date for the file, for example: gpcp_v01r03_daily_d20170323_c20171211.nc

* 

gpcp_v02r03_monthly_dYYYYMMDD_cYYYYMMDD.nc

same conventions as for daily v1.3

If preliminary files are present (currently applies only to v2.3 monthly) then this is indicated in the filenames as in this example:

*gpcp_v02r03-preliminary_monthly_d201908_c20190910.nc

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

File:Publish your data.pdf

2022-06-16T02:12:38Z

P.petrelli: New version

== Summary ==
New version

Admin

2022-06-10T02:13:59Z

P.petrelli:

= Organisation =

== Research programs (RPs) and RP leads ==

{|
|-
| Research Program
| RP leads
| CMS person
|-
| Climate Variability
| Andy Hogg, Mat England, Dietmar Dommenget, Neil Holbrooke
| Aidan Heerdegen, Holger Wolff
|-
| Heatwaves and Cold air outbreaks
| Michael Reeder, Jason Evans
| Paola Petrelli
|-
| Rainfall Extremes
| Todd Lane, Julie Arblaster
| Scott Wales
|-
| Drought
| Mike Roderick, Andy Pitman
| Danny Eisenberg
|}

== CIs ==

Andy Pitman: Director

Todd Lane: Vice-director

Christian Jakob

Michael Reeder

Dietmar Dommenget

Steven Sherwood

Lisa Alexander

Julie Arblaster

Nerilie Abram

Gab Abramowitz

Nathaniel Bindoff

Jason Evans

Andy Hogg

Peter Strutton

= Policies =

== Blogposts ==

We can publish blogs whenever we have something worth to be published. Just write the post and ask for a review. Note: it can be faster to publish a post from a helpdesk question right after finishing helping the user than to postpone it.

== News ==

To communicate new work:

#update all documentation
#advertise on [[News|CMS news]] page

== Help desk ==

There is now a roster for the help desk:

{|
|-
| Monday
| Holger
|-
| Tuesday
| Ramzi
|-
| Wednesday
| CodeBreak 
|-
| Thursday
| Aidan
|-
| Friday
| Paola
|}

Main tasks on "help desk day" are:

#Helpdesk
#Documentation / training update. Don't forget this page with information on [[Editing_the_Wiki|editing the wiki]]
#Other work

== Priorities ==

We need to update our priorities regularly (~3 months). Update on Trello (and/or spreadsheet?) in consultation with the Infrastructure Committee. Communicate to CIs.

== NCI Projects husbandry ==

#Every 6 months, send a list of members for each CoE projects to CIs. It's a reminder to clean up the memberships.
#Everyone is in charge of caring for one or several projects. See tables below.
#Need to keep track every week or so of the storage and SU usage (use ncimonitor)
#'''Contact users''' if people are on track to exceed '''storage'''
#'''Contact Aidan''' if people are on track to exceed '''SU'''.

= NCI Projects =

See this [[CLEX_projects_at_NCI|page]]

== Other CoE-Related: ==

=== Compute ===

{|
|-
| dt6
| Andy Pitman
| Land stuff
|-
| e14
| Matthew England
| Ocean stuff
|}

=== Storage ===

{| style="width: 667px;"
|-
! Project
! Lead CI (or delegate)
! Description
! style="width: 62px;" | Carer
|-
| access
| Scott Wales, Claire Carouge
| Software group for ACCESS, need membership to run ACCESS
| style="width: 62px;" | Scott
|-
| ks32
| Paola Petrelli
| ARCCSS and CLEX data
| style="width: 62px;" | Paola
|-
| ub4
| Paola Petrelli
| Era-Interim Data
| style="width: 62px;" | Paola
|-
| rr3
| Yiling Liu
| ESGF CMIP5 Australian Data Publication
| style="width: 62px;" | Paola
|-
| al33
| Yiling Liu
| ESGF CMIP5 Data Replication
| style="width: 62px;" | Paola
|-
| oi10
| Yiling Liu
| ESGF CMIP6 Data Replication
| style="width: 62px;" | Paola
|-
| fs38
| Yiling Liu
| ESGF CMIP6 Australian Data Publication
| style="width: 62px;" | Paola
|-
| p66
| Tony Hirst
| ACCESS GCM
| style="width: 62px;" |  
|-
| rq7
| Paola Petrelli
| YOTC
| style="width: 62px;" | Paola
|-
| rr4
|  
| BoM ACCESS NWP data
| style="width: 62px;" |  
|-
| rr7
| Paola Petrelli
| Reanalyses
| style="width: 62px;" | Paola
|-
| wd9
| Paola Petrelli
| CABLE evaluation datasets
| style="width: 62px;" | Paola
|-
| hh5
| Paola Petrelli
| Climate LIEF storage grant
| style="width: 62px;" | Paola
|-
| ua8
| Paola Petrelli
| Miscellaneous data
| style="width: 62px;" | Paola
|-
| ik11
| Andy Hogg
| Shared COSIMA input data
| style="width: 62px;" | Aidan
|-
| sx70
| Holger Wolff
| Shared WRF input data
| style="width: 62px;" | Claire
|-
| ia39
| Richard Matear
| ACS public data collection
| style="width: 62px;" | Paola
|}

=== Cloud projects ===

{|
|-
| fe2_2
|  
| ARCCSS cloud
|-
| ju7_1
|  
| CWSlab
|}

== Requesting Met Office accounts for users ==

New accounts need a sponsor from their institution Sponsors are listed at [https://code.metoffice.gov.uk/trac/admin/wiki/GroupSponsors https://code.metoffice.gov.uk/trac/admin/wiki/GroupSponsors], Current ARCCSS sponsors are Scott, Claire and Holger Instructions for an account request are at [https://code.metoffice.gov.uk/trac/admin/wiki/NewUserChecklist https://code.metoffice.gov.uk/trac/admin/wiki/NewUserChecklist]

In brief, create a ticket at https:''code.metoffice.gov.uk/trac/admin/newticket?type=account-request with the user's name and email address, like https:''code.metoffice.gov.uk/trac/admin/ticket/758

= Useful contacts =

{|
|-
! DATA
!  
|-
| Ben Evans
| ben.evans@anu.edu.au
|-
| Claire Trenham
| claire.trenham@csiro.au
|-
| Clare Richards
| clare.richards@anu.edu.au
|-
| Kelsey Drunken
| kelsey.drunken@anu.edu.au
|}

{|
|-
! User Support
!  
|-
| Roger Edberg
| roger.edberg@anu.edu.au
|-
| Brian Davis
| brian.davis@anu.edu.au
|}

{|
|-
! ACCESS
!  
!  
|-
| Bob Pitt
|  
| In charge of access-svn.nci.org.au. (e.g. certificate renewal etc)
|}

= conda environment =

=== Process for update to the conda environment ===

Scott is responsible for the update of the stable environment every quarter and deletion of old environments. Aidan is responsible if Scott is away.

The stable environment is updated on '''the Tuesday after NCI's quarterly maintenance'''.

An announcement of the change is sent '''one week before the update'''.

CMS folks are responsible for listing highlights for the update on [http://climate-cms.wikis.unsw.edu.au/Conda|the conda wiki page] '''before the announcement'''.

If you want to ask for a delay for the update or you need to ensure the update goes as plan, or you need to retain an old environment, '''discuss it at a team meeting before the announcement'''.

=== Add package ===

CMS maintains an anaconda installation on the NCI systems. To add new packages to the unstable analysis environments clone the [https://github.com/coecms/conda-envs conda-envs repository]. The master branches contains nothing but a README. The magic happens in the analysis3 branch.

Checkout the analysis branch, add the package name to environment.yml and then commit your change.

Next push the altered branch to the repository.
<syntaxhighlight lang="bash">
git push origin analysis3
</syntaxhighlight>

Pushing the branch will trigger testing and updating on the jenkins instance. The status of the conda environment update is viewable here:

[https://accessdev.nci.org.au/jenkins/blue/pipelines?search=conda https://accessdev.nci.org.au/jenkins/blue/pipelines?search=conda]

=== Tips and tricks if problems with conda environments ===

*Scott said: "for whatever reason (something to do with external c libraries I think) loading packages is sometimes order-dependent." If a package breaks the tests, put its import at the top of the list. It would then be loaded before whatever package is causing the issue.

[[Category:Admin]]

CMS

2022-06-10T02:02:00Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.4); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
==== Ramzi Kutteh ====

*Land Surface Modelling
*[[CABLE|CABLE]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">

==== Paola Petrelli ====

*[[Data|Data]]

 

== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T02:01:21Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.4); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
==== Ramzi Kutteh ====

*Land Surface Modelling
*[[CABLE|CABLE]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T02:00:40Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
==== Ramzi Kutteh ====

*Land Surface Modelling
*[[CABLE|CABLE]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T01:59:00Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>

=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>

=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>

=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
==== Ramzi Kutteh ====

*Land Surface Modelling
*[[CABLE|CABLE]]
</div> </div>

=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T01:57:58Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
==== Ramzi Kutteh ====

*Land Surface Modelling
*[[CABLE|CABLE]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]
</div> </div>
[[Category:Admin]]

CMS

2022-06-10T01:55:52Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
===   ===

=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
=== Ramzi Kutteh ===

*Land Surface Modelling
*[[CABLE|CABLE]]

=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T01:53:46Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
===   ===

=== University of New South Wales ===
[[File:RamziKutteh-250x250.jpg|100x100px]]
=== Ramzi Kutteh ===

*Land Surface Modelling
*[[CABLE|CABLE]]

=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T01:50:18Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
===   ===

=== University of New South Wales ===

Ramzi Kutteh

 

*Land Surface Modelling
*[[CABLE|CABLE]]

=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T01:49:12Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
====   ====
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%"> </div> </div>
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
===   ===

=== University of New South Wales ===

Ramzi Kutteh

 

*Land Surface Modelling
*[[CABLE|CABLE]]

=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2022-06-10T01:46:39Z

P.petrelli:

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Claire-Carouge-2-440x440.jpg</div> <div style="width:80%">

====   ====
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%"> </div> </div>

==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Scott-Wales-3-440x440.jpg</div> <div style="width:80%">
==== Scott Wales ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMORPH

2022-06-01T00:14:19Z

P.petrelli:

'''NOAA CPC MORPHing technique: high resolution precipitation (60S-60N) v1.0'''

CMORPH produces global precipitation analyses at very high spatial and temporal resolution. This technique uses precipitation estimates that have been derived from low orbiter satellite microwave observations exclusively, and whose features are transported via spatial propagation information that is obtained entirely from geostationary satellite IR data. The grid resolution is 0.07277 degrees lat/lon (8 km at the equator) and temporal resolution is 30 minutes. With regard to spatial resolution, although the precipitation estimates are available on a grid with a spacing of 8 km (at the equator), the resolution of the individual satellite-derived estimates is coarser than that - more on the order of 12 x 15 km or so. The finer "resolution" is obtained via interpolation. (source [http://www.cpc.ncep.noaa.gov/products/janowiak/cmorph_description.html CPC website] ) The original data was downloaded from the [ftp://ftp.cpc.ncep.noaa.gov/precip/CMORPH_V1.0/RAW/8km-30min/| NOAA CPC ftp server].

New updates are now available from the [https://rda.ucar.edu/datasets/ds502.2/index.html#!description RDA UCAR data portal]. More information is available on the CPC website and the [https://climatedataguide.ucar.edu/climate-data/cmorph-cpc-morphing-technique-high-resolution-precipitation-60s-60n data climate guide website].

 

=== '''Citation''' ===

This dataset is freely available, please acknowledge the creators of the dataset. 

Xie, P., R. Joyce, S. Wu, S. -H. Yoo, Y. Yarosh, F. Sun, and R. Lin. 2020, updated daily.''NOAA Climate Data Record (CDR) of CPC Morphing Technique (CMORPH) High Resolution Global Precipitation Estimates, Version 1''. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. [https://doi.org/10.5065/0EFN-KZ90 https://doi.org/10.5065/0EFN-KZ90]. Accessed†dd mmm yyyy.

==== '''References''' ====

Ferraro, R. R., 1997: SSM/I derived global rainfall estimates for climatological applications. J. Geophys. Res., 102, 16715-16735.

Ferraro, R. R., F. Weng, N. C. Grody and L. Zhao, 2000: Precipitation characteristics over land from the NOAA-15 AMSU sensor. Geophys. Res. Ltr., 27, 2669-2672.

Joyce, R. J., J. E. Janowiak, P. A. Arkin, and P. Xie, 2004: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution.. J. Hydromet., 5, 487-503.

Kummerow, C., Y. Hong, W. S. Olson, S. Yang, R. F. Adler, J. McCollum, R. Ferraro, G. Petty, D-B Shin, and T. T. Wilheit, 2001: Evolution of the Goddard profiling algorithm (GPROF) for rainfall estimatin from passive microwave sensors. J. Appl. Meteor., 40, 1801-1820.

=== '''CMORPH at NCI''' ===

This dataset is available on gdata under the ua8 project

/g/data/ua8/CMORPH/CMORPH_V1.0/ --> raw/8km-30min/YYYY/ for the original binary files

--> netcdf/YYYY/pr_30min_CMORPH_V1_<fromdate>_<todate>.nc for the converted netcdf files where from date and today are in the format YYYYMMDD

For CMORPH V0.x data is in /g/data/ua8/CMORPH/CMORPH_V1.0/netcdf/

=== '''Important update: 25 June 2018''' ===

Since there are delays in releasing new data for V1-0 we downloaded its precursor V0.x from July 2017 to date included. The two versions should be compatible but please take into account that V0.x could potentially have issues.

We also concatenated the existing daily files for version v1-0 into monthly files. In two weeks time (09 July) we will delete the daily files and only the monthly will remain available. Finally raw files are now only available as tar or gzip files, i.e. as originally downloaded from the web.

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

CMORPH

2022-05-31T23:05:13Z

P.petrelli:

'''NOAA CPC MORPHing technique: high resolution precipitation (60S-60N) v1.0'''

CMORPH produces global precipitation analyses at very high spatial and temporal resolution. This technique uses precipitation estimates that have been derived from low orbiter satellite microwave observations exclusively, and whose features are transported via spatial propagation information that is obtained entirely from geostationary satellite IR data. The grid resolution is 0.07277 degrees lat/lon (8 km at the equator) and temporal resolution is 30 minutes. With regard to spatial resolution, although the precipitation estimates are available on a grid with a spacing of 8 km (at the equator), the resolution of the individual satellite-derived estimates is coarser than that - more on the order of 12 x 15 km or so. The finer "resolution" is obtained via interpolation. (source [http://www.cpc.ncep.noaa.gov/products/janowiak/cmorph_description.html CPC website] ) The original data is downloaded from the [ftp://ftp.cpc.ncep.noaa.gov/precip/CMORPH_V1.0/RAW/8km-30min/| NOAA CPC ftp server]. More information is available on the CPC website and the [https://climatedataguide.ucar.edu/climate-data/cmorph-cpc-morphing-technique-high-resolution-precipitation-60s-60n data climate guide website].

=== '''Citation''' ===

This dataset is freely available, please acknowledge the creators of the dataset. 

Xie, P., R. Joyce, S. Wu, S. -H. Yoo, Y. Yarosh, F. Sun, and R. Lin. 2020, updated daily.''NOAA Climate Data Record (CDR) of CPC Morphing Technique (CMORPH) High Resolution Global Precipitation Estimates, Version 1''. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. https://doi.org/10.5065/0EFN-KZ90. Accessed†dd mmm yyyy.

==== '''References''' ====

Ferraro, R. R., 1997: SSM/I derived global rainfall estimates for climatological applications. J. Geophys. Res., 102, 16715-16735.

Ferraro, R. R., F. Weng, N. C. Grody and L. Zhao, 2000: Precipitation characteristics over land from the NOAA-15 AMSU sensor. Geophys. Res. Ltr., 27, 2669-2672.

Joyce, R. J., J. E. Janowiak, P. A. Arkin, and P. Xie, 2004: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution.. J. Hydromet., 5, 487-503.

Kummerow, C., Y. Hong, W. S. Olson, S. Yang, R. F. Adler, J. McCollum, R. Ferraro, G. Petty, D-B Shin, and T. T. Wilheit, 2001: Evolution of the Goddard profiling algorithm (GPROF) for rainfall estimatin from passive microwave sensors. J. Appl. Meteor., 40, 1801-1820.

=== '''CMORPH at NCI''' ===

This dataset is available on gdata under the ua8 project

/g/data/ua8/CMORPH/CMORPH_V1.0/ --> raw/8km-30min/YYYY/ for the original binary files

--> netcdf/YYYY/pr_30min_CMORPH_V1_<fromdate>_<todate>.nc for the converted netcdf files where from date and today are in the format YYYYMMDD

For CMORPH V0.x data is in /g/data/ua8/CMORPH/CMORPH_V1.0/netcdf/

=== '''Important update: 25 June 2018''' ===

Since there are delays in releasing new data for V1-0 we downloaded its precursor V0.x from July 2017 to date included. The two versions should be compatible but please take into account that V0.x could potentially have issues.

We also concatenated the existing daily files for version v1-0 into monthly files. In two weeks time (09 July) we will delete the daily files and only the monthly will remain available. Finally raw files are now only available as tar or gzip files, i.e. as originally downloaded from the web.

 

[[Category:Dataset]] [[Category:Clex-managed-data]]

Large Ensembles

2022-05-23T05:29:55Z

P.petrelli:

{{Working on}}

We are in the process of collecting all the available Large Ensemble models data in one place for convenience. We usually download only subset of these models, folowing users requests, because of their huge size. If you need more of this data depedning on the size we might have to consult with the Infrastructure Committee before downloading it as there is not dedicated storage allocated to it.  

 

=== '''Large Ensembles at NCI''' ===

The dataset are accessible on gadi in the ua8 project:

/g/data/ua8/LE_models/<dataset>

The CESM1-CAM-BGC-LE, CCSM and CESM1-LME subsets are actually stored in ua6 and they are only linked in ua8 for convenience.

To actually get access to them you will need access to ua6.

NB ua6 process is in the process of being decomissioned so as for the data in ua8 these are all temporary solutions.

 

[[20CRv3|20CRv3]] -> 

CESM1-CAM5-BGC-LE -> /g/data/ua6/CESM1-CAM5-BGC-LE/

[[CESM1-LME]] -> /g/data/ua8/LE_models/CESM1-LME/

CCSM -> /g/data/ua6/CCSM/

[[OLENS|OLENS]] - Observational Large Ensemble. McKinnon, Karen, 2017, "Members of the Observational Large Ensemble", [https://doi.org/10.7910/DVN/VYH544 https://doi.org/10.7910/DVN/VYH544], Harvard Dataverse, V1

UM and ACCESS FAQ

2022-05-10T01:36:10Z

P.petrelli:

__TOC__

= Who can gain access to the UM and the ACCESS models? =

The full ACCESS model is made up of a number of sub-models for different processes - UM for the atmosphere, MOM for the ocean and CABLE for the land surface. Each of these components was created by different groups with different conditions of use.

The Met Office Unified Model is proprietary software released under a commercial licence. The Australian University community has a sub-licence to use the UM from CAWCR, managed by ARCCSS. Australian researchers outside of ARCCSS may request access to the model by emailing access_help@nf.nci.org.au. Note that our licence is restricted to running the model at NCI and does not permit sharing of model code outside of the UM partnership.

GFDL's Modular Ocean Model is open-source software available under the GPL. Any researcher is free to use, modify and share the model code.

CSIRO's CABLE model is proprietary software. See the [http://cawcr.gov.au/projects/access/cable/cable_licence_agreement.pdf | licence agreement] for conditions of use.

= What to do to gain access to the UM and the ACCESS models? =

ARCCSS researchers can get access to the UM and ACCESS models by contacting the CMS team at climate_help@nf.nci.org.au. Researchers outside of ARCCSS should contact the general ACCESS helpdesk at access_help@nf.nci.org.au. Using the models requires an account at [http://nci.org.au/ | NCI], the Australian national supercomputing centre.

= What to do to use the software in /projects/access/bin? =

Several tools useful for working with ACCESS are available at NCI under the path /projects/access/bin. These tools include output viewers and converters between the UM file format and standard formats like NetCDF and GRIB. To access these tools you must be registered as an ACCESS user (see above).

= How to convert UM hybrid vertical coordinates to pressure levels =

The formula for the vertical coordinate is: [http://climate-cms.wikis.unsw.edu.au/Vertical_Coordinates http://climate-cms.wikis.unsw.edu.au/Vertical_Coordinates] 

Iris will compute the height variable directly if the file you read in has the orography in. 

If you have the pressure field in your file as well, Metpy might then help you to interpolate your fields to specific pressure levels: [https://unidata.github.io/MetPy/latest/examples/sigma_to_pressure_interpolation.html#sphx-glr-examples-sigma-to-pressure-interpolation-py https://unidata.github.io/MetPy/latest/examples/sigma_to_pressure_interpolation.html#sphx-glr-examples-sigma-to-pressure-interpolation-py]

[[Category:Unified Model]] [[Category:ACCESS]] [[Category:FAQ]]

FAQs

2022-05-05T02:46:42Z

P.petrelli:

This page lists FAQs for the different model supported by the CMS team, as well as FAQs around accessing and publishing data.

= FAQ pages =

*[[UM_and_ACCESS_FAQ|UM and ACCESS FAQ]]
*[[Um-errors|UM error messages]]
*[[WRF_FAQ|WRF FAQ]]
*[[LIS_and_NUWRF_FAQ|LIS and NUWRF FAQ]]
*[[MOM_FAQ|MOM FAQ]]
*[[Data_FAQ|Data FAQ]]
*[[CESM_FAQ|CESM FAQ]]

= FAQ miscellaneous =

== How to debug issues with gadi_jupyter? ==

The script submitted to PBS and the PBS logs are saved under the directory tmp/runjp directory under the user's /scratch directory for the group used to run the script. These will often have more information if the script was started alright but you can't seem to be able to connect to the Jupyer Notebooks. This usually happens when the job started alright but exited due to an error in the script.

For example, let's say I (ccc561) requests gadi_jupyter to run using the project w35. The information will then be stored under '''/scratch/w35/ccc561/tmp/runjp'''

If I then run gadi_jupyter again but this time running on w40, the information will be stored under '''/scratch/w40/ccc561/tmp/runjp'''

== How do I run independent jobs in parallel? ==

If you have to run the same script with a lot of independent sets of inputs, or you have lots of similar scripts to run at NCI. And these scripts run on 1 processor. You could submit each of your jobs independently via PBS but then small jobs are not efficient at NCI. It would probably be better for you to have a master script that dispatches your jobs to available processors. You then submit this master script to PBS and this script requires a lot more processors and hence might be more efficient with the scheduler. There is an example of such a master script [[Embarrassingly_parallel_job|on this page]].

== How to transform 1D lat/lon arrays into 2D lat/lon arrays with Python? ==

If using numpy or masked arrays, one can use the resize() method. Assuming slat and slon are 1D latitude and longitude arrays, do this:
<syntaxhighlight lang="python">
tmp = numpy.resize(slat, (slon.size, slat.size))
slat2D = numpy.transpose(tmp)
slon2D = numpy.resize(slon, (slat.size, slon.size))

</syntaxhighlight>
If you want masked arrays, replace "numpy." by "numpy.ma."  
== How do I push my local code changes on gadi back to github using ssh keys? ==

You have an account on [http://github.com | github]. You have cloned a repository to raijin and made changes to it. Now you want to push those changes back to the github repository but you get errors like this (where <username> is your github username):
<syntaxhighlight lang="text">
error: The requested URL returned error: 403 Forbidden while accessing https://github.com/<username>/test.git/info/refs

fatal: HTTP request failed
</syntaxhighlight>
Firstly, you need to [https://help.github.com/articles/generating-ssh-keys/ | make an ssh key and add the public part of the key to your account on github]. Check the url to which you are trying to push your changes:   <syntaxhighlight lang="text">
git remote -v
origin https://github.com/<username>/test.git (fetch)
origin https://github.com/<username>/test.git (push)
</syntaxhighlight>

If it looks like the output above you need to alter the url and put in a login name in front of the address. For ssh access the login name is always git. The command to set the remote url is:
<syntaxhighlight lang="text">
git remote set-url origin ssh://git@github.com/<username>/test.git

</syntaxhighlight>
but replace <username> with your github username.
== Why do I get segfaults? ==

A segmentation fault means the program you are using has tried to read or write a memory location it does not have access to. This can mean there is an error in the code, or there is a system limit set. If the program runs for others, but not you, it may well be a system limit issue. This seems to be quite common when trying to write netcdf files. In some cases this can be overcome by setting your stack size to unlimited

For tcsh/csh add this to your .cshrc:
<syntaxhighlight lang="text">
limit stacksize unlimited
</syntaxhighlight>

For bash add this to your .bashrc:
<syntaxhighlight lang="text">
ulimit -s unlimited
</syntaxhighlight>

Note: you must log out and log back in again for this change to take effect.

== No files found or non-existent path error while running a PBS job ==

When submitting a job on gadi you need to declare explicitly what gdata and/or scratch projects your script need to access.

This is done by setting the storage flag in the job you are submitting:

PBS -l storage=gdata/hh5+gdata/ua8+gdata/e14

In the above case I want to use input data from gdata/ua8 save my results to gdata/e14 and use the conda modules which are in gdata/hh5

== Weird errors for a script that was previously working  ==

Make sure you have loaded the right modules and only them! Avoid to load modules directly in your .bash_profile, .bashrc files . They will be loaded every time and occasionally interfere with the modules you are loading for a specific task. 

It is useful and safe to add instructions as

module use /g/data/hh5/public/modules

as this is making the modules available without loading any.

Sometimes you need to install a python module in your user space ( $HOME/.local/... ) as it is not available in a managed conda environment. Again, occasionally modules stored in your local user space can interfere with other loaded modules, for example if they have different dependencies. If you find this is causing issues you can exclude the local environment by setting this environment variable

PYTHONNOUSERSITE=x (  =True and =1 should also work as values)

 

== Running out of memory when running a relatively small timeseries analysis using xarray/dask ==

Xarray loads data lazily, which means that it will only load the data values when you need them. However, the data is stored in the files in chunks and all values in a chunk will be loaded as a block. The most common pattern for chunks in netcdf files is to have a grid stored in the same chunk. So, if you run a calculation for a specific timestep, you will only load one chunk. This means that often timesteps are on separate chunks, often `time` has a chunk size of 1, where all timesteps for a particular grid point are stored on different chunks. If you don’t change the file chunking and try to run a calculation that needs all the timeseries for a single grid point, you will be effectively loading the entire file in memory.

If you want to rechunk your data to have all timesteps on the same chunks you can run:
<syntaxhighlight lang="text">
array = array.chunk(chunks={‘time’=-1})
</syntaxhighlight>

-1 means that the ‘time’ dimension size will be used as chunk size. If you have 100 timesteps, it would be equivalent to set the new chunks as {‘time’=100}

 

== Issues installing a R module on gadi ==

First of all, you might not need to install the module, as NCI is now managing some R environments which might include already what you need: [https://opus.nci.org.au/display/DAE/Getting+Started [https://opus.nci.org.au/display/DAE/Getting+Started]].

If this approach doesn't work for you and you need to install a module, make sure you have all the necessary modules loaded. As outlined in the [https://opus.nci.org.au/display/Help/R Gadi user guide] you need to load the same intel module that was used to install the R module you are using.
If you still have issues, contacting the NCI Helpdesk, rather than us, is more efficient as they have more experience with R.

== I want to use Matlab on gadi ==

You will need to setup your institutional Matlab license. We cannot help you troubleshooting any of this, just follow the [https://opus.nci.org.au/display/Help/MATLAB NCI instructions] and contact their Helpdesk if you run into any issue.

[[Category:FAQ]]

FAQs

2022-05-05T02:45:42Z

P.petrelli:

This page lists FAQs for the different model supported by the CMS team, as well as FAQs around accessing and publishing data.

= FAQ pages =

*[[UM_and_ACCESS_FAQ|UM and ACCESS FAQ]]
*[[Um-errors|UM error messages]]
*[[WRF_FAQ|WRF FAQ]]
*[[LIS_and_NUWRF_FAQ|LIS and NUWRF FAQ]]
*[[MOM_FAQ|MOM FAQ]]
*[[Data_FAQ|Data FAQ]]
*[[CESM_FAQ|CESM FAQ]]

= FAQ miscellaneous =

== How to debug issues with gadi_jupyter? ==

The script submitted to PBS and the PBS logs are saved under the directory tmp/runjp directory under the user's /scratch directory for the group used to run the script. These will often have more information if the script was started alright but you can't seem to be able to connect to the Jupyer Notebooks. This usually happens when the job started alright but exited due to an error in the script.

For example, let's say I (ccc561) requests gadi_jupyter to run using the project w35. The information will then be stored under '''/scratch/w35/ccc561/tmp/runjp'''

If I then run gadi_jupyter again but this time running on w40, the information will be stored under '''/scratch/w40/ccc561/tmp/runjp'''

== How do I run independent jobs in parallel? ==

If you have to run the same script with a lot of independent sets of inputs, or you have lots of similar scripts to run at NCI. And these scripts run on 1 processor. You could submit each of your jobs independently via PBS but then small jobs are not efficient at NCI. It would probably be better for you to have a master script that dispatches your jobs to available processors. You then submit this master script to PBS and this script requires a lot more processors and hence might be more efficient with the scheduler. There is an example of such a master script [[Embarrassingly_parallel_job|on this page]].

== How to transform 1D lat/lon arrays into 2D lat/lon arrays with Python? ==

If using numpy or masked arrays, one can use the resize() method. Assuming slat and slon are 1D latitude and longitude arrays, do this:
<syntaxhighlight lang="python">
tmp = numpy.resize(slat, (slon.size, slat.size))
slat2D = numpy.transpose(tmp)
slon2D = numpy.resize(slon, (slat.size, slon.size))

</syntaxhighlight>
If you want masked arrays, replace "numpy." by "numpy.ma."  
== How do I push my local code changes on gadi back to github using ssh keys? ==

You have an account on [http://github.com | github]. You have cloned a repository to raijin and made changes to it. Now you want to push those changes back to the github repository but you get errors like this (where <username> is your github username):
<syntaxhighlight lang="text">
error: The requested URL returned error: 403 Forbidden while accessing https://github.com/<username>/test.git/info/refs

fatal: HTTP request failed
</syntaxhighlight>
Firstly, you need to [https://help.github.com/articles/generating-ssh-keys/ | make an ssh key and add the public part of the key to your account on github]. Check the url to which you are trying to push your changes:   <syntaxhighlight lang="text">
git remote -v
origin https://github.com/<username>/test.git (fetch)
origin https://github.com/<username>/test.git (push)
</syntaxhighlight>

If it looks like the output above you need to alter the url and put in a login name in front of the address. For ssh access the login name is always git. The command to set the remote url is:
<syntaxhighlight lang="text">
git remote set-url origin ssh://git@github.com/<username>/test.git

</syntaxhighlight>
but replace <username> with your github username.
== Why do I get segfaults? ==

A segmentation fault means the program you are using has tried to read or write a memory location it does not have access to. This can mean there is an error in the code, or there is a system limit set. If the program runs for others, but not you, it may well be a system limit issue. This seems to be quite common when trying to write netcdf files. In some cases this can be overcome by setting your stack size to unlimited

For tcsh/csh add this to your .cshrc:
<syntaxhighlight lang="text">
limit stacksize unlimited
</syntaxhighlight>

For bash add this to your .bashrc:
<syntaxhighlight lang="text">
ulimit -s unlimited
</syntaxhighlight>

Note: you must log out and log back in again for this change to take effect.

== No files found or non-existent path error while running a PBS job ==

When submitting a job on gadi you need to declare explicitly what gdata and/or scratch projects your script need to access.

This is done by setting the storage flag in the job you are submitting:

#PBS -l storage=gdata/hh5+gdata/ua8+gdata/e14

In the above case I want to use input data from gdata/ua8 save my results to gdata/e14 and use the conda modules which are in gdata/hh5

== Weird errors for a script that was previously working  ==

Make sure you have loaded the right modules and only them! Avoid to load modules directly in your .bash_profile, .bashrc files . They will be loaded every time and occasionally interfere with the modules you are loading for a specific task. 

It is useful and safe to add instructions as

module use /g/data/hh5/public/modules

as this is making the modules available without loading any.

Sometimes you need to install a python module in your user space ( $HOME/.local/... ) as it is not available in a managed conda environment. Again, occasionally modules stored in your local user space can interfere with other loaded modules, for example if they have different dependencies. If you find this is causing issues you can exclude the local environment by setting this environment variable

PYTHONNOUSERSITE=x (  =True and =1 should also work as values)

 

== Running out of memory when running a relatively small timeseries analysis using xarray/dask ==

Xarray loads data lazily, which means that it will only load the data values when you need them. However, the data is stored in the files in chunks and all values in a chunk will be loaded as a block. The most common pattern for chunks in netcdf files is to have a grid stored in the same chunk. So, if you run a calculation for a specific timestep, you will only load one chunk. This means that often timesteps are on separate chunks, often `time` has a chunk size of 1, where all timesteps for a particular grid point are stored on different chunks. If you don’t change the file chunking and try to run a calculation that needs all the timeseries for a single grid point, you will be effectively loading the entire file in memory.

If you want to rechunk your data to have all timesteps on the same chunks you can run:
<syntaxhighlight lang="text">
array = array.chunk(chunks={‘time’=-1})
</syntaxhighlight>

-1 means that the ‘time’ dimension size will be used as chunk size. If you have 100 timesteps, it would be equivalent to set the new chunks as {‘time’=100}

 

== Issues installing a R module on gadi ==

First of all, you might not need to install the module, as NCI is now managing some R environments which might include already what you need: [https://opus.nci.org.au/display/DAE/Getting+Started [https://opus.nci.org.au/display/DAE/Getting+Started]].

If this approach doesn't work for you and you need to install a module, make sure you have all the necessary modules loaded. As outlined in the [https://opus.nci.org.au/display/Help/R Gadi user guide] you need to load the same intel module that was used to install the R module you are using.
If you still have issues, contacting the NCI Helpdesk, rather than us, is more efficient as they have more experience with R.

== I want to use Matlab on gadi ==

You will need to setup your institutional Matlab license. We cannot help you troubleshooting any of this, just follow the [https://opus.nci.org.au/display/Help/MATLAB NCI instructions] and contact their Helpdesk if you run into any issue.

[[Category:FAQ]]

FAQs

2022-05-05T02:09:29Z

P.petrelli:

This page lists FAQs for the different model supported by the CMS team, as well as FAQs around accessing and publishing data.

= FAQ pages =

*[[UM_and_ACCESS_FAQ|UM and ACCESS FAQ]]
*[[Um-errors|UM error messages]]
*[[WRF_FAQ|WRF FAQ]]
*[[LIS_and_NUWRF_FAQ|LIS and NUWRF FAQ]]
*[[MOM_FAQ|MOM FAQ]]
*[[Data_FAQ|Data FAQ]]
*[[CESM_FAQ|CESM FAQ]]

= FAQ miscellaneous =

== How to debug issues with gadi_jupyter? ==

The script submitted to PBS and the PBS logs are saved under the directory tmp/runjp directory under the user's /scratch directory for the group used to run the script. These will often have more information if the script was started alright but you can't seem to be able to connect to the Jupyer Notebooks. This usually happens when the job started alright but exited due to an error in the script.

For example, let's say I (ccc561) requests gadi_jupyter to run using the project w35. The information will then be stored under '''/scratch/w35/ccc561/tmp/runjp'''

If I then run gadi_jupyter again but this time running on w40, the information will be stored under '''/scratch/w40/ccc561/tmp/runjp'''

== How do I run independent jobs in parallel? ==

If you have to run the same script with a lot of independent sets of inputs, or you have lots of similar scripts to run at NCI. And these scripts run on 1 processor. You could submit each of your jobs independently via PBS but then small jobs are not efficient at NCI. It would probably be better for you to have a master script that dispatches your jobs to available processors. You then submit this master script to PBS and this script requires a lot more processors and hence might be more efficient with the scheduler. There is an example of such a master script [[Embarrassingly_parallel_job|on this page]].

== How to transform 1D lat/lon arrays into 2D lat/lon arrays with Python? ==

If using numpy or masked arrays, one can use the resize() method. Assuming slat and slon are 1D latitude and longitude arrays, do this:
<syntaxhighlight lang="python">
tmp = numpy.resize(slat, (slon.size, slat.size))
slat2D = numpy.transpose(tmp)
slon2D = numpy.resize(slon, (slat.size, slon.size))

</syntaxhighlight>
If you want masked arrays, replace "numpy." by "numpy.ma."  
== How do I push my local code changes on gadi back to github using ssh keys? ==

You have an account on [http://github.com | github]. You have cloned a repository to raijin and made changes to it. Now you want to push those changes back to the github repository but you get errors like this (where <username> is your github username):
<syntaxhighlight lang="text">
error: The requested URL returned error: 403 Forbidden while accessing https://github.com/<username>/test.git/info/refs

fatal: HTTP request failed
</syntaxhighlight>
Firstly, you need to [https://help.github.com/articles/generating-ssh-keys/ | make an ssh key and add the public part of the key to your account on github]. Check the url to which you are trying to push your changes:   <syntaxhighlight lang="text">
git remote -v
origin https://github.com/<username>/test.git (fetch)
origin https://github.com/<username>/test.git (push)
</syntaxhighlight>

If it looks like the output above you need to alter the url and put in a login name in front of the address. For ssh access the login name is always git. The command to set the remote url is:
<syntaxhighlight lang="text">
git remote set-url origin ssh://git@github.com/<username>/test.git

</syntaxhighlight>
but replace <username> with your github username.
== Why do I get segfaults? ==

A segmentation fault means the program you are using has tried to read or write a memory location it does not have access to. This can mean there is an error in the code, or there is a system limit set. If the program runs for others, but not you, it may well be a system limit issue. This seems to be quite common when trying to write netcdf files. In some cases this can be overcome by setting your stack size to unlimited

For tcsh/csh add this to your .cshrc:
<syntaxhighlight lang="text">
limit stacksize unlimited
</syntaxhighlight>

For bash add this to your .bashrc:
<syntaxhighlight lang="text">
ulimit -s unlimited
</syntaxhighlight>

Note: you must log out and log back in again for this change to take effect.

== No files found or non-existent path error while running a PBS job ==

When submitting a job on gadi you need to declare explicitly what gdata and/or scratch projects your script need to access.

This is done by setting the storage flag in the job you are submitting:

#PBS -l storage=gdata/hh5+gdata/ua8+gdata/e14

In the above case I want to use input data from gdata/ua8 save my results to gdata/e14 and use the conda modules which are in gdata/hh5

== Weird errors for a script that was previously working  ==

Make sure you have loaded the right modules and only them! Avoid to load modules directly in your .bash_profile, .bashrc files . They will be loaded every time and occasionally interfere with the modules you are loading for a specific task. 

It is useful and safe to add instructions as

module use /g/data/hh5/public/modules

as this is making the modules available without loading any.

Sometimes you need to install a python module in your user space ( $HOME/.local/... ) as it is not available in a managed conda environment. Again, occasionally modules stored in your local user space can interfere with other loaded modules, for example if they have different dependencies. If you find this is causing issues you can exclude the local environment by setting this environment variable

PYTHONNOUSERSITE=x (  =True and =1 should also work as values)

 

== Running out of memory when running a relatively small timeseries analysis using xarray/dask ==

Xarray loads data lazily, which means that it will only load the data values when you need them. However, the data is stored in the files in chunks and all values in a chunk will be loaded as a block. The most common pattern for chunks in netcdf files is to have a grid stored in the same chunk. So, if you run a calculation for a specific timestep, you will only load one chunk. This means that often timesteps are on separate chunks, often `time` has a chunk size of 1, where all timesteps for a particular grid point are stored on different chunks. If you don’t change the file chunking and try to run a calculation that needs all the timeseries for a single grid point, you will be effectively loading the entire file in memory.

If you want to rechunk your data to have all timesteps on the same chunks you can run:
<syntaxhighlight lang="text">
array = array.chunk(chunks={‘time’=-1})
</syntaxhighlight>

-1 means that the ‘time’ dimension size will be used as chunk size. If you have 100 timesteps, it would be equivalent to set the new chunks as {‘time’=100}

== Issues installing a R module on gadi ==

First of all, you might not need to install the module, as NCI is now managing some R environments which might include already what you need: [https://opus.nci.org.au/display/DAE/Getting+Started [https://opus.nci.org.au/display/DAE/Getting+Started]].

If this approach doesn't work for you and you need to install a module, make sure you have all the necessary modules loaded. As outlined in the [https://opus.nci.org.au/display/Help/R Gadi user guide] you need to load the same intel module that was used to install the R module you are using.

[[Category:FAQ]]