climate-cms wikis.unsw.edu.au - User contributions [en]

Compiling MOM6

2023-02-10T01:07:48Z

A.heerdegen: Change to https git url

These are gadi specific instructions to compile the ocean-only version of MOM6 using [https://github.com/angus-g/mom6-ninja-nci scripts developed by Angus Gibson]. 

To build ocean-only MOM6 on NCI:
<pre>cd /scratch/$PROJECT/$USER
git clone https://github.com/coecms/mom6-ninja-nci.git
cd mom6-ninja-nci
./cms_build.sh
</pre>

When this completes (it takes a few minutes) there should be a MOM6 executable at <code>ocean_only_symmetric/MOM6</code>.

See the [https://github.com/angus-g/mom6-ninja-nci original repository] for information on customising the build.

[[Category:MOM]] [[Category:MOM6]] [[Category:Compile]] [[Category:Ocean Models]]

CMS

2022-10-21T00:41:09Z

A.heerdegen: Removed Aidan

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===

=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px"> </div> <div style="width:80%">
==== <TBA> ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of New South Wales ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.4); height:100px">https://climateextremes.org.au/wp-content/uploads/RamziKutteh-250x250.jpg</div> <div style="width:80%">
==== Ramzi Kutteh ====

*Land Surface Modelling
*[[CABLE|CABLE]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]

 

== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]
</div> </div>
[[Category:Admin]]

CLEx induction

2022-08-21T07:05:56Z

A.heerdegen: Remove Aidan from CMS Team

The CMS team is here to help you with the technical aspects of your work. A lot of you (probably all) will have to work with computer code and software, usually in a Linux/Unix environment. You might also need to access and/or share large amounts of data, work on different servers, etc.

Some of you will be somewhat familiar with these tasks whereas they will be new to others. We are here to help whatever your technical knowledge is. We are happy to answer any question, we can't necessarily do everything but feel free to ask as we might have ideas or we might know who you should contact and what to exactly ask.

The members of the [[CMS|CMS]] team are:

{| class="wikitable"
|-
! Name
! Sitting at
! Email
! CMS "local" support person for
|-
| Paola Petrelli
| U Tasmania
| paola.petrelli@utas.edu.au
| U Tasmania
|-
| Holger Wolff
| Monash
| holger.wolff@monash.edu
| Monash
|-
| Ramzi Kutteh
| UNSW
| r.kutteh@unsw.edu.au
| UNSW
|}

The "local" support person will be the CMS person welcoming you to the Centre and possibly giving you an induction. But you are more than welcome to contact any of the CMS team members for support.

Although we won't ignore emails sent to our individual addresses, our preferred way of contact is through our help desk email address: [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Contacting the help desk will make sure your email is seen by one of us in case the person at your institution is away for example. Don't worry about the help desk email being an NCI account, you can still ask any question there.

[[Getting_Help|Here]] are some guidelines on formulating your requests to CMS

 

= Set up your computer =

We have put instructions for the setup we recommend for each operating system on the following pages:

*[[Getting_started_on_Windows|Windows]]
*[[Getting_started_on_Mac|Mac]]
*[[Getting_started_on_Linux|Linux]]

= Step 1: Get an account with NCI =

NCI operates the computers that you will be doing most of your work on. NCI provides services to the Centre, they are not part of the Centre of Excellence.

To get access to NCI servers, you need to do three things:

1. You need to find out which project(s) you need to be connected to.

Your project decides who gets billed for what you do on the NCI Servers. Your supervisor should be able to tell you which project(s) you should get connected to. Be mindful that there are two types of projects: projects for computation and projects for datasets. The mapping of the computational projects for CLEX can be found [[CLEX_projects_at_NCI|here]]. We recommend to join all the projects relevant to your research project but to choose 1 default project to work from. The same page with the list of computational projects explains how to set your default project. The projects for datasets do not have any computational time added to them so you need to make sure you ask for at least one computation project. Your supervisor should know. Note if you will run the ACCESS model or the UM model, you need to ask a connection to the "access" project in addition to at least one computation project.

2. You need to register with NCI.

This can be done on [https://my.nci.org.au/mancini/signup/0 NCI registration page] You will need to supply some information, and read and accept their policies. In order to be added to the system, you need to request connection to at least one project and be accepted on the project. Note that this can take a bit of time until the Lead chief investigator of the project has received and confirmed your request via email. Note that this is an automatic email process, you don't need to send an email to the lead chief investigator. Although, you might want to do so if you think you need to introduce yourself to him/her. Note that you should be able to request connection to several projects while getting signed up.

3. You might want to ask your supervisor whether you also should be connected to any of the following data projects.

These don’t provide any computing allowances, but they give access to certain datasets or models that you might need.

{| class="wikitable"
|-
! Project
! Purpose
! Administrator
|-
| '''ua8'''
|  GSWP3, NCEP Polar sst, OISST, CMIP5 ocean processing, model ensemble data + others
| CMS
|-
| '''rq7'''
| [[YOTC|ECMWF Year of Tropical Convection re-analysis]]
| CMS
|-
| '''ks32'''
| ARCCSS and CLEX published data
| CMS
|-
| '''ub4'''
| [[ERA_INTERIM|ECMWF ERA Interim re-analysis 6 hrs data]] and
| CMS
|-
| '''cb20'''
| [[CMIP|CMIP3]] replica data
| NCI
|-
| '''al33'''
| [[CMIP|CMIP5]] replica data
| NCI
|-
| '''rr3'''
| [[CMIP|CMIP5]] published data
| NCI
|-
| '''oi10'''
| [[CMIP|CMIP6]] replica data
| NCI
|-
| '''fs38'''
| [[CMIP|CMIP6]] published data
| NCI
|-
| '''rr7'''
| Use only for JRA55, MERRA2 the other datasets have been removed
| BoM
|-
| '''cable'''
| Give access to the [https://trac.nci.org.au/trac/cable/wiki CABLE model] and [[CABLE_Data|CABLE benchmarking dataset]]. Please read the [https://trac.nci.org.au/trac/cable/wiki/CableRegistration CABLE registration] information.
| CSIRO
|-
| '''access'''
| Give access to the [[Unified_Model|Unified Model]] and ACCESS tools
| CMS - BoM - CSIRO
|-
| '''rt52'''
| [[ERA5|ECMWF ERA5 re-analysis]] single and pressure levels
| NCI
|-
| '''zz93'''
| [[ERA5|ECMWF ERA5 re-analysis]] Land data
| NCI
|-
| '''sx70'''
| Shared input data for WRF
| CMS
|-
| '''ik11'''
| Shared data for COSIMA. Inputs and outputs
| COSIMA
|-
| '''ia89'''
| 400m resolution ACCESS limited-area simulation over Australia
| CMS
|-
| '''gy85'''
| Himawari-8 GeoCat 1.0.3 Australian Domain
| CMS
|}

 

= Step 2: Set up your Connection =

The standard method of connecting to the NCI systems is SSH. We want you to use a passphrase protected private key authorisation method with an ssh-agent and agent forwarding for convenience. If you understood every part of that sentence, go ahead and set it up. If not, the next part describes what to do.

The instructions describe how to set up your connection to both gadi.nci.org.au and accessdev.nci.org.au. You '''can not''' set up a connection to accessdev if you are not going to use the ACCESS or UM models. So please only follow the instructions you need!

 

== Linux and MacOS: ==

#'''Create the ssh directory''' on your computer <syntaxhighlight lang="bash">$ mkdir -p ~/.ssh</syntaxhighlight>

 

#'''Create a config file''' <syntaxhighlight lang="bash">$ nano ~/.ssh/config </syntaxhighlight>

Enter the following contents, replacing <span style="font-family:monospace">xx0000</span> with your NCI user name.
<syntaxhighlight lang="text">
Host gadi gadi.nci.org.au
HostName gadi.nci.org.au
User xx0000
ForwardX11 yes
ForwardX11Trusted yes
Host access accessdev.nci.org.au
HostName accessdev.nci.org.au
User xx0000
ForwardX11 yes
ForwardX11Trusted yes
ForwardAgent yes
</syntaxhighlight>
If you use a Mac, and you want to store the key in your keychain so that it gets automatically loaded when you log in, you can add this section to your config file: <syntaxhighlight lang="text">
Host *
AddKeysToAgent yes
UseKeychain yes
</syntaxhighlight>

#'''Create a key pair.''' It is imperative that you select a strong passphrase when asked for it. <syntaxhighlight lang="bash">$ ssh-keygen -t rsa</syntaxhighlight>

 

#'''Distribute the key pair.''' On most systems, there is a useful script to distribute the key pairs called '''ssh-copy-id''': <syntaxhighlight lang="bash">
$ ssh-copy-id gadi
$ ssh-copy-id access
</syntaxhighlight>

If that script fails, you have to distribute the keys manually with these commands:
<syntaxhighlight lang="bash">
$ cat ~/.ssh/id_rsa.pub | ssh gadi "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"
$ cat ~/.ssh/id_rsa.pub | ssh access "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"
</syntaxhighlight>

#'''Test whether you have an agent.''' If you have an agent already running, this command will ask for the passphrase for the just-created key pair: <syntaxhighlight lang="bash">$ ssh-add</syntaxhighlight>

If the last command told you that it couldn’t open a connection, then you don’t have an agent. Come talk to someone of the CMS team, we will help you set it up.

 

== Windows ==

Since Windows has a different heritage to Mac and Linux some extra setup work is required to use SSH. See the [[Getting_started_on_Windows|Getting started on Windows]] page for details.

If you're using [[Getting_started_on_Windows#Windows_Subsystem_for_Linux_.28WSL.29|WSL]] or [[Getting_started_on_Windows#Cygwin|Cygwin]] use the [[#Linux_and_MacOS:|SSH instructions for Mac and Linux]] above.

If using Power Shell, generate your ssh key with (Power Shell doesn't support 'rsa' keys)
<syntaxhighlight lang="unknown">
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa
</syntaxhighlight>
The contents of C:\Users\USER\.ssh\id_ecdsa.pub then need to be copied into ~/.ssh/authorized_keys on Gadi manually, as Power Shell doesn't have the ssh-copy-id command. The Power Shell SSH config file is located in your user directory, e.g. C:\Users\USER\.ssh\config
== SSH-keys for file transfers ==

Additionally, some groups transfer data from NCI to outside NCI, for example for storage on University maintained servers. For these transfers, the best way to proceed is to create a restricted ssh key pair used only for file transfer. The setup is explained in [https://opus.nci.org.au/display/Help/Using+SSH+keys Using restricted commands for transferring files]. Most of the instructions are still valid, except for step 3. The restricted command prefix should now be:
<syntaxhighlight lang="text">
from="gadi-dm*.nci.org.au,gadi*.nci.org.au",command="~/bin/rrsync /data/archive",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa
</syntaxhighlight>

= Step 3: Basics of the NCI system =

<span style="line-height: 1.5;">We are mainly using 3 systems of the NCI: gadi, accessdev and VDI. Gadi is the supercomputer that we run our models on. Accessdev is a virtual server that we have software installed that are required to allow the running the models. VDI is a virtual desktop designed for interactive analysis of your results. All these servers run Linux as operating system.</span>

== Read the manual! ==

Considering you will be using systems that are provided by the NCI, you should take the time to read the documentation provided by them about their systems. That is everything on their [https://opus.nci.org.au/display/Help/NCI+Help help knowledge base].

== Additional notes ==

=== Modules: ===

The servers have a lot of software installed, some of which they have multiple versions of. If you need to use a software, you usually will need to load a certain module. A full help of the module command is accessible through the man pages on gadi. Here are the most important commands regarding the modules:

{|
|-
| $ module <span style="color: #8000ff;">help</span>
| Show help about the module system
|-
| $ module <span style="color: #8000ff;">avail</span>
| List all available modules (a lot!)
|-
| $ module <span style="color: #8000ff;">list</span>
| List the currently loaded modules
|-
| $ module <span style="color: #8000ff;">load</span> intel-compiler
| Load the default version of the intel compiler
|-
| $ module <span style="color: #8000ff;">load</span> intel-compiler/2019.3.199
| Load a specific version of the intel compiler
|-
| $ module <span style="color: #8000ff;">switch</span> intel-compiler/2019.3.199
| Switch to a different version of the intel compiler (unload and load)
|-
| $ module <span style="color: #8000ff;">unload</span> intel-compiler/2019.3.199
| Unload the specified version of the intel compiler
|-
| $ module <span style="color: #8000ff;">use</span> /g/data/hh5/public/modules
| Add our Centre's curated modules to the list of available modules
|}

Some modules conflict with one another, for example different versions of the same software. You might have to unload one module first to load a different one.

=== PBS: ===

The other important system you need to understand in order to work on gadi is the scheduler PBS. This is the system for running software on dedicated nodes inside gadi. Basically, a user tells PBS that they want to run this job on this many CPUs with these resources for at least so long, and PBS places it in a queue, and when the required resources are available, it will assign this job to them.

Again, NCI has a very good documentation about how to run this [https://opus.nci.org.au/display/Help/How+to+submit+a+job here]

The most important commands are:

*'''qsub''' – Submit a job to the queue
*'''qstat''' – Display the status of the queued jobs
*'''qdel''' – Remove a job from the queue

Of these, qsub is the most important, and there are so many parameters that we won’t tell them all here.

In general, the fewer resources you request, the earlier your job will run. But if your program exceeds the requested resources at any time, PBS will kill the job, and you have to start over.

More specifically:

*Gadi is comprised of nodes with 48 CPUs each. If you request less than 48 CPUs, you will get the number of CPUs that you request. If you request more, you will need to use a multiple of 48 CPUs.
*Nodes come with 192GB of memory to share within the 48 CPUs. So if your job requires 48 or more CPUs, you might as well request all the memory, for example 384GB for a 96-cpu job.

Running a job on the queue costs so-called '''SU'''. On the old raijin, one SU was equivalent to one core for one hour. On the much faster new gadi, one core for one hour costs 2 SU. Note that on gadi, the SU cost is calculated using the higher of the number of cores and the memory usage. So if you request 5 cores but only 2GB memory, its SU cost is based on the 5 cores. If you request 1 core but 96GB of memory (half of the whole node), the cost will be based on 24 cores (half the whole node).

= Step 4: How to run models =

A lot of information to gain access to models or run the models can be found on the ARCCSS CMS wiki (see link at the end).

A very important message from the CMS team: <span style="background-color: #fdf988; display: block; text-align: center;">Often you will work with source code. Please ensure that your version of the source code is under version control! We use SVN and git (usually hosted on github) for the model codes and smaller utilities.</span><span style="background-color: #fdf988; display: block; text-align: center;">'''DO NOT JUST COPY SOURCE CODE! CHECK IT OUT, OR CHECK IT IN!'''</span>

== UM ==

The Unified model is currently transitioning from the old user environment umui to the new rose/cylc user interface. Either way, you need to log into accessdev.nci.org.au.

=== umui(x) ===

The old user interface has to be used for versions up to 7.x, and can be used for 8.x. umui was developed by the UM, umuix is very similar, but more convenient developed by CSIRO. An introduction into its use is on our wiki: [http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI]

=== rose/cylc ===

The new user interface can be used from version 8.x, and has to be used for versions 9.0 and later.

== WRF ==

The CMS team ports the WRF model at NCI. Information on using the model is on [[WRF|the CMS wiki]]. Users should also be aware that we do not reproduce the information from the [http://www2.mmm.ucar.edu/wrf/users/ WRF model users website] which stays a very important resource for all users. In particular, every new user is strongly encouraged to run through the official tutorials (found under User Support on the WRF model users site). As found on the WRF pages on this wiki, only the compilation steps differ from the NCAR tutorial.

== CABLE ==

CABLE is under licence so you need to request access to the code. All the information you need to get access and use CABLE is on the [https://trac.nci.org.au/trac/cable CABLE trac site]

= Data management =

Most of the information on how the data is managed is stored in this wiki in the [[Data|Data Services]] page. Here you can find information on the datasets managed by the Center as well the relevant policies in regard to data. Take some time to get familiar with it, here are some suggestions:

#We now offer a 1-day Data Induction training, if you have an opportunity to participate do so, particularly if you are just starting. If you missed out, all the [[Data_induction|course materials]] are stored on the wiki and we tried to create a path for you touching all the relevant wiki pages. There are 3 parts to the data induction, the first covers the policies, the second focuses on publishing and the third the data and code management best practices. 
#We strongly suggest to have a [[Data_Management_Plan|data management plan]] from the start of your project. You can use the [https://clex.dmponline.cloud.edu.au/ CLEX DMPonline tool] to do this. Because this tool stores your plan into a database, you can save it and access it as many times as you want, which means you can build your data management plan as you progress with your research. It is good though to start as soon as you can, DMPonline is structured as to teach about data practices and the resources available to you, it also informs the CMS team on which data, software and/or infrastructure you are planning to use, so we can better tailor our efforts to your needs.
#Get familiar with [[Publishing_options|publishing options]], while this might seem faraway, it is a good idea to keep track from now of what you are doing in regard to data and metadata, as it is often much more difficult to retrieve this information at the end of your project. There are also some technical times involved in getting data properly published.
#Consider creating a [[Persistent_research_identifiers|researcher-ID]] to uniquely identify your work.
#Make sure you browse through all the available data sources before you download data, this wiki page cover [[Finding_datasets|how to find out if a dataset]] is available at NCI. You could be wasting time and storage downloading something which might be already available, or that we could download and manage for you
#Become familiar with the different filesystems available on any server you are using, make sure you know which are meant to be working spaces, where you can archive your data, how to share data with collaborators and best practices to transfer data within and out of the system. If you are using gadi on NCI you can find this information on their [http://nci.org.au/nci-systems/national-facility/data-storage/filesystems/ user support ]and training sections.
#You can explore a lot of data related information in the [https://ardc.edu.au Australian Research Data Commons] website
#Don't hesitate in contacting cws_help<at>nci.org.au for any question you might have, or also e-mail directly paola.petrelli<at>utas.edu.au, if you have a more general question or you are confused about data management. We are always available to discuss with you the details of your project and give you advice.

= How to learn more =

== General setup: ==

a blog post by a UNSW's PhD student on a few things to know and learn: [http://web.maths.unsw.edu.au/~oangelil/blog/blog2/page_blog2.html Oliver Angélil's post]

== CMS: ==

*E-Mail: [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*Chat with us on [https://arccss.slack.com Slack] (Use your university email in your sign up.)
*This [[Home|Wiki]]
*In person.

== NCI: ==

[https://opus.nci.org.au/display/Help/NCI+Help Guides and reference pages]

[http://nci.org.au/services-support/training/ Training material] and [https://training.nci.org.au/ this]

== Version Control with git: ==

(Note: git might also be very useful for your thesis!) [https://www.youtube.com/playlist?list=PLg7s6cbtAD165JTRsXh8ofwRw0PqUnkVH Git video tutorial by GitHub]

== Data management: ==

[[Data_induction|Data induction]]

= Getting help =

Your first port of call for help should be the CMS team via the help-desk: [mailto:cws_help@nci.org.au cws_help@nci.org.au]. This help desk is followed by all the members of the CMS team and by some staff from NCI. [[Getting_Help|Here]] are some guidelines on formulating your request

[[Category:Training]]

Conda

2022-07-08T06:05:50Z

A.heerdegen: Updated conda history

 

== Conda Python Environments ==

CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.

You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.

To use any of the conda environments:

#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once)
#You must first run (to do at each session, a qsub job is a new session)

<code>module use /g/data/hh5/public/modules</code>

You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.

If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5
</syntaxhighlight>

Note that these <code><tt>conda</tt></code> environments will work on gadi login and compute nodes, and in the OOD cloud environment. They '''do not work '''on <code>accessdev</code>, as this has a very old system version that is no longer compatible.

=== Stable Environment ===

We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.

<code>module load conda/analysis3</code>

=== Unstable Environment ===

The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.

<code>module load conda/analysis3-unstable</code>

When we do our quarterly update the unstable environment becomes the new stable environment.

=== Removed Environments ===

Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].

== Creating personal environments ==

You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create.

'''Make a file <tt><code>~/.condarc</code></tt>''' like:
<syntaxhighlight lang="none">auto_activate_base: false
envs_dirs:
- /scratch/$PROJECT/$USER/conda/envs
- /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
- /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
root-dir: /scratch/$PROJECT/$USER/conda/bld
</syntaxhighlight>

This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.

 

'''To create the conda environment, load the conda module, then deactivate it''' with
<syntaxhighlight lang="bash">conda deactivate
conda env create ...
conda activate ...
</syntaxhighlight>

 

'''Create an environment file: <tt>environment.yml</tt>.''' Files on <tt>scratch</tt> are deleted if no longer used, so this file allows you to re-create your environment if some of the files are deleted. You can use any name for the environment file. Keep it secure.
<pre>conda env export > environment.yml</pre>

To recreate the environment from the environment file:
<pre>conda env create -f environment.yml</pre>

== Interactive Analysis / Jupyter ==

[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.

The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service [https://ood.nci.org.au https://ood.nci.org.au]. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:

*Module Directories: /g/data/hh5/public/modules
*Modules: conda/analysis3

The Centre has developed a script ('''<span style="font-family:Arial,Helvetica,sans-serif;">gadi_jupyter</span>'''<span style="font-family:Courier New,Courier,monospace;">)</span> that can run Jupyter on gadi compute nodes and display the notebook interface on your local computer. These scripts are available at [https://github.com/coecms/nci_scripts https://github.com/coecms/nci_scripts], see the instructions there for usage.

You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.

Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).

== Requesting new packages ==

You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.

It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment

As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.

== Update History ==

=== 22.04 (Current Stable) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://lmfit.github.io/lmfit-py/ lmfit] Non-Linear Least-Squares Minimization and Curve-Fitting for Python
*'''Climate specific'''
**[https://github.com/ccarouge/benchcab benchcab] CABLE benchmarking
**[https://xclim.readthedocs.io/en/stable/ xclim] library of functions to compute climate indices from observations or model simulations. It is built using xarray and can benefit from the parallelization handling provided by dask
**[https://github.com/aus-ref-clim-data-nci/acs-replica-intake acs-replica-intake] Intake-esm catalogue for the Australian Reference Climate Data at NCI collection
*'''Developer'''
**[https://github.com/jupyter-server/jupyter-resource-usage jupyter-resource-usage] extension for Jupyter Notebooks and JupyterLab that displays an indication of how much resources your current notebook server and its children (kernels, terminals, etc) are using

=== 22.01 (Unsupported) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://pygam.readthedocs.io/en/latest/ pygam] build Generalized Additive Models in Python
**[https://pyspectrum.readthedocs.io/en/latest/ spectrum] tools to estimate Power Spectral Densities based on Fourier transform, parametric methods or eigenvalues analysis
*'''Climate specific'''
**[https://xmhw.readthedocs.io/en/latest/ xmhw] xarray compatible Marine Heatwave Detection
**[https://argopy.readthedocs.io/en/latest/ argopy] library dedicated to [https://argopy.readthedocs.io/en/latest/what_is_argo.html#what-is-argo Argo] data access, manipulation and visualisation for standard users as well as Argo experts.
**[https://www.ilamb.org/doc/ ilamb] the International Land Model Benchmarking ([http://www.ilamb.org/ ILAMB]) project is a model-data intercomparison and integration project designed to improve the performance of land models
*'''Developer'''
**[https://github.com/lidatong/dataclasses-json dataclasses-json] provides a simple API for encoding and decoding [https://docs.python.org/3/library/dataclasses.html dataclasses] to and from JSON
*'''Geospatial'''
**''''''[https://pysal.org pysal] spatial analysis library for open, cross platform geospatial data science

=== <del>21.10</del> ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays
**[https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset

*'''Geospatial'''
**[https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline
**[https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes
**[https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs

*'''Developer Tools'''
**[https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols
**[https://github.com/gorakhargosh/watchdog watchdog] monitor file system events
**[https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more
**[https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure
**[https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol
**[https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material
**[https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data
**[https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder

'''Known Issues'''

*MetPy is incompatible with the Matplotlib version in this environment [https://github.com/Unidata/MetPy/issues/2281 https://github.com/Unidata/MetPy/issues/2281]

=== <s>21.07</s> ===

Python has been updated to version 3.9. See what's new at [https://docs.python.org/3/whatsnew/3.9.html https://docs.python.org/3/whatsnew/3.9.html]

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning
***Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes
**[https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model

*'''Geospatial'''
**[https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models

*'''Visualisation'''
**[https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python

*'''Developer Tools'''
**[https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line

=== <s>21.04</s> ===

Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems
**[https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library
**[https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends

*'''Geospatial'''
**[https://xarray-spatial.org/ xarray-spatial] common raster analysis functions

*'''Visualisation'''
**[https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects

*'''Developer Tools'''
**[https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git
**[https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB
**[https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]
**[https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information

=== <s>21.01</s> ===

The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf

era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.

'''Notable new packages'''

*'''Statistics'''
**[https://ml.dask.org/index.html dask-ml] Scalable machine learning
**[https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit
**[https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data

*'''Geospatial'''
**[http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis
**[https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python

*'''Visualisation'''
**[https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation
**[https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation
**[https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps

*'''Developer Tools'''
**[https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup
**[https://github.com/psf/black black] Python code formatter
**[https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool

=== <s>20.10 (Unsupported)</s> ===

Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])

There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working

*pynio ([https://github.com/conda-forge/pynio-feedstock/issues/90 https://github.com/conda-forge/pynio-feedstock/issues/90])
*<s>cf_units</s> (was renamed to 'cf-units')
*<s>pymunge</s> (fixed)
*<s>ants</s> (fixed)

'''Notable new packages'''

*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib
*[https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries
*[https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution
*[https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios
*[https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data
*[https://github.com/jdowner/gist python-gist] a command line interface for working with github gists
*[https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo

=== <s>20.07 (Unsupported)</s> ===

'''Notable New Packages'''

*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles)
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output

=== <s>20.04 (Unsupported)</s> ===

Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes

Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI

'''Notable New Packages'''

*geopy: Locate lat/lon coordinates of places
*ninja: mom6 build system

=== <s>20.01 (Unsupported)</s> ===

Python has been updated to 3.7

'''Notable New Packages'''

*xlrd: Read excel files
*ants: Unified Model Ancillary tools
*climtas: Dask-aware Xarray timeseries processing

=== <s>19.10 (Unsupported)</s> ===

basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages

'''Notable New Packages'''

*addmeta
*[http://holoviews.org/ holoviews]: Fancy plots
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes

=== <s>19.07 (Unsupported)</s> ===

'''Notable New Packages'''

*pyferret
*pyngl
*pynio
*xgcm
*xrft

=== <s>19.04 (unsupported)</s> ===

'''Notable Changes'''

*arccssive has been renamed to clef
*A bug preventing Iris from opening UM files without a date has been fixed

'''Notable New Packages'''

*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py
*[https://github.com/intake/intake intake] Lightweight data catalogues
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays

[[Category:NCI]] [[Category:Python]]

Migrate experiment projects

2022-06-28T23:10:21Z

A.heerdegen:

This page contains information on what to do if you want to change project to work at NCI.

 

= Setting your default project =

To change the default project that jobs get submitted to when you run 'qsub', edit the file on Gadi '~/.config/gadi-login.conf'

If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'

If the project listed in those files is either w35 or w48, please edit the file and list w97 or w40 as appropriate.

In either case you will need to log out and back in again for the change to take effect

= Setting your working space =

You will need to create a new working directory for you under the new project if you weren't already a member. You can create the directory under /g/data/<PROJECT> as needed. Please make sure to use your NCI login id as your top directory name.

All data copied from w35 will be stored under: /g/data/w97/W35_GDATA_MOVED and /scratch/w97/W35_SCRATCH_MOVED. Make sure to move your data into your proper working directory under /g/data and /scratch before working with it.

= Climate model experiments =

*[[Migrate_payu_experiment_project|Payu experiment]]
*[[Migrate_umui_experiment_project|UMUI experiment]]
*[[Migrate_rose_experiment_project|Rose/Cylc experiment]]
*[[Migrate_NUWRF_experiment_project|NUWRF experiment]]

WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.

 

= Tips for other scripts =

In your script, you may have used w35 or w48 in the following lines:

*PBS directive -P to specify the project
*PBS directive -l storage to load the storage space for these projects.
*Full paths of input or output data

It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.

It is important for you to make sure you know what changes you are doing, that's why we do not provide a magic <code>sed</code> one-liner command to change all the project IDs in all your files at once.

== Find files with the project listed in ==

To find all the files that contain the mention of <samp>w35</samp>, you can use the following command:
<pre>grep --exclude=*.nc -irlI w35 <directory_name> > files_to_update.txt</pre>

<samp>files_to_update.txt </samp>will contain a list of files where <samp>w35</samp> appear in at least one line. The command will not search in binary files and netCDF files and you can exclude more files from the search (e.g. figures) with the <code>--exclude=</code> option. Replace <samp><directory_name></samp> with the directory you want to search through.

At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only these:
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt
</pre>

== Backup before modifications ==

You will want to create some sort of backup of your text files before changing them. <span style="color:#c0392b">DO NOT create copies. </span>We do not have heaps of free storage around to allow for it. Our recommended method is to use Git. If your files are not under the Git version control yet, here is a simple workflow that allows to record the state of your files before change. And some tips on comparing the modified and initial versions of the files.

Let's say, you want to change some files under a directory called <tt>my_scripts</tt>. You will first need to create a Git repository from that directory, record the current state of your files, do the changes you want to do, test the modified files work:
<pre># Create repository. Go in your my_scripts directory
git init

# Record current state
git add .
git commit -m "State at time of merge"

# Change your files as you require, test the new files</pre>

If something does not work, you can then use the following commands depending on what you want to do:
<pre># Difference between modified file and previous version
git diff <filename>

# Re-instate the previous version and delete the modified file
git checkout <filename></pre>

== Find all and replace all ==

All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality, including reviewing the changes before making the change, for your favourite text editor.

= MDSS data migration =

After w35 and w48 are deleted any data stored on the [[How_to_use_MDSS_tape_storage_at_NCI|MDSS tape storage system]] would be inaccessible. To allow continued access all data has been migrated to a directory in the corresponding project (w35 to w97, w48 to w40).
<pre style="background-color: rgb(238, 238, 238); border: 1px solid rgb(204, 204, 204); padding: 5px 10px;">$ mdss -P w40 ls -ld W48_MASSDATA_MOVED
drwxrws--- 22 root w40 411 Mar 2 10:33 W48_MASSDATA_MOVED
$ mdss -P w97 ls -ld W35_MASSDATA_MOVED
drwxrws--- 27 root w97 143360 Mar 2 11:47 W35_MASSDATA_MOVED</pre>

Migrate experiment projects

2022-06-28T23:09:09Z

A.heerdegen: Add note about mdss

This page contains information on what to do if you want to change project to work at NCI.

 

= Setting your default project =

To change the default project that jobs get submitted to when you run 'qsub', edit the file on Gadi '~/.config/gadi-login.conf'

If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'

If the project listed in those files is either w35 or w48, please edit the file and list w97 or w40 as appropriate.

In either case you will need to log out and back in again for the change to take effect

= Setting your working space =

You will need to create a new working directory for you under the new project if you weren't already a member. You can create the directory under /g/data/<PROJECT> as needed. Please make sure to use your NCI login id as your top directory name.

All data copied from w35 will be stored under: /g/data/w97/W35_GDATA_MOVED and /scratch/w97/W35_SCRATCH_MOVED. Make sure to move your data into your proper working directory under /g/data and /scratch before working with it.

= Climate model experiments =

*[[Migrate_payu_experiment_project|Payu experiment]]
*[[Migrate_umui_experiment_project|UMUI experiment]]
*[[Migrate_rose_experiment_project|Rose/Cylc experiment]]
*[[Migrate_NUWRF_experiment_project|NUWRF experiment]]

WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.

 

= Tips for other scripts =

In your script, you may have used w35 or w48 in the following lines:

*PBS directive -P to specify the project
*PBS directive -l storage to load the storage space for these projects.
*Full paths of input or output data

It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.

It is important for you to make sure you know what changes you are doing, that's why we do not provide a magic <code>sed</code> one-liner command to change all the project IDs in all your files at once.

== Find files with the project listed in ==

To find all the files that contain the mention of <samp>w35</samp>, you can use the following command:
<pre>grep --exclude=*.nc -irlI w35 <directory_name> > files_to_update.txt</pre>

<samp>files_to_update.txt </samp>will contain a list of files where <samp>w35</samp> appear in at least one line. The command will not search in binary files and netCDF files and you can exclude more files from the search (e.g. figures) with the <code>--exclude=</code> option. Replace <samp><directory_name></samp> with the directory you want to search through.

At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only these:
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt
</pre>

== Backup before modifications ==

You will want to create some sort of backup of your text files before changing them. <span style="color:#c0392b">DO NOT create copies. </span>We do not have heaps of free storage around to allow for it. Our recommended method is to use Git. If your files are not under the Git version control yet, here is a simple workflow that allows to record the state of your files before change. And some tips on comparing the modified and initial versions of the files.

Let's say, you want to change some files under a directory called <tt>my_scripts</tt>. You will first need to create a Git repository from that directory, record the current state of your files, do the changes you want to do, test the modified files work:
<pre># Create repository. Go in your my_scripts directory
git init

# Record current state
git add .
git commit -m "State at time of merge"

# Change your files as you require, test the new files</pre>

If something does not work, you can then use the following commands depending on what you want to do:
<pre># Difference between modified file and previous version
git diff <filename>

# Re-instate the previous version and delete the modified file
git checkout <filename></pre>

== Find all and replace all ==

All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality, including reviewing the changes before making the change, for your favourite text editor.

= MDSS data migration =

After w35 and w48 are deleted any data stored on the [[How_to_use_MDSS_tape_storage_at_NCI|MDSS tape storage system]] would be inaccessible. To allow continued access all data has been migrated to a directory in the corresponding project (w35 to w97, w48 to w40).
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><code>$ mdss -P w40 ls -ld W48_MASSDATA_MOVED<br/> drwxrws--- 22 root w40 411 Mar  2 10:33 W48_MASSDATA_MOVED<br/> $ mdss -P w97 ls -ld W35_MASSDATA_MOVED<br/> drwxrws--- 27 root w97 143360 Mar  2 11:47 W35_MASSDATA_MOVED</code></div>

Useful resources

2022-06-03T07:08:23Z

A.heerdegen: Added NCAr tutorial link from Claire

This place is to create a library of links to external resources that might be useful to the Centre's staff and students.

= Data services =

----

[https://earthdata.nasa.gov/ NASA Earthdata]: this is a portal to NASA datasets. The User Resources tab also have a list of recorded [https://earthdata.nasa.gov/user-resources/webinars-and-tutorials webinars] around the use of these datasets.

*<span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[https://giovanni.gsfc.nasa.gov/giovanni/%20GIOVANNI [1]] GIOVANNI] - GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure</span>
*<span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[https://dl.tpac.org.au/tpacportal/ TPAC] Climate and Ocean data portal</span>
*<span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[http://www.ardc.org.au/ The Australian Research Data Commons (ARDC, previously known as ANDS)]</span>
*<span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[http://researchdata.ands.org.au/ Research Data Australia]</span>
*<span style="background-color: #ffffff; color: #000000;">[http://cfconventions.org/ CF metadata conventions website]</span>
*<span style="background-color: #ffffff; color: #000000;">[http://www.bom.gov.au/climate/data/ BoM Climate data services]</span>
*<span style="background-color: #ffffff; color: #000000;">[https://terranova.org.au/ Terra Nova] - The Australian Climate Change Adaptation Information Hub</span>

= Tools =

----

<span style="background-color: #ffffff; line-height: 1.5;">[http://www.marvl.org.au/ MARVL - Marine virtual Laboratory] : </span><span style="font-family: Arial,Helvetica,sans-serif; line-height: 1.5;">The Australian Marine Virtual Laboratory (MARVL) is a new development in modelling frameworks for researchers in Australia, co-funded by NeCTAR ([http://www.nectar.org.au/ www.nectar.org.au]). MARVL allows a non-specialist user to configure and run a model, automating many of the modelling preparation steps needed to bring the researcher faster to the stage of simulation and analysis. The tool is seen as enhancing the efficiency of researchers and marine managers, and is being considered as a educational aid in teaching. Most researchers, students and academic staff with have access to MARVL via their [http://aaf.edu.au/ AAF](Australian Access Federation) identity, in most cases this is your normal login identity at your local university, institute or federal agency.</span>

<span style="font-family: Arial,Helvetica,sans-serif;">[http://www.unidata.ucar.edu/blogs/news/entry/metpy_an_open_source_python MetPy] : </span>MetPy is an Open Source project aimed at providing a Pythonic library for meteorological data analysis that meshes well with the rest of the scientific Python ecosystem.

= Tutorials and other tips and tricks =

----

[http://web.maths.unsw.edu.au/~oangelil/blog/blog2/page_blog2.html help to start]: a blog post written by Oliver Angélil, a PhD student at the CCRC. It goes through a few practical tips and tricks to help with what to know when starting working in the field of climate science.

CMS blog: [https://climate-cms.org https://climate-cms.org]

CMS training: [https://docs.google.com/presentation/d/13eCaZCLwh48-3MHg8ks5O3kXTc3Wz4WbkdB0ZqZnZZM/edit?usp=sharing Publishing]

CMS training: [https://docs.google.com/presentation/d/1EclbLy3x28F6u821P7_XJT7cfBQpanIo4BxSpJhHcAM/edit?usp=sharing Storage management] and a separate document with[https://docs.google.com/document/d/17gpAuinpf5DKVhCXABAwgtbwGBzmdgi1diJiZVHyZrc/edit?usp=sharing links for the training]

CMS training: [https://docs.google.com/presentation/d/1Bl1SI_v3s-kgqs96FYROO--JhDw_Trujb5TYnxhBaD8/edit#slide=id.g2739569a98_1_58 Shell tips and tricks]

CMS training: [https://docs.google.com/presentation/d/1Rvn7a8Y4ghz0SVprwyIN6KHrXqpOYvw2GMcdwFriFEk/edit Debugging 101]

CMS training: [https://docs.google.com/presentation/d/1n3b_gRNhb164MQNrFYv0oj12tUCqSnaWgSEzAZNWBL8/edit#slide=id.p Debugging - Run time checks]

CMS training: [https://github.com/ScottWales/training Xarray dask training GitHub]

CMS training: [http://%20https: ''youtu.be/ya-xXY5uzeo Introduction to 3D visualisation with ParaView'']''by Martin Jucker (youtube video)''

CMS training: [https://youtu.be/QXIj1nYwGcY Introduction to CleF: Climate Finder a tool to discover ESGF datasets] (youtube video)

CMS training: [https://www.youtube.com/watch?v=fHtLbMmgoMY&feature=youtu.be Python: transitioning from 2 to 3]

NCAR [https://ncar.github.io/python-tutorial/tutorials/yourfirst.html tutorial on python for atmospheric and oceanic scientists]

cheat sheets:

*[https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing Tape ARchive - tar]
*[https://docs.google.com/document/d/1pGgVAoEjErHTn4joiCXbdnZDppEXwq9RcNofedrViz4/edit?usp=sharing Access Control Lists - acls]
*[https://docs.google.com/document/d/1ugmBLX385YCuEj1Rd1MEyzC85cZEfE6sDzbfEYiv8ZM/edit?usp=sharing NCI accounting:] data#_report_files, short_report_files, lquota, ncimonitor and mdssdiff

= Professional development =

----

Data science: [https://www.dataquest.io Dataquest] and [https://www.dataquest.io/blog its blog].

Gadi

2022-06-01T06:37:00Z

A.heerdegen: Added explanation of copying and moving

= Gadi processors =

Gadi is NCI's primary supercomputer since January 2020. The supercomputer is composed of an assortment of processors, the majority of those are Cascade Lake processors. Different queues give access to different processors and have a different charging rate

{| class="wikitable"
|-
! Queue
! Memory
! Priority
! Charging rate per walltime-hour
! CPU per node
! Processor type
|-
| normal
| 192 GB
| Normal
| 2SU
| 48
| Cascade Lake (CL), 3200 nodes
|-
| normalbw
| 256 GB
| Normal
| 1.25SU
| 28
| Broadwell (BW), 800 nodes
|-
| express
| 192 GB
| High
| 6SU
| 48
| Cascade Lake (CL), 3200 nodes
|-
| copyq
| 192 GB
| Normal
| 2SU
| 1 cpu jobs only
| Cascade Lake (CL), 3200 nodes
|-
| gpuvolta
| 340 GB
| Normal
| 3SU
| 48CPUs, 4 GPU
| 640 Nvidia V100 GPUs, 160 nodes
|}

Full [https://opus.nci.org.au/display/Help/Queue+Limits details of all available queues are available from the NCI help pages].

= Getting an account =

To get a new account at NCI, you will need to get connected to a NCI project. Before you start the process, talk to your CI or supervisor to know which project code to use. You will need to apply via [https://my.nci.org.au/ my.nci.org.au]. NCI will send you a password via SMS once your application has been processed, this usually takes under a day to do.

Once you have an account, [https://my.nci.org.au/mancini/login?next=/mancini/ my.nci.org.au] will allow you to ask for membership of other projects you might need. Those could be projects for additional compute time or projects to access data etc.

= Connecting to Gadi =

To connect to Gadi, you'll need to use a SSH connection to <span style="font-family:monospace">gadi.nci.org.au</span>. If you're using Windows, you'll need to use something like [http://www.putty.org/ PuTTY], or if you're connecting from linux or mac run on the commandline (substitute <span style="font-family:monospace">abc123</span> with your own username)
<syntaxhighlight lang="bash">ssh -Y abc123@gadi.nci.org.au
</syntaxhighlight>

You can make a shortcut for this by editing (or creating) the file <span style="font-family:monospace">~/.ssh/config</span> and adding the lines:
<syntaxhighlight lang="text">
Host gadi
HostName gadi.nci.org.au
User abc123
ForwardX11 true
ForwardX11Trusted true
</syntaxhighlight>
This way you just need to type '<span style="font-family:monospace">ssh gadi</span>' to connect.
= Swapping Projects =

If you use more than one project you can swap between them with the command '<span style="font-family:monospace">switchproj</span>', e.g.
<syntaxhighlight lang="text">switchproj w35
</syntaxhighlight>
will start a new shell and change your current project to w35. This is useful if you want to interactively create files that will be owned by a specific project. You can also change your default project by editing the file on Gadi <span style="font-family:monospace">~/.config/gadi-login.conf</span>, it should have a line like <syntaxhighlight lang="text">PROJECT w35
</syntaxhighlight>
You simply need to change the project code to the one you'd like. Then you need to log out and back in for the change to take effect. In the <span style="font-family:monospace">~/.config/gadi-login.conf</span> file, you can also change your default shell. That is the active shell when you log into Gadi.
= Resources on Gadi =

To see how much compute time you have available run the command
<syntaxhighlight lang="text">nci_account -P $PROJECT -q 2013.q3
</syntaxhighlight>

To see how much storage space you have available run the command
<syntaxhighlight lang="text">lquota -P $PROJECT
</syntaxhighlight>

For more information see [[Accounting_at_NCI|Accounting_at_NCI]].

 

= Submitting Jobs =

To run a job on the supercomputer you submit it to a job queue using the '<span style="font-family:monospace">qsub</span>' command. Jobs are shell script files, they contain special markers to say what resources the job needs.

As an example the script "hello.sh"
<syntaxhighlight lang="text">#!/bin/bash
#PBS -l ncpus=2
#PBS -l walltime=10:00
#PBS -l mem=1gb
#PBS -v PROJECT

echo "Hello"
</syntaxhighlight>

says to run with 2 cpus for a maximum time of 10 minutes. The job can use up to 1 GB of memory. Anything after the #PBS lines is what gets run on the supercomputer, in this instance it just prints "Hello" (any output goes to files in the directory you submitted the job named like "hello.sh.o123456", error messages go to files named like "hello.sh.e123456). The command '-v PROJECT' means run using the current project, you can also specify a project to use like '-v PROJECT=w35'. If the job tries to use more resources than it's asked for it will be automatically stopped. The less resources you ask for the more likely it is that your job will run quickly however, you should try to request an amount close to what the job actually uses.

See the [https://opus.nci.org.au/display/Help/How+to+submit+a+job NCI PBS documentation] for more detail.

= Managing Jobs =

To see a list of your submitted & currently running jobs run
<syntaxhighlight lang="text">nqstat
</syntaxhighlight>
This also shows how much resources each job has requested & is currently using. Each job in the queue has a run id number associated with it (this is also printed when you submit a job with <span style="font-family:monospace">qsub</span>). To get more information on a job run <syntaxhighlight lang="text">qstat -s 123456 # Show any status information, e.g. why the job isn't currently running
qstat -f 123456 # Show full information, including resources requested & environment variables
</syntaxhighlight>
To remove a job from the queue use <span style="font-family:monospace">qdel</span> <syntaxhighlight lang="text">qdel 123456 # Remove the job 123456 from the queue
</syntaxhighlight>

= Copying and moving files =

In general the project code is retained when moving files from one filesystem to another om gadi (using the <tt>mv</tt> command). If you instead copy the files (using <tt>cp</tt>) then the project code is changed to be the same as the project for the directory they are being copied to if the <tt>setgid</tt> bit is set, which it often is, e.g.
<pre>$ ls -ld /scratch/v45
drwxrws--- 170 root v45 16384 May 20 20:16 '''/scratch/v45'''</pre>

the <tt>s</tt> in the group permissions indicates the setgid bit is set, which means any files '''created''' there will have the group <tt>v45</tt>, rather than your default project code, and copying creates a new file. When using <tt>mv</tt> across filesystems it will effectively do a copy and on success change the attributes to match the original and then delete the original. <tt>rsync</tt> will also do something similar depending on the options used.

See here for more details on the setgid bit<br/> [https://linuxconfig.org/how-to-use-special-permissions-the-setuid-setgid-and-sticky-bits https://linuxconfig.org/how-to-use-special-permissions-the-setuid-setgid-and-sticky-bits]

[[Category:NCI]]

 

Tips: Custom file permissions at creation

2022-05-31T04:14:53Z

A.heerdegen:

This page provides information on how to create files with group read permissions at NCI.

By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.

== PBS logs ==

By default, all log files from PBS are only readable by the user owning the file. You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:
<syntaxhighlight lang="text">#PBS -W umask=0022
</syntaxhighlight>

You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.

For PBS log files that have already been created you will need to do the following:
<pre>chmod g+r <filename></pre>

== HOME directory permissions ==

By default your home directory is not accessible to anyone but you. If you need others to access files in your home directory you will need to give them access. If they are members of the same group. To find your home directory permissions and group:
<syntaxhighlight lang="text">$ ls -ld $HOME
drwx------ 36 usr123 g00 20480 May 26 09:17 /home/001/usr123
</syntaxhighlight>

If the other user you wish to give permissions to is also a member of the same group (in this case g00) then use this command:
<syntaxhighlight lang="text">$ chmod g+rx $HOME
</syntaxhighlight>

Then the permissions will reflect the change:
<syntaxhighlight lang="text">$ ls -ld $HOME
drwxr-x--- 36 usr123 g00 20480 May 26 09:17 /home/001/usr123
</syntaxhighlight>

If the other user is not a member of the correct group, they can either join the project in question, or you can use [[Useful_resources|Access Control Lists (ACLs)]].

== Files created by programs ==

Some programs might impose a different default of the files they create which might not be what you'd like.

=== You have access to the source code ===

If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact [mailto:cws_help@nci.org.au the CMS team] if you need help.

=== You don't have access to the source code ===

You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the <code>chmod</code> command:

chmod g+r <filename>

This command can be run recursively through a directory tree:

chmod -R g+r <directory_name>

=== Files owned by the wrong group ===

If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. '''Note:''' NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. '''Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.'''

To change the group ownership of a file:

chgrp <newgroup> <filename>

Or recursively through a directory tree:

chgrp -R <newgroup> <directory_name>

[[Category:Data]]

Tips: Custom file permissions at creation

2022-05-25T23:45:50Z

A.heerdegen: Add explicit instructions for home directories

This page provides information on how to create files with group read permissions at NCI.

By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.

== PBS logs ==

By default, all log files from PBS are only readable by the user owning the file. You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:
<syntaxhighlight lang="text">#PBS -W umask=0022
</syntaxhighlight>

You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.

== HOME directory permissions ==

By default your home directory is not accessible to anyone but you. If you need others to access files in your home directory you will need to give them access. If they are members of the same group. To find your home directory permissions and group:
<syntaxhighlight lang="text">$ ls -ld $HOME
drwx------ 36 usr123 g00 20480 May 26 09:17 /home/001/usr123
</syntaxhighlight>

If the other user you wish to give permissions to is also a member of the same group (in this case g00) then use this command:
<syntaxhighlight lang="text">$ chmod g+rx $HOME
</syntaxhighlight>

Then the permissions will reflect the change:
<syntaxhighlight lang="text">$ ls -ld $HOME
drwxr-x--- 36 usr123 g00 20480 May 26 09:17 /home/001/usr123
</syntaxhighlight>

If the other user is not a member of the correct group, they can either join the project in question, or you can use [[Useful_resources|Access Control Lists (ACLs)]].

== Files created by programs ==

Some programs might impose a different default of the files they create which might not be what you'd like.

=== You have access to the source code ===

If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact [mailto:cws_help@nci.org.au the CMS team] if you need help.

=== You don't have access to the source code ===

You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the <code>chmod</code> command:

chmod g+r <filename>

This command can be run recursively through a directory tree:

chmod -R g+r <directory_name>

=== Files owned by the wrong group ===

If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. '''Note:''' NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. '''Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.'''

To change the group ownership of a file:

chgrp <newgroup> <filename>

Or recursively through a directory tree:

chgrp -R <newgroup> <directory_name>

[[Category:Data]]

Getting Help

2022-05-10T01:33:52Z

A.heerdegen:

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left- hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences. Simply outline your issue in an email and send it to [mailto:cws_help@nci.org.au cws_help@nci.org.au] which will open a help ticket on our helpdesk.

'''Note: '''We have [[Updated_support_level|updated our user support levels]] due to staffing issues.

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Follow the [[Troubleshooting|troubleshooting guide]]
*Have you read the documentation on this wiki (if it exists), any other relevant documentation, or looked on [https://climate-cms.org our blog]?
*Do you need to do some [[Training|training]] to improve your skills?
*Include all relevant information:
**what have you tried already that hasn't worked?
**provide some background for why you are doing the task that isn't working. Sometimes we might be able to suggest a different approach rather than trying to solve your specific error
**what machine are you running on: your machine (what OS), Gadi at NCI, OOD at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working on Gadi was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. In particular if this is an error with the conda environment we support please include which version of the environment you are using. Tell us if you load modules in your .bashrc or .bash_profile files (not recommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

 

== What can users expect ==

Users can expect:

*to receive a reply to a query within 24 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

CLEx induction

2022-05-09T04:37:56Z

A.heerdegen: Removed Claire and Scott

The CMS team is here to help you with the technical aspects of your work. A lot of you (probably all) will have to work with computer code and software, usually in a Linux/Unix environment. You might also need to access and/or share large amounts of data, work on different servers, etc.

Some of you will be somewhat familiar with these tasks whereas they will be new to others. We are here to help whatever your technical knowledge is. We are happy to answer any question, we can't necessarily do everything but feel free to ask as we might have ideas or we might know who you should contact and what to exactly ask.

The members of the [[CMS|CMS]] team are:

{| class="wikitable"
|-
! Name
! Sitting at
! Email
! CMS "local" support person for
|-
| Aidan Heerdegen
| ANU
| aidan.heerdegen@anu.edu.au
| ANU
|-
| Paola Petrelli
| U Tasmania
| paola.petrelli@utas.edu.au
| U Tasmania
|-
| Holger Wolff
| Monash
| holger.wolff@monash.edu
| Monash
|-
| Ramzi Kutteh
| UNSW
| r.kutteh@unsw.edu.au
| UNSW
|}

The "local" support person will be the CMS person welcoming you to the Centre and possibly giving you an induction. But you are more than welcome to contact any of the CMS team members for support.

Although we won't ignore emails sent to our individual addresses, our preferred way of contact is through our help desk email address: [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Contacting the help desk will make sure your email is seen by one of us in case the person at your institution is away for example. Don't worry about the help desk email being an NCI account, you can still ask any question there.

[[Getting_Help|Here]] are some guidelines on formulating your requests to CMS

 

= Set up your computer =

We have put instructions for the setup we recommend for each operating system on the following pages:

*[[Getting_started_on_Windows|Windows]]
*[[Getting_started_on_Mac|Mac]]
*[[Getting_started_on_Linux|Linux]]

= Step 1: Get an account with NCI =

NCI operates the computers that you will be doing most of your work on. NCI provides services to the Centre, they are not part of the Centre of Excellence.

To get access to NCI servers, you need to do three things:

1. You need to find out which project(s) you need to be connected to.

Your project decides who gets billed for what you do on the NCI Servers. Your supervisor should be able to tell you which project(s) you should get connected to. Be mindful that there are two types of projects: projects for computation and projects for datasets. The mapping of the computational projects for CLEX can be found [[CLEX_projects_at_NCI|here]]. We recommend to join all the projects relevant to your research project but to choose 1 default project to work from. The same page with the list of computational projects explains how to set your default project. The projects for datasets do not have any computational time added to them so you need to make sure you ask for at least one computation project. Your supervisor should know. Note if you will run the ACCESS model or the UM model, you need to ask a connection to the "access" project in addition to at least one computation project.

2. You need to register with NCI.

This can be done on [https://my.nci.org.au/mancini/signup/0 NCI registration page] You will need to supply some information, and read and accept their policies. In order to be added to the system, you need to request connection to at least one project and be accepted on the project. Note that this can take a bit of time until the Lead chief investigator of the project has received and confirmed your request via email. Note that this is an automatic email process, you don't need to send an email to the lead chief investigator. Although, you might want to do so if you think you need to introduce yourself to him/her. Note that you should be able to request connection to several projects while getting signed up.

3. You might want to ask your supervisor whether you also should be connected to any of the following data projects.

These don’t provide any computing allowances, but they give access to certain datasets or models that you might need.

{| class="wikitable"
|-
! Project
! Purpose
! Administrator
|-
| '''ua8'''
|  GSWP3, NCEP Polar sst, OISST, CMIP5 ocean processing, model ensemble data + others
| CMS
|-
| '''rq7'''
| [[YOTC|ECMWF Year of Tropical Convection re-analysis]]
| CMS
|-
| '''ks32'''
| ARCCSS and CLEX published data
| CMS
|-
| '''ub4'''
| [[ERA_INTERIM|ECMWF ERA Interim re-analysis 6 hrs data]] and
| CMS
|-
| '''cb20'''
| [[CMIP|CMIP3]] replica data
| NCI
|-
| '''al33'''
| [[CMIP|CMIP5]] replica data
| NCI
|-
| '''rr3'''
| [[CMIP|CMIP5]] published data
| NCI
|-
| '''oi10'''
| [[CMIP|CMIP6]] replica data
| NCI
|-
| '''fs38'''
| [[CMIP|CMIP6]] published data
| NCI
|-
| '''rr7'''
| Use only for JRA55, MERRA2 the other datasets have been removed
| BoM
|-
| '''cable'''
| Give access to the [https://trac.nci.org.au/trac/cable/wiki CABLE model] and [[CABLE_Data|CABLE benchmarking dataset]]. Please read the [https://trac.nci.org.au/trac/cable/wiki/CableRegistration CABLE registration] information.
| CSIRO
|-
| '''access'''
| Give access to the [[Unified_Model|Unified Model]] and ACCESS tools
| CMS - BoM - CSIRO
|-
| '''rt52'''
| [[ERA5|ECMWF ERA5 re-analysis]] single and pressure levels
| NCI
|-
| '''zz93'''
| [[ERA5|ECMWF ERA5 re-analysis]] Land data
| NCI
|-
| '''sx70'''
| Shared input data for WRF
| CMS
|-
| '''ik11'''
| Shared data for COSIMA. Inputs and outputs
| COSIMA
|-
| '''ia89'''
| 400m resolution ACCESS limited-area simulation over Australia
| CMS
|-
| '''gy85'''
| Himawari-8 GeoCat 1.0.3 Australian Domain
| CMS
|}

 

= Step 2: Set up your Connection =

The standard method of connecting to the NCI systems is SSH. We want you to use a passphrase protected private key authorisation method with an ssh-agent and agent forwarding for convenience. If you understood every part of that sentence, go ahead and set it up. If not, the next part describes what to do.

The instructions describe how to set up your connection to both gadi.nci.org.au and accessdev.nci.org.au. You '''can not''' set up a connection to accessdev if you are not going to use the ACCESS or UM models. So please only follow the instructions you need!

 

== Linux and MacOS: ==

#'''Create the ssh directory''' on your computer <syntaxhighlight lang="bash">$ mkdir -p ~/.ssh</syntaxhighlight>

 

#'''Create a config file''' <syntaxhighlight lang="bash">$ nano ~/.ssh/config </syntaxhighlight>

Enter the following contents, replacing <span style="font-family:monospace">xx0000</span> with your NCI user name.
<syntaxhighlight lang="text">
Host gadi gadi.nci.org.au
HostName gadi.nci.org.au
User xx0000
ForwardX11 yes
ForwardX11Trusted yes
Host access accessdev.nci.org.au
HostName accessdev.nci.org.au
User xx0000
ForwardX11 yes
ForwardX11Trusted yes
ForwardAgent yes
</syntaxhighlight>
If you use a Mac, and you want to store the key in your keychain so that it gets automatically loaded when you log in, you can add this section to your config file: <syntaxhighlight lang="text">
Host *
AddKeysToAgent yes
UseKeychain yes
</syntaxhighlight>

#'''Create a key pair.''' It is imperative that you select a strong passphrase when asked for it. <syntaxhighlight lang="bash">$ ssh-keygen -t rsa</syntaxhighlight>

 

#'''Distribute the key pair.''' On most systems, there is a useful script to distribute the key pairs called '''ssh-copy-id''': <syntaxhighlight lang="bash">
$ ssh-copy-id gadi
$ ssh-copy-id access
</syntaxhighlight>

If that script fails, you have to distribute the keys manually with these commands:
<syntaxhighlight lang="bash">
$ cat ~/.ssh/id_rsa.pub | ssh gadi "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"
$ cat ~/.ssh/id_rsa.pub | ssh access "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"
</syntaxhighlight>

#'''Test whether you have an agent.''' If you have an agent already running, this command will ask for the passphrase for the just-created key pair: <syntaxhighlight lang="bash">$ ssh-add</syntaxhighlight>

If the last command told you that it couldn’t open a connection, then you don’t have an agent. Come talk to someone of the CMS team, we will help you set it up.

 

== Windows ==

Since Windows has a different heritage to Mac and Linux some extra setup work is required to use SSH. See the [[Getting_started_on_Windows|Getting started on Windows]] page for details.

If you're using [[Getting_started_on_Windows#Windows_Subsystem_for_Linux_.28WSL.29|WSL]] or [[Getting_started_on_Windows#Cygwin|Cygwin]] use the [[#Linux_and_MacOS:|SSH instructions for Mac and Linux]] above.

If using Power Shell, generate your ssh key with (Power Shell doesn't support 'rsa' keys)
<syntaxhighlight lang="unknown">
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa
</syntaxhighlight>
The contents of C:\Users\USER\.ssh\id_ecdsa.pub then need to be copied into ~/.ssh/authorized_keys on Gadi manually, as Power Shell doesn't have the ssh-copy-id command. The Power Shell SSH config file is located in your user directory, e.g. C:\Users\USER\.ssh\config
== SSH-keys for file transfers ==

Additionally, some groups transfer data from NCI to outside NCI, for example for storage on University maintained servers. For these transfers, the best way to proceed is to create a restricted ssh key pair used only for file transfer. The setup is explained in [https://opus.nci.org.au/display/Help/Using+SSH+keys Using restricted commands for transferring files]. Most of the instructions are still valid, except for step 3. The restricted command prefix should now be:
<syntaxhighlight lang="text">
from="gadi-dm*.nci.org.au,gadi*.nci.org.au",command="~/bin/rrsync /data/archive",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa
</syntaxhighlight>

= Step 3: Basics of the NCI system =

<span style="line-height: 1.5;">We are mainly using 3 systems of the NCI: gadi, accessdev and VDI. Gadi is the supercomputer that we run our models on. Accessdev is a virtual server that we have software installed that are required to allow the running the models. VDI is a virtual desktop designed for interactive analysis of your results. All these servers run Linux as operating system.</span>

== Read the manual! ==

Considering you will be using systems that are provided by the NCI, you should take the time to read the documentation provided by them about their systems. That is everything on their [https://opus.nci.org.au/display/Help/NCI+Help help knowledge base].

== Additional notes ==

=== Modules: ===

The servers have a lot of software installed, some of which they have multiple versions of. If you need to use a software, you usually will need to load a certain module. A full help of the module command is accessible through the man pages on gadi. Here are the most important commands regarding the modules:

{|
|-
| $ module <span style="color: #8000ff;">help</span>
| Show help about the module system
|-
| $ module <span style="color: #8000ff;">avail</span>
| List all available modules (a lot!)
|-
| $ module <span style="color: #8000ff;">list</span>
| List the currently loaded modules
|-
| $ module <span style="color: #8000ff;">load</span> intel-compiler
| Load the default version of the intel compiler
|-
| $ module <span style="color: #8000ff;">load</span> intel-compiler/2019.3.199
| Load a specific version of the intel compiler
|-
| $ module <span style="color: #8000ff;">switch</span> intel-compiler/2019.3.199
| Switch to a different version of the intel compiler (unload and load)
|-
| $ module <span style="color: #8000ff;">unload</span> intel-compiler/2019.3.199
| Unload the specified version of the intel compiler
|-
| $ module <span style="color: #8000ff;">use</span> /g/data/hh5/public/modules
| Add our Centre's curated modules to the list of available modules
|}

Some modules conflict with one another, for example different versions of the same software. You might have to unload one module first to load a different one.

=== PBS: ===

The other important system you need to understand in order to work on gadi is the scheduler PBS. This is the system for running software on dedicated nodes inside gadi. Basically, a user tells PBS that they want to run this job on this many CPUs with these resources for at least so long, and PBS places it in a queue, and when the required resources are available, it will assign this job to them.

Again, NCI has a very good documentation about how to run this [https://opus.nci.org.au/display/Help/How+to+submit+a+job here]

The most important commands are:

*'''qsub''' – Submit a job to the queue
*'''qstat''' – Display the status of the queued jobs
*'''qdel''' – Remove a job from the queue

Of these, qsub is the most important, and there are so many parameters that we won’t tell them all here.

In general, the fewer resources you request, the earlier your job will run. But if your program exceeds the requested resources at any time, PBS will kill the job, and you have to start over.

More specifically:

*Gadi is comprised of nodes with 48 CPUs each. If you request less than 48 CPUs, you will get the number of CPUs that you request. If you request more, you will need to use a multiple of 48 CPUs.
*Nodes come with 192GB of memory to share within the 48 CPUs. So if your job requires 48 or more CPUs, you might as well request all the memory, for example 384GB for a 96-cpu job.

Running a job on the queue costs so-called '''SU'''. On the old raijin, one SU was equivalent to one core for one hour. On the much faster new gadi, one core for one hour costs 2 SU. Note that on gadi, the SU cost is calculated using the higher of the number of cores and the memory usage. So if you request 5 cores but only 2GB memory, its SU cost is based on the 5 cores. If you request 1 core but 96GB of memory (half of the whole node), the cost will be based on 24 cores (half the whole node).

= Step 4: How to run models =

A lot of information to gain access to models or run the models can be found on the ARCCSS CMS wiki (see link at the end).

A very important message from the CMS team: <span style="background-color: #fdf988; display: block; text-align: center;">Often you will work with source code. Please ensure that your version of the source code is under version control! We use SVN and git (usually hosted on github) for the model codes and smaller utilities.</span><span style="background-color: #fdf988; display: block; text-align: center;">'''DO NOT JUST COPY SOURCE CODE! CHECK IT OUT, OR CHECK IT IN!'''</span>

== UM ==

The Unified model is currently transitioning from the old user environment umui to the new rose/cylc user interface. Either way, you need to log into accessdev.nci.org.au.

=== umui(x) ===

The old user interface has to be used for versions up to 7.x, and can be used for 8.x. umui was developed by the UM, umuix is very similar, but more convenient developed by CSIRO. An introduction into its use is on our wiki: [http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI]

=== rose/cylc ===

The new user interface can be used from version 8.x, and has to be used for versions 9.0 and later.

== WRF ==

The CMS team ports the WRF model at NCI. Information on using the model is on [[WRF|the CMS wiki]]. Users should also be aware that we do not reproduce the information from the [http://www2.mmm.ucar.edu/wrf/users/ WRF model users website] which stays a very important resource for all users. In particular, every new user is strongly encouraged to run through the official tutorials (found under User Support on the WRF model users site). As found on the WRF pages on this wiki, only the compilation steps differ from the NCAR tutorial.

== CABLE ==

CABLE is under licence so you need to request access to the code. All the information you need to get access and use CABLE is on the [https://trac.nci.org.au/trac/cable CABLE trac site]

= Data management =

Most of the information on how the data is managed is stored in this wiki in the [[Data|Data Services]] page. Here you can find information on the datasets managed by the Center as well the relevant policies in regard to data. Take some time to get familiar with it, here are some suggestions:

#We now offer a 1-day Data Induction training, if you have an opportunity to participate do so, particularly if you are just starting. If you missed out, all the [[Data_induction|course materials]] are stored on the wiki and we tried to create a path for you touching all the relevant wiki pages. There are 3 parts to the data induction, the first covers the policies, the second focuses on publishing and the third the data and code management best practices. 
#We strongly suggest to have a [[Data_Management_Plan|data management plan]] from the start of your project. You can use the [https://clex.dmponline.cloud.edu.au/ CLEX DMPonline tool] to do this. Because this tool stores your plan into a database, you can save it and access it as many times as you want, which means you can build your data management plan as you progress with your research. It is good though to start as soon as you can, DMPonline is structured as to teach about data practices and the resources available to you, it also informs the CMS team on which data, software and/or infrastructure you are planning to use, so we can better tailor our efforts to your needs.
#Get familiar with [[Publishing_options|publishing options]], while this might seem faraway, it is a good idea to keep track from now of what you are doing in regard to data and metadata, as it is often much more difficult to retrieve this information at the end of your project. There are also some technical times involved in getting data properly published.
#Consider creating a [[Persistent_research_identifiers|researcher-ID]] to uniquely identify your work.
#Make sure you browse through all the available data sources before you download data, this wiki page cover [[Finding_datasets|how to find out if a dataset]] is available at NCI. You could be wasting time and storage downloading something which might be already available, or that we could download and manage for you
#Become familiar with the different filesystems available on any server you are using, make sure you know which are meant to be working spaces, where you can archive your data, how to share data with collaborators and best practices to transfer data within and out of the system. If you are using gadi on NCI you can find this information on their [http://nci.org.au/nci-systems/national-facility/data-storage/filesystems/ user support ]and training sections.
#You can explore a lot of data related information in the [https://ardc.edu.au Australian Research Data Commons] website
#Don't hesitate in contacting cws_help<at>nci.org.au for any question you might have, or also e-mail directly paola.petrelli<at>utas.edu.au, if you have a more general question or you are confused about data management. We are always available to discuss with you the details of your project and give you advice.

= How to learn more =

== General setup: ==

a blog post by a UNSW's PhD student on a few things to know and learn: [http://web.maths.unsw.edu.au/~oangelil/blog/blog2/page_blog2.html Oliver Angélil's post]

== CMS: ==

*E-Mail: [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*Chat with us on [https://arccss.slack.com Slack] (Use your university email in your sign up.)
*This [[Home|Wiki]]
*In person.

== NCI: ==

[https://opus.nci.org.au/display/Help/NCI+Help Guides and reference pages]

[http://nci.org.au/services-support/training/ Training material] and [https://training.nci.org.au/ this]

== Version Control with git: ==

(Note: git might also be very useful for your thesis!) [https://www.youtube.com/playlist?list=PLg7s6cbtAD165JTRsXh8ofwRw0PqUnkVH Git video tutorial by GitHub]

== Data management: ==

[[Data_induction|Data induction]]

= Getting help =

Your first port of call for help should be the CMS team via the help-desk: [mailto:cws_help@nci.org.au cws_help@nci.org.au]. This help desk is followed by all the members of the CMS team and by some staff from NCI. [[Getting_Help|Here]] are some guidelines on formulating your request

[[Category:Training]]

Getting Help

2022-05-05T23:54:42Z

A.heerdegen:

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left- hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences. Simply outline your issue in an email and send it to [mailto:cws_help@nci.org.au cws_help@nci.org.au] which will open a help ticket on our helpdesk.

'''Note: '''We have [[Updated_support_level|updated our user support levels]] due to staffing issues.

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Follow the [[Troubleshooting|troubleshooting guide]]
*Have you read the documentation on this wiki (if it exists), any other relevant documentation, or looked on [https://climate-cms.org our blog]?
*Do you need to do some [[Training|training]] to improve your skills?
*Include all relevant information:
**what have you tried already that hasn't worked?
**provide some background for why you are doing the task that isn't working. Sometimes we might be able to suggest a different approach rather than trying to solve your specific error
**what machine are you running on: your machine (what OS), Gadi at NCI, OOD at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working on Gadi was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. Tell us if you load modules in your .bashrc or .bash_profile files (unrecommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

 

== What can users expect ==

Users can expect:

*to receive a reply to a query within 24 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

Home

2022-05-05T23:52:57Z

A.heerdegen: Added Updated user support levels

[[File:CLEX-ARC-logo.png|center|800px|CLEX-ARC-logo.png]]

 

'''CLEX [[CMS|CMS]]''' is a team providing data and computational support to researchers at the ARC Centre of Excellence for Climate Extremes

'''<span style="font-size:larger;">[[Updated_support_level|Updated user support levels]]</span>'''

<big>[[Scratch_file_expiry|'''Scratch automatic file deletion''']]</big>

<big>'''Working from home: [[Working_from_home|some advice on how to setup]]'''</big>

<big>'''NCI project's migration: [[Migrate_experiment_projects|modifications to apply after the projects' migration]]'''</big>

{| class="wikitable" style="width: 100%"
|-
| colspan="2" style="text-align:center; background:skyblue" | '''Our Services'''
|- style="vertical-align:top"
| style="width: 50%" |
*[[ARCCSS_induction|Induction]]
*[[CMS#Our_People|Local Support]]
*[[FAQs|Checking the FAQs]]
*[[Getting_Help|Getting Help from CMS]]
*[[Training|Training]]
*[mailto:cws_help@nci.org.au Help Desk: cws_help@nci.org.au]
*[https://arccss.slack.com/ Slack]
*[[One-on-one_calls|Video meetings with CMS]]

| style="width: 50%" |
*[[News|News]]
*[https://coecms.github.io/ Blog]
*[https://www.youtube.com/channel/UCSmoK6oWV9O0Hmyt9UdDNsQ Youtube]
*[https://github.com/coecms Github]
*[https://anaconda.org/coecms Conda Packages]

|}

{| class="wikitable" style="width: 100%"
|-
| colspan="2" style="text-align:center; background:skyblue" | '''NCI'''
|- style="vertical-align:top"
| style="width: 50%" |
[[Accounting_at_NCI|Accounting at NCI]]

| style="width: 50%" |  
|}

{| class="wikitable" style="width: 100%"
|-
| colspan="2" style="text-align:center; background:skyblue" | '''Data'''
|- style="vertical-align:top"
| style="width: 50%" |
[[Finding_datasets|Finding datasets]]

*Common Datasets
**[[CMIP|CMIP]]
**[[ERA5|ERA5]]
**[[ERA_INTERIM|ERA Interim]]
*'''[[:Category:Dataset|All Datasets]]'''
*[[CLEX_and_ARCCSS_data_collection|Data published by ARCCSS and CLEX]]
*[https://geonetwork.nci.org.au NCI Data Catalogue]

| style="width: 50%" |
Data Management

*[[Data_induction|An overview]]
*[[CLEX_Data_policy|CLEX Data policy]]
*[[Publishing_options|How to Publish Data]]
*[[Persistent_research_identifiers|Persistent research identifiers]]
*[[Publisher_policies|Journal requirements]]
*[[Leaving_the_Centre_guidelines|Leaving the Centre guidelines]]
*[[Storage|Storage]]

[[Data|All data resources]]

|}

{| class="wikitable" style="width: 100%"
|-
| colspan="2" style="text-align:center; background:skyblue" | '''Models'''
|- style="vertical-align:top"
| style="width: 50%" |
[[ACCESS|ACCESS]] (Coupled)

*[https://accessdev.nci.org.au/trac/wiki/GettingConnected Getting Started with ACCESS]
*[[ACCESS#Model_Versions|CMIP Experiments]]
*[[:Category:ACCESS|Other ACCESS Configurations]]

| style="width: 50%" |
[[MOM|MOM]] (Ocean)

*[https://github.com/COSIMA/access-om2/wiki ACCESS-OM 2]
*[http://cosima.org.au COSIMA]

|- style="vertical-align:top"
| [[Unified_Model|Unified Model]] (Atmosphere)
*[[Unified_Model#Getting_Started|Getting Started with the UM]]
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM Documentation]
*[https://code.metoffice.gov.uk/trac/home MOSRS]
*[[Unified_Model#Supported_Experiments|Experiment Configurations]]

| [[CABLE|CABLE]] (Land)
|- style="vertical-align:top"
| [[WRF|WRF]] (Atmosphere)
| [[NU-WRF|NU-WRF]] (Atmosphere-Land)
|- style="vertical-align:top"
| [[:Category:Models|Other Models]]
|}

{| class="wikitable" style="width: 100%"
|-
| colspan="2" style="text-align:center; background:skyblue" | '''Resources'''
|- style="vertical-align:top"
| colspan="2" |
[[Glossary|Glossary]]

'''Tips:'''

*[[Tips:_Custom_file_permissions_at_creation|Custom file permissions]]
*[[VS_Code|Getting started with VS Code]]

'''CLEx information:'''<br/> [https://climateextremes.org.au CLEX - ARC Centre of Excellence for Climate Extremes]

[https://twitter.com/ClimateExtremes @ClimateExtremes]

'''Partners'''

*[https://nci.org.au NCI - National Computational Infrastructure]
*[https://ardc.edu.au/ ARDC - Australian Research Data Commons]
*[https://www.csiro.au/en/Research/OandA CSIRO]
*[https://bom.gov.au Bureau of Meterology]
*[https://www.gfdl.noaa.gov/ GFDL - Geophysical Fluid Dynamics Laboratory]
*[https://www.metoffice.gov.uk/ UK Met Office]

|}
<p style="text-align:right">[[:Category:Contents|Contents]] | [[:Category:Admin|Admin]]</p>
 

[[Category:Contents]]

Updated support level

2022-05-05T23:51:19Z

A.heerdegen: Fixed formatting

6/5/2022

= Context =

The CMS team is currently at a substantially reduced capacity. 

Consequently we cannot continue the same level of user support we have previously provided. The helpdesk ([[Cws_help@nci.org.au|cws_help@nci.org.au]]) and direct user support is an important and valued part of the role of the CMS, but we have other tasks and duties that require our attention. New CMS team members also require support and will take time to be as effective and efficient as those they are replacing. 

= What is changing? =

== '''Helpdesk''' ==

The helpdesk (cws_help@nci.org.au) will operate only 4 days a week (Mo/Tu/Th/Fr) and responses (not necessarily resolutions) to helpdesk queries will occur within 24 hours of submission on those days (up from the previous 2 hours).

== '''CodeBreak''' ==

CodeBreak sessions will run every Wednesday (up from the previous fortnightly frequency) as video sessions tend to be more efficient than email for dealing with some types of queries.

== '''Slack''' ==

Slack will no longer be actively monitored by the CMS team for user queries but feel free to keep using it to get help from other CLEXers.q

== '''Prioritization''' ==

Work on user requests will be prioritised more stringently. Please let us know if your request is urgent (e.g., you are close to finishing your PhD). Conversely, low priority help might not be actively worked on unless there are free resources. We will inform you if your support request has been classified as low priority.

 

= What can you do to help?  =

To get the most out of this new system

== '''Before asking for help''' ==

Consult the updated [http://climate-cms.wikis.unsw.edu.au/Getting_Help#Guidelines_for_help_requests guidelines for asking for help] on the wiki which emphasise some of the steps you need to take before submitting a help ticket, including consulting the FAQs and following relevant suggestions in our new [[Troubleshooting|troubleshooting guide.]] 

== '''Feedback'''  ==

Please tell us if you think the documentation on the [http://climate-cms.wikis.unsw.edu.au/Home Wiki] is insufficient, erroneous, or out of date. Your feedback is valuable and will improve this resource for everyone.

== '''Collaboration''' ==

Please answer any colleague's question on slack (or in person!) if you are able and confident to do so.

 

= How long will the changes last? =

The changes outlined above are likely to remain in place until the end of 2022, subject to how quickly we can recruit and train new CMS members. Note also that we will be without a CMS team leader until staffing issues have been resolved. 

For any request or information, please contact us through the helpdesk: cws_help@nci.org.au.<br/>  

Updated support level

2022-05-05T23:47:34Z

A.heerdegen: Created page with " 6/5/2022 = Context = The CMS team is currently at a substantially reduced capacity.  Consequently we cannot continue the same level of user support we have previously..."

6/5/2022

= Context =

The CMS team is currently at a substantially reduced capacity. 

Consequently we cannot continue the same level of user support we have previously provided. The helpdesk ([[cws_help@nci.org.au]]) and direct user support is an important and valued part of the role of the CMS, but we have other tasks and duties that require our attention. New CMS team members also require support and will take time to be as effective and efficient as those they are replacing. 

= What is changing? =

*
'''Helpdesk''': the helpdesk (cws_help@nci.org.au) will operate only 4 days a week (Mo/Tu/Th/Fr) and responses (not necessarily resolutions) to helpdesk queries will occur within 24 hours of submission on those days (up from the previous 2 hours).

*
'''CodeBreak''': the CodeBreak sessions will run every Wednesday (up from the previous fortnightly frequency) as video sessions tend to be more efficient than email for dealing with some types of queries.

*
'''Slack''': Slack will no longer be actively monitored by the CMS team for user queries but feel free to keep using it to get help from other CLEXers.

*
'''Prioritization''': we will prioritize work on user requests more stringently. Please let us know if your request is urgent (e.g., you are close to finishing your PhD). Conversely, low priority help might not be actively worked on unless there are free resources. We will inform you if your support request has been classified as low priority.

 
= What can you do to help?  =

To get the most out of this new system:

*
'''Before asking for help''': consult the updated [http://climate-cms.wikis.unsw.edu.au/Getting_Help#Guidelines_for_help_requests guidelines for asking for help] on the wiki which emphasise some of the steps you need to take before submitting a help ticket, including consulting the FAQs and our new troubleshooting guide. 

*
'''Feedback''': please tell us if you think the documentation on the [http://climate-cms.wikis.unsw.edu.au/Home Wiki] is insufficient, erroneous, or out of date. Your feedback is valuable and will improve this resource for everyone.

*
'''Collaboration''': Please answer any colleague's question on slack (or in person!) if you are able and confident to do so.

 
= How long will the changes last? =
 
The changes outlined above are likely to remain in place until the end of 2022, subject to how quickly we can recruit and train new CMS members. Note also that we will be without a CMS team leader until staffing issues have been resolved. 

For any request or information, please contact us through the helpdesk: cws_help@nci.org.au.
<br/>  

Getting Help

2022-05-05T23:42:57Z

A.heerdegen: Removed slack channel stuff

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left- hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences. Simply outline your issue in an email and send it to [mailto:cws_help@nci.org.au cws_help@nci.org.au] which will open a help ticket on our helpdesk.

 

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Follow the [[Troubleshooting|troubleshooting guide]]
*Have you read the documentation on this wiki (if it exists), any other relevant documentation, or looked on [https://climate-cms.org our blog]?
*Do you need to do some [[Training|training]] to improve your skills?
*Include all relevant information:
**what have you tried already that hasn't worked?
**provide some background for why you are doing the task that isn't working. Sometimes we might be able to suggest a different approach rather than trying to solve your specific error
**what machine are you running on: your machine (what OS), Gadi at NCI, OOD at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working on Gadi was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. Tell us if you load modules in your .bashrc or .bash_profile files (unrecommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

 

== What can users expect ==

Users can expect:

*to receive a reply to a query within 24 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

Accounting at NCI

2022-05-04T23:52:48Z

A.heerdegen:

This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by NCI and the CMS team.

= Grafana =

CMS has put in place a Grafana server for visualising a range of accounting statistics for CLEx. You can access this server using your NCI credentials at: [https://accessdev.nci.org.au/grafana/login https://accessdev.nci.org.au/grafana/login]

Unfortunately, we haven't yet put in place the collection of all the statistics for Gadi. Currently, you can see timeseries of your storage and SU usage on the [https://accessdev.nci.org.au/grafana/d/toeLAYDWz/user-report?orgId=1 User Report].

We will update this section as more statistics become available.

= Computing resources =

== nci_account ==

You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. If the project has a <tt>/g/data</tt> quota it will also report how much of this disk space is utilised. Using the <code>-v</code> option, you can see the usage per user:

nci_account -v

== nqstat ==

<code>qstat</code> allows you to monitor your jobs at NCI. <code>nqstat</code> allows you to see the jobs currently submitted:

*for a given project
*for a given queue and project
*for a given user and project

== uqstat ==

The information output by <code>qstat</code> and <code>nqstat</code> isn't always enough. CMS has developed <code>uqstat</code> to output more information by default such as the job efficiency (cpu%), the queueing time, the walltime, the cost in SU etc.

To use this command, please load the <code>nci-scripts</code> module:

module use /g/data/hh5/public/modules
module load nci-scripts

Contrary to what we usually recommend, it should be safe to load this module in your .bashrc file so this module is loaded by default.

uqstat

The <code>-x</code> option allows you to see the same information on current jobs and jobs finished up to 24 hours previously.

== qcost ==

CMS has also developed <code>qcost</code> a tool to help you choose the best PBS configuration for your job. <code>qcost </code>calculates the cost, in SU, for a job submitted to the PBS system. The PBS [https://opus.nci.org.au/display/Help/Queue+Limits queue information is provided by NCI] but it can be tedious to determine which configuration of queue and memory request should be used to minimise job cost. qcost was created to make this process easier.

To use this command, please load the <code>nci-scripts</code> module (see above). Usage:

qcost -h
 usage: qcost [-h] -q QUEUE -n NCPUS -m MEM [-t TIME]

Return the cost (in SUs) for a PBS job submitted on gadi. No checking is done to ensure requests are within queue limits.

optional arguments:
-h, --help show this help message and exit
-q QUEUE, --queue QUEUE PBS queue
-n NCPUS, --ncpus NCPUS Number of cpus
-m MEM, --mem MEM Requested memory
-t TIME, --time TIME Job walltime (hours)

e.g. qcost -q normal -n 4 -m 60GB -t 3:00:00

Note that if no time is specified it defaults to 1 hour (1:00:00). Walltime can be specified as <tt>H:M</tt> or <tt>H:M:S</tt>. Memory must be specified in units of bytes (B), e.g. <tt>160GB</tt>, <tt>2000MB</tt>. For example:
<pre> qcost -q normal -n 4 -m 60GB -t 3:00:00
90 SU

qcost -q express -n 4 -m 60GB -t 3:00:00
270 SU
</pre>

= Storage resources =

== lquota ==

This command will list the quota and usage for all the projects you are a member of on both /scratch and /g/data. It will give usage and quota for both space and number of files.

== nci-files-report ==

This command will list the usage in both space and number of files along different dimensions depending on options.

=== Usage owned by a project per user ===

If you want to know who owns files in your project or where those files are, you should use:

nci-files-report --group w35 --filesystem gdata

This is probably the most useful options for the project's managers to find who has the most data owned by a project.

=== Usage in a project directory per user ===

If you want to know who owns files in the main directory of one of your projects, you should use:

nci-files-report --project w35 --filesystem gdata

This is probably the least useful option for this function.

=== Usage per user ===

If you want to know your data footprint across all your projects in a filesystem, you should use:

nci-files-report --user --filesystem gdata

'''Why are the totals different in <code>nci-files-report</code> and Grafana?'''

CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project

== ncfind and nccompres ==

These utilities are maintained by the CMS team. These allow you to find uncompressed netcdf files and to easily compress them. You'll find a complete documentation on [[NetCDF_Compression_Tools|this wiki page]]

== du, find and wc ==

These commands can help you identify where in your area are the large files or which directories contain lots of files. These are Unix commands and all have man pages for information.

Useful options:

*Check the sizes of sub-directories:

du -shc *

*Find files larger than X

find . -type f -size +100M

*Count number of files and subdirectories in directory

ls -1 | wc -l

Note: the Internet is the perfect source for commands and options variations to find this type of information on your files and directories.

== ncdu ==

[https://dev.yorhel.nl/ncdu ncdu] is a disk usage analyzer with an ncurses interface. It is designed to find space hogs and aims to be fast, simple and easy to use.

This command is part of the conda environment maintained by CMS. To use:

module purge
module use /g/data3/hh5/public/modules
module load conda/analysis3

To invoke run ncdu directory_path and replace directory_path with a path to a directory you wish to check.

Some example output:

ncdu /g/data3/hh5/public/apps

Gives a screen items in the directory sorted by size:

--- /g/data3/hh5/public/apps ----------------------------------------------------
64.0 GiB [##########] /miniconda3
799.6 MiB [ ] /lrose
152.3 MiB [ ] /easybuild
4.0 KiB [ ] README

Total disk usage: 64.9 GiB Apparent size: 64.6 GiB Items: 2461484

Pushing <tt><g></tt> twice changes the way the usage is displayed

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /g/data3/hh5/public/apps ----------------------------------------------------
64.0 GiB [ 98.6% ##########] /miniconda3
799.6 MiB [ 1.2% ] /lrose
152.3 MiB [ 0.2% ] /easybuild
4.0 KiB [ 0.0% ] README

Total disk usage: 64.9 GiB Apparent size: 64.6 GiB Items: 2461484

Select a directory, push <tt><return></tt> and it will show you a view of that directory sorted by size

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /g/data3/hh5/public/apps/miniconda3 ------------------------------------------
/..
33.2 GiB [ 51.9% ##########] /pkgs
22.6 GiB [ 35.3% ###### ] /envs
11.8 GiB [ 18.5% ### ] /old-envs
4.6 GiB [ 7.1% # ] /conda-bld
236.6 MiB [ 0.4% ] /lib
8.7 MiB [ 0.0% ] /share
6.1 MiB [ 0.0% ] /bin
5.1 MiB [ 0.0% ] /include
4.9 MiB [ 0.0% ] /conda-meta
4.3 MiB [ 0.0% ] /ssl
1.9 MiB [ 0.0% ] /compiler_compat
28.0 KiB [ 0.0% ] /etc
12.0 KiB [ 0.0% ] /x86_64-conda_cos6-linux-gnu
4.0 KiB [ 0.0% ] /locks
4.0 KiB [ 0.0% ] LICENSE.txt

Total disk usage: 64.0 GiB Apparent size: 63.7 GiB Items: 2446705

Select <tt>/..</tt> at the top to move back up the directory tree. In this way it is possible to navigate the directory structure and pinpoint the largest users of disk space. The disk usage information is cached, so once it has analysed a directory tree it is very fast to navigate.

Press <q>to quit the program.</q>

[[Category:NCI]]

Getting Help

2022-05-04T05:46:59Z

A.heerdegen:

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left- hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*The CLEX slack workspace [https://arccss.slack.com arccss.slack.com]  
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The slack workspace is a good way to contact other CLEX members as well as the CMS team. It is best suited to shorter less complex queries in general. There are a number of channels focussing on different topics. When asking for help it may be that someone else from the Centre will have some useful feedback. It is best not to contact team members directly: we rotate support duties to allow time for other tasks, so asking in a general channel means the team member tasked with support, or who is best placed to answer the question, can assist.

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences. Simply outline your issue in an email and send it to [mailto:cws_help@nci.org.au cws_help@nci.org.au] which will open a help ticket on our helpdesk.

 

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Follow the [[Troubleshooting|troubleshooting guide]]
*Have you read the documentation on this wiki (if it exists), any other relevant documentation, or looked on [https://climate-cms.org our blog]?
*Do you need to do some [[Training|training]] to improve your skills?
*Include all relevant information:
**what have you tried already that hasn't worked?
**provide some background for why you are doing the task that isn't working. Sometimes we might be able to suggest a different approach rather than trying to solve your specific error
**what machine are you running on: your machine (what OS), Gadi at NCI, OOD at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working on Gadi was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. Tell us if you load modules in your .bashrc or .bash_profile files (unrecommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

 

== What can users expect ==

Users can expect:

*to receive a reply to a query within 24 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

Troubleshooting

2022-05-04T03:58:41Z

A.heerdegen:

= I have an error or trying to perform a new task =

These are some general guidelines to help you solve your problem '''before submitting''' a help ticket, or contacting the CMS

== FAQs ==

Check if this is a [[FAQs|frequently asked question for which there is already an answer]]

== Google your error/task ==

Put your error into the search engine of your choice. Begin with the maximum number of keywords and reduce as appropriate to get better quality of search hits/more hits.

If you have a very specific error it may help to put it in quotes so the search engine knows to look for the phrase.

== Trusted sources ==

Not all search hits will have the same quality. These are sites that we recommend:

[[Stackoverflow.com|Stackoverflow]] 

[https://towardsdatascience.com Towards Data Science]

[https://github.com GitHub Issues]

 

== Other Options ==

Try searching or using keywords to find related content in [https://climate-cms.org our blog]

Search THIS WIKI for useful information using the search bar at the top of the page

Stackoverflow has a [https://stackoverflow.com/help/searching specific page] on searching that site, which may help

For NCI related questions check the [https://opus.nci.org.au/display/Help/User+Guides NCI user guides] first

= Data related query =

== Finding Data  ==

See the [[Finding_datasets|Finding Datasets]] page

== Publishing Data ==

See the [[Publishing_options|Publishing Options]] page

== Other Storage Options ==

See [[Storage|Storage]] page. We also have a [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html page on the blog to help you manage your storage]

 

 

 

Troubleshooting

2022-05-04T03:57:53Z

A.heerdegen:

= I have an error or trying to perform a new task =

These are some general guidelines to help you solve your problem '''before submitting''' a help ticket, or contacting the CMS

== FAQs ==

Check if this is a [[FAQs|frequently asked question for which there is already an answer]]

== Google your error/task ==

Put your error into the search engine of your choice. Begin with the maximum number of keywords and reduce as appropriate to get better quality of search hits/more hits.

If you have a very specific error it may help to put it in quotes so the search engine knows to look for the phrase.

== Trusted sources ==

Not all search hits will have the same quality. These are sites that we recommend:

[[Stackoverflow.com|Stackoverflow]] 

[https://towardsdatascience.com Towards Data Science]

[https://github.com GitHub Issues]

 

== Other Options ==

Try searching or using keywords to find related content in [https://climate-cms.org our blog]

Use the search bar at the top of the page to search THIS WIKI for useful information

Stackoverflow has a [https://stackoverflow.com/help/searching specific page] on searching that site, which may help

For NCI related questions check the [https://opus.nci.org.au/display/Help/User+Guides NCI user guides] first

= Data related query =

== Finding Data  ==

See the [[Finding_datasets|Finding Datasets]] page

== Publishing Data ==

See the [[Publishing_options|Publishing Options]] page

== Other Storage Options ==

See [[Storage|Storage]] page. We also have a [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html page on the blog to help you manage your storage]

 

 

 

Troubleshooting

2022-05-04T03:37:04Z

A.heerdegen: Added FAQs section

= I have an error or trying to perform a new task =

These are some general guidelines to help you solve your problem '''before submitting''' a help ticket, or contacting the CMS

== FAQs ==

Check if this is a [[FAQs|frequently asked question for which there is already an answer]]

== Google your error/task ==

Put your error into the search engine of your choice. Begin with the maximum number of keywords and reduce as appropriate to get better quality of search hits/more hits.

If you have a very specific error it may help to put it in quotes so the search engine knows to look for the phrase.

== Trusted sources ==

Not all search hits will have the same quality. These are sites that we recommend:

[[Stackoverflow.com|Stackoverflow]] 

[https://towardsdatascience.com Towards Data Science]

[https://github.com GitHub Issues]

 

== Other Options ==

Try searching or using keywords to find related content in [https://climate-cms.org our blog]

Try searching THIS WIKI

Stackoverflow has a [https://stackoverflow.com/help/searching specific page] on searching that site, which may help

For NCI related questions check the [https://opus.nci.org.au/display/Help/User+Guides NCI user guides] first

= Data related query =

== Finding Data  ==

See the [[Finding_datasets|Finding Datasets]] page

== Publishing Data ==

See the [[Publishing_options|Publishing Options]] page

== Other Storage Options ==

See [[Storage|Storage]] page. We also have a [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html page on the blog to help you manage your storage]

 

 

 

Getting Help

2022-05-04T02:48:01Z

A.heerdegen: Added some expectations from users

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left- hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*The CLEX slack workspace [https://arccss.slack.com arccss.slack.com]  
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The slack workspace is a good way to contact other CLEX members as well as the CMS team. It is best suited to shorter less complex queries in general. There are a number of channels focussing on different topics. When asking for help it may be that someone else from the Centre will have some useful feedback. It is best not to contact team members directly: we rotate support duties to allow time for other tasks, so asking in a general channel means the team member tasked with support, or who is best placed to answer the question, can assist.

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences. Simply outline your issue in an email and send it to [mailto:cws_help@nci.org.au cws_help@nci.org.au] which will open a help ticket on our helpdesk.

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Follow the [[Troubleshooting|troubleshooting guide]]
*Have you read the documentation on this wiki (if it exists), any othe relevant documentation, or looked on [https://climate-cms.org our blog]?
*Do you need to do some [[Training|training]] to improve your skills?
*Include all relevant information:
**what have you tried already that hasn't worked?
**what machine are you running on: your machine (what OS), Gadi at NCI, VDI at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working at NCI was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. Tell us if you load modules in your .bashrc or .bash_profile files (unrecommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

 

== What can users expect ==

Users can expect:

*to receive a reply to a query within 24 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

Scratch file expiry

2022-05-03T01:45:20Z

A.heerdegen:

'''On 17th May 2022''', NCI is introducing an automatic system to purge unused data from the scratch filesystem.   Below we are giving some steps on how best to prepare for that change. We also give some information on some useful tools to prepare a good data workflow going forward.

= Preparation =

Your best course of action is to start preparing now to avoid having any useful data going into quarantine on the 17th May. Below are the steps we recommend you to follow:

*[https://opus.nci.org.au/pages/viewpage.action?pageId=156434436 '''Read''' '''the information'''] provided by NCI
*'''Clean up /g/data''': delete what you can, archive to tape or outside NCI what you can.
*'''Clean up /scratch''': delete what you can, move to /g/data or tape or outside NCI if you need long term storage without accessing it.
*Run “'''nci-file-expiry list-warnings -p <project> > expiry_warning_<project>.txt'''” for all the projects you are a member.
*'''Check the output''' of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Run “'''nci-file-expiry list-warnings'''” : this will catch any file you own in any project you may have forgotten about and decide what to do with anything that appear there.
*'''Rethink your data pipeline''': do not leave data you are not using anymore in /scratch. Decide what you need to do with it when you stop using it: delete, move to /g/data or outside NCI or archive to tape.
*'''Manage your data under /g/data regularly''': review your data, archive to tape or outside NCI as necessary.

= Additional information =

Below is a list of resources you might find useful to prepare for the automatic file expiry and build a good data workflow for the future.

[[Archiving_data|Archiving data at NCI]]

[https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html Blog on building a sustainable data workflow]

[[Storage|Description of the various filesystems at NCI]]

 

 

= Long term strategy =

#'''Keep using /scratch'''. There is not enough disk space if nobody is using /scratch. Your data is safe on /scratch as long as you access it, i.e. read, modify, create. Only the data you won't use for some time needs to be managed.
#'''Managing your data does not mean storing everything on /g/data. '''You need to think about what future use you have for that data. If you don't need it anymore and it can be reproduced, delete it. If you will need to publish the data, publish it now. It is a lot easier to publish a new version if you need a small change than to publish the initial version. For information, on what data needs to be published see the [[CLEX_Data_policy|CLEX Data Policy]] and [[Which_data_should_I_publish|Which_data_should_I_publish]] page. If you can't publish now but won't need it for a long time, put the data on tape. If you won't need this data for a short time but you know you will get back to it soon, put it on /g/data
#'''Learn how to use the tape system'''. See [[How_to_use_MDSS_tape_storage_at_NCI|how to use the MDSS tape system]] or  [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html our blog].
#Try and '''avoid your data getting into quarantine'''. Make managing your data '''a periodic, frequent task''' in your calendar. This includes all data: /scratch, /g/data and tape.
#'''Be careful when going on leave!''' Make sure to check your data holdings under /scratch before your leave.
#'''For extended leave''' (e.g. ship cruises, parental leave, sabbatical), '''do not leave any important data under /scratch.'''

How to use MDSS tape storage at NCI

2022-05-03T01:43:02Z

A.heerdegen:

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term storage of large files. It is possible to retrieve data from MDSS, so this is a good place to store data that will not be required for some time, for backup, or for archiving. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">MDSS proper usage</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">MDSS is designed for medium to long-term storage of large files, this means it is optimised for storing big amounts of data. This means it is most suitable for:</span></span>

*<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files you are required to keep for a long term, like data underlining published datasets, publications, PhD thesis etc.</span></span>
*<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files that you or someone else are likely to reuse or analyse again in the future but not in the next few months. For example, restart files or other model output you are not immediately using should be moved from disk to massdata as soon as possible.</span></span>
*<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">MDSS is suitable for backup of big data projects, like model output which could not be backed up elsewhere.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Preparing your data for mdss</span> =

#<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Organise your files and delete anything which you will not be re-using. Do not transfer data before organising it. It is difficult to get a list of what is stored on massdata, let alone to list what is in a tarred file once it is uploaded. </span></span>
#<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Big files: use tools like <tt>tar</tt> to bundle files together into archive files. Create reasonably big archive files but also think of how you might want to access the data later. There is no point of tarring together two different simulations if you would want to access them separately, as then you would need to transferred back a big amount of data you do not need. Your upload will fail if any of your files are less than 20MB or the average size is less than 250 MB.</span></span>
#<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files should be group readable, with group execute permissions for directories. This helps with long term maintenance, allowing administrators to track the type and size of archived data. You can change the permissions on data you own with the <tt>chmod</tt> unix utility.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">While you are preparing your data to be moved it is an opportunity to also document it, if you have not done so already. You should document what you are archiving and how you are archiving it. Even a simple readme file added to your main directory can help others and your future self. If you are archiving data underlining a publication or published dataset then it is important to have a summary of what is stored in /massdata. This is part of the [[Data_Management_Plan|dataset management plan]] and/or [[Data_Availability_Statement|data availiability statement]].</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Useful tools:</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">         [[TAR_guidelines|TAR- to create archives]] cheatsheet</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">         [[NetCDF_Compression_Tools|Compressing tools]]</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Accessing MDSS</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Massdata cannot be accessed directly via a directory path. All access of MDSS is via the command '''<tt>mdss</tt>.'''</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Users connected to the project have read, write and execute permissions in the corresponding directory on mdss and so may create their own files in it.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif"><tt>mdss</tt> has several sub-commands and options to see all of them use either:</span></span>
<syntaxhighlight lang="bash"> $mdss --help
or
$man mdss
</syntaxhighlight>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">If you don't specify a project, it will use your default project. Then you add a sub-command and the path of the files and directories you want to upload, list etc.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">mdss -P <project-id> + <sub-command> + <path></span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Most useful sub-commands are:</span></span>
<syntaxhighlight lang="bash">mdss put - upload files
mdss get - retrieve files
mdss ls - list directories and files
mdss dmdu - get the size of a directory/file
mdss dmls - show what is on cache and what is on tape

NB "mdss du" will also work but only return the size of what is still cached, dmdu will give the full size of what is on tape regardless if it is cached or not.
</syntaxhighlight>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Please note <tt>mdss</tt> sub-commands work only interactively or on the <tt>copyq </tt>queue. To use it on <tt>copyq</tt> remember to set the storage flag as</span></span>
<pre>-l storage=massdata/<project_code></pre>

= <span style="font-family:Arial,Helvetica,sans-serif">Monitoring MDSS usage</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Unfortunately, there is no command to check the usage by user-id as for /g/data and /scratch. The only way to get this information currently is to ask help<at>nci.org.au. The NCI administrators can access this information for any CI of the group.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Transferring data to and from MDSS</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">NCI also supports the <tt>netmv</tt> and <tt>netcp</tt> commands to work with MDSS. These commands create a copyq job to transfer multiple files. Files can be automatically tarred and compressed as part of the copy process.</span></span>

<span style="color:#c0392b"><span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">'''Warning: The automatic archiving and compression of these tools can use a lot of storage on /scratch if you're moving lots of data!'''</span></span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">For more info run `man netmv`</span></span><span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">The CMS team has also developed a utility called [https://github.com/coecms/mdssdiff mdssdiff] available from [[Conda|our conda environments]]. This utility allows users to compare the contents of the local directory and a directory under /massdata. It will also recursively update the content on the massdata directory to copy the local directory or vice versa.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Modifications to MDSS datasets</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Ask the Lead CI of the project to contact NCI at help@nci.org.au if large metadata operations are needed on massdata, such as changing ownership, project code, permissions etc. of existing datasets</span></span>

[[Category:NCI Guidelines]] [[Category:Data induction]]

How to use MDSS tape storage at NCI

2022-05-03T01:33:12Z

A.heerdegen: A.heerdegen moved page Archiving data to How to use MDSS tape storage at NCI: Better title, reflects usage better

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">All universities have their own archives where you can store your data, for more information contact your Library or IT department. Here we will focus on archiving data using the NCI archive storage, for all cases where using a university archive is not applicable.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">MDSS proper usage</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">MDSS is designed for medium to long-term archive of large files, this means it is optimised for storing big amounts of data, but not for retrieving this data frequently. This means it is most suitable for:</span></span>

*<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files you are required to keep for a long term, like data underlining published datasets, publications, PhD thesis etc.</span></span>
*<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files that you or someone else are likely to reuse or analyse again in the future but not in the next few months. For example, restart files or other model output you are not immediately using should be moved from disk to massdata as soon as possible.</span></span>
*<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">MDSS is suitable for backup of big data projects, like model output which could not be backed up elsewhere.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Preparing your data for mdss</span> =

#<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Organise your files and delete anything which you will not be re-using. Do not transfer data before organising it. It is difficult to get a list of what is stored on massdata, let alone to list what is in a tarred file once it is uploaded. </span></span>
#<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Big files: use tools like <tt>tar</tt> to bundle files together into archive files. Create reasonably big archive files but also think of how you might want to access the data later. There is no point of tarring together two different simulations if you would want to access them separately, as then you would need to transferred back a big amount of data you do not need. Your upload will fail if any of your files are less than 20MB or the average size is less than 250 MB.</span></span>
#<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files should be group readable, with group execute permissions for directories. This helps with long term maintenance, allowing administrators to track the type and size of archived data. You can change the permissions on data you own with the <tt>chmod</tt> unix utility.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">While you are preparing your data to be moved it is an opportunity to also document it, if you have not done so already. You should document what you are archiving and how you are archiving it. Even a simple readme file added to your main directory can help others and your future self. If you are archiving data underlining a publication or published dataset then it is important to have a summary of what is stored in /massdata. This is part of the [[Data_Management_Plan|dataset management plan]] and/or [[Data_Availability_Statement|data availiability statement]].</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Useful tools:</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">         [[TAR_guidelines|TAR- to create archives]] cheatsheet</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">         [[NetCDF_Compression_Tools|Compressing tools]]</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Accessing MDSS</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Massdata cannot be accessed directly via a directory path. All access of MDSS is via the command '''<tt>mdss</tt>.'''</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Users connected to the project have read, write and execute permissions in the corresponding directory on mdss and so may create their own files in it.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif"><tt>mdss</tt> has several sub-commands and options to see all of them use either:</span></span>
<syntaxhighlight lang="bash"> $mdss --help
or
$man mdss
</syntaxhighlight>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">If you don't specify a project, it will use your default project. Then you add a sub-command and the path of the files and directories you want to upload, list etc.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">mdss -P <project-id> + <sub-command> + <path></span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Most useful sub-commands are:</span></span>
<syntaxhighlight lang="bash">mdss put - upload files
mdss get - retrieve files
mdss ls - list directories and files
mdss dmdu - get the size of a directory/file
mdss dmls - show what is on cache and what is on tape

NB "mdss du" will also work but only return the size of what is still cached, dmdu will give the full size of what is on tape regardless if it is cached or not.
</syntaxhighlight>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Please note <tt>mdss</tt> sub-commands work only interactively or on the <tt>copyq </tt>queue. To use it on <tt>copyq</tt> remember to set the storage flag as</span></span>
<pre>-l storage=massdata/<project_code></pre>

= <span style="font-family:Arial,Helvetica,sans-serif">Monitoring MDSS usage</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Unfortunately, there is no command to check the usage by user-id as for /g/data and /scratch. The only way to get this information currently is to ask help<at>nci.org.au. The NCI administrators can access this information for any CI of the group.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Transferring data to and from MDSS</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">NCI also supports the <tt>netmv</tt> and <tt>netcp</tt> commands to work with MDSS. These commands create a copyq job to transfer multiple files. Files can be automatically tarred and compressed as part of the copy process.</span></span>

<span style="color:#c0392b"><span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">'''Warning: The automatic archiving and compression of these tools can use a lot of storage on /scratch if you're moving lots of data!'''</span></span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">For more info run `man netmv`</span></span><span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">.</span></span>

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">The CMS team has also developed a utility called [https://github.com/coecms/mdssdiff mdssdiff] available from [[Conda|our conda environments]]. This utility allows users to compare the contents of the local directory and a directory under /massdata. It will also recursively update the content on the massdata directory to copy the local directory or vice versa.</span></span>

= <span style="font-family:Arial,Helvetica,sans-serif">Modifications to MDSS datasets</span> =

<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Ask the Lead CI of the project to contact NCI at help@nci.org.au if large metadata operations are needed on massdata, such as changing ownership, project code, permissions etc. of existing datasets</span></span>

[[Category:NCI Guidelines]] [[Category:Data induction]]

Archiving data

2022-05-03T01:33:12Z

A.heerdegen: A.heerdegen moved page Archiving data to How to use MDSS tape storage at NCI: Better title, reflects usage better

#REDIRECT [[How to use MDSS tape storage at NCI]]

Troubleshooting

2022-05-03T00:34:02Z

A.heerdegen:

= I have an error or trying to perform a new task =

These are some general guidelines to help you solve your problem '''before submitting''' a help ticket, or contacting the CMS

== Google your error/task ==

Put your error into the search engine of your choice. Begin with the maximum number of keywords and reduce as appropriate to get better quality of search hits/more hits.

If you have a very specific error it may help to put it in quotes so the search engine knows to look for the phrase.

== Trusted sources ==

Not all search hits will have the same quality. These are sites that we recommend:

[[Stackoverflow.com|Stackoverflow]] 

[https://towardsdatascience.com Towards Data Science]

[https://github.com GitHub Issues]

== Other Options ==

Try searching or using keywords to find related content in [https://climate-cms.org our blog]

Try searching THIS WIKI

Stackoverflow has a [https://stackoverflow.com/help/searching specific page] on searching that site, which may help

= Data related query =

== Finding Data  ==

See the [[Finding_datasets|Finding Datasets]] page

== Publishing Data ==

See the [[Publishing_options|Publishing Options]] page

== Other Storage Options ==

See [[Storage|Storage]] page. We also have a [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html page on the blog to help you manage your storage]

 

 

 

Troubleshooting

2022-05-03T00:32:26Z

A.heerdegen: Created page with " = Troubleshooting = == I have an error or trying to perform a new task == These are some general guidelines to help you solve your problem '''before submitting'''..."

= Troubleshooting =

== I have an error or trying to perform a new task ==

These are some general guidelines to help you solve your problem '''before submitting''' a help ticket, or contacting the CMS

=== Google your error/task ===

Put your error into the search engine of your choice. Begin with the maximum number of keywords and reduce as appropriate to get better quality of search hits/more hits.

If you have a very specific error it may help to put it in quotes so the search engine knows to look for the phrase.

=== Trusted sources ===

Not all search hits will have the same quality. These are sites that we recommend:

[[stackoverflow.com|Stackoverflow]] 

[https://towardsdatascience.com Towards Data Science]

[https://github.com GitHub Issues]

=== Other Options ===

Try searching or using keywords to find related content in [https://climate-cms.org our blog]

Try searching THIS WIKI

Stackoverflow has a [https://stackoverflow.com/help/searching specific page] on searching that site, which may help

== Data related query ==

=== Finding Data  ===

See the [[Finding_datasets|Finding Datasets]] page

=== Publishing Data ===

See the [[Publishing_options|Publishing Options]] page

=== Other Storage Options ===

See [[Storage]] page. We also have a [https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html page on the blog to help you manage your storage]

 

 

 

NetCDF Compression Tools

2022-04-26T00:23:14Z

A.heerdegen:

== Why compress netCDF files? ==

Space. You are likely to store your netCDF files on a shared filesystem either at NCI or at your University, as such, it is your responsibility to manage your space correctly and avoid wasting storage space. Compressing your netCDF data files can shrink your data files to one third of its original size. This is the equivalent of being given three times as much disk space.

NetCDF compression is lossless: the data is exactly as it was when read from disk. It can still be read using the same programming interface. As long as the program reading the data has been compiled with the latest netCDF library (version 4) then the task of decompressing the data is handled by the library and as far as the programs are concerned there is no difference in the data. The usual tools, such as ncdump, can be used to examine the variables contained within the netCDF file. However, if you rely on an old piece of software that you think may not have been compiled with netCDF4 you should test that it can read compressed netCDF4 files before converting all your data.

Compression with tools such as gzip is possible but not recommended except for archival purposes. It has the disadvantage that the file must be decompressed to be read and then recompressed again when you have finished, which can be time consuming and degrade your productivity, not to mention the data in question will take up much more room while it is being analysed.

= General guidelines =

The netCDF library has several options for compressing data, which all compression programs will use, as they all use the underlying library to perform the compression. There is a [http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf_compression more detailed explanation] if you wish to understand more, but briefly:

=== Deflate level ===

This is an integer value from ranging from 0 to 9. A value of 0 means no compression, and 9 is the highest level of compression possible. The higher this value the smaller your file will be once compressed. However, there is a trade-off, the higher the deflate level, the longer it will take to compress, particularly so with very high deflate levels. At deflate level 9 it can take six times longer to compress the data, with only a few percent improvement in compression. The recommended deflate level is 5. This combines good compression with a small increase in compression time.

=== Shuffle ===

Turn shuffle on. Simple. It usually results in a smaller compressed file with little performance overhead.

=== Chunking ===

The netCDF library writes the data to disk in "chunks". There is a [http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters very good description of chunking and how it works]. All you really have to know is that in order to use netCDF compression '''your data must be chunked'''.The question then is, do I care how the program I use chooses the size of my data chunks? The answer is almost certainly yes, but maybe not a lot. An optimal chunking strategy is [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes largely determined by the structure of your data and how you will access it].

Specific details about chunking strategies are largely dependent on the tool used to compress your data, and will be covered in more detail in the next section. However, all tools still utilise the underlying netCDF4 library, and so can implement the default chunking strategy, which has changed over time. For many versions the default strategy has been to create chunks that are simply the same size as the dimensions of the variable, which can be a disastrous choice in terms of performance if the data is also compressed. The entire variable must be read into memory to be uncompressed even if only a single slice is required.

= Compression tools =

There are some software packages available on raijin that can be used to compress netCDF data:

=== nco ===

The [http://nco.sourceforge.net netCDF Operator (NCO) program suite] can compress netCDF files and has recently [http://nco.sourceforge.net/nco.html#Compression included some ability to choose different chunking strategies]. It may be that for some cases this is a reasonable solution based on the available options, but a weakness is the inability to use their optimised chunking strategy for variables with four dimensions or more.

=== cdo ===

<span class="cmsl-10">[https://code.zmaw.de/projects/cdo Climate Data Operators] (cdo) can also compress netCDF and offers limited chunking options: </span>auto, grid or lines.

 

=== netcdf ===

One of the standard tools included in a netCDF installation is [https://docs.unidata.ucar.edu/nug/current/netcdf_utilities_guide.html#guide_nccopy nccopy]. nccopy can compress files and define the chunking using a command line argument (-c). nccopy is a good option if your data file structure changes little, so a chunking scheme can be decided upon and hard coded into scripts. It is not so useful if the dimensions and variables change. Another major limitation is that the chunking is defined by dimensions, not variables. If your data file has variables that share dimensions, but have different combinations or numbers of dimensions it is not possible to determine an optimal chunking strategy for each variable.

=== nccompress ===

The nccompress package is available on gadi in the CMS conda environment. At present it consists of three python programs, ncfind, nc2nc and nccompress, written and supported by [http://www.climatescience.org.au/staff/profile/AHeerdegen Aidan]. nc2nc can copy netCDF files with compression and an optimised chunking strategy that has reasonable performance for many datasets. This two main limitations: it is slower than the other programs, and it can only compress netCDF3 or netCDF4 classic format. There is more detail in the following sections.

=== ncvarinfo ===

The convenience utility ncvarinfo is also included, and though it has no direct relevance to compression, it is a convenient way to get a summary of the contents of a netCDF file.

== Identifying files to be compressed ==

ncfind, part of the nccompress package, can be used to find netCDF files and discriminate between compressed and uncompressed:

 
<syntaxhighlight lang="text">
$ ncfind -h
usage: ncfind [-h] [-r] [-u | -c] [inputs [inputs ...]]

Find netCDF files. Can discriminate by compression

positional arguments:
inputs netCDF files or directories (-r must be specified to
recursively descend directories). Can accept piped
arguments.

optional arguments:
-h, --help show this help message and exit
-r, --recursive Recursively descend directories to find netCDF
files (default False)
-u, --uncompressed Find only uncompressed netCDF files (default False)
-c, --compressed Find only compressed netCDF files (default False)

</syntaxhighlight>
There are other methods for finding files, namely the unix utility find utility. For example, to find all files in the directory "directoryname" which end in ".nc": <syntaxhighlight lang="text">
find directoryname -iname "*.nc"

</syntaxhighlight>
However, if your netCDF files do not use the convention of ending in ".nc" or cannot be systematically found based on filename, you can use the ncfind to recursively descend into a directory structure looking for netCDF files: <syntaxhighlight lang="text">
ncfind -r directoryname
</syntaxhighlight>
You can refine the search further by requesting to return only those files that are uncompressed: <syntaxhighlight lang="text">
ncfind -r -u directoryname
</syntaxhighlight>
If you want to find out how much space these uncompressed files occupy you can combine this command with other unix utilities such as xargs and du: <syntaxhighlight lang="text">
ncfind -r -u directoryname | xargs du -h
</syntaxhighlight>
du is the disk usage utility. The output looks something like this: <syntaxhighlight lang="text">
67M output212/ice__212_223.nc
1003M output212/ocean__212_223.nc
1.1G total

</syntaxhighlight>
It is even possible to combine the system find utility with ncfind, using a unix pipe (|). This command will find all files ending in ".nc", pipe the results to ncfind, and only those that are uncompressed will be printed to the screen: <syntaxhighlight lang="text">
find directoryname -iname "*.nc" | ncfind -u
</syntaxhighlight>

== Batch Compressing files ==

Having identified where the netCDF files you wish to compress are located, there is a convenience program, nccompress, which can be used to easily step through and compress each file in turn:
<syntaxhighlight lang="text">
$ nccompress -h
usage: nccompress [-h] [-d {1-9}] [-n] [-b BUFFERSIZE] [-t TMPDIR] [-v] [-r]
[-o] [-m MAXCOMPRESS] [-p] [-f] [-c] [-pa] [-np NUMPROC]
[--nccopy]
inputs [inputs ...]

Run nc2nc (or nccopy) on a number of netCDF files

positional arguments:
inputs netCDF files or directories (-r must be specified to
recursively descend directories)

optional arguments:
-h, --help show this help message and exit
-d {1-9}, --dlevel {1-9}
Set deflate level. Valid values 0-9 (default=5)
-n, --noshuffle Don't shuffle on deflation (default is to shuffle)
-b BUFFERSIZE, --buffersize BUFFERSIZE
Set size of copy buffer in MB (default=50)
-t TMPDIR, --tmpdir TMPDIR
Specify temporary directory to save compressed files
-v, --verbose Verbose output
-r, --recursive Recursively descend directories compressing all netCDF
files (default False)
-o, --overwrite Overwrite original files with compressed versions
(default is to not overwrite)
-m MAXCOMPRESS, --maxcompress MAXCOMPRESS
Set a maximum compression as a paranoid check on
success of nccopy (default is 10, set to zero for no
check)
-p, --paranoid Paranoid check : run nco ndiff on the resulting file
ensure no data has been altered
-f, --force Force compression, even if input file is already
compressed (default False)
-c, --clean Clean tmpdir by removing existing compressed files
before starting (default False)
-pa, --parallel Compress files in parallel
-np NUMPROC, --numproc NUMPROC
Specify the number of processes to use in parallel
operation
--nccopy Use nccopy instead of nc2nc (default False)

</syntaxhighlight>
The simplest way to invoke the program would be with a single file: <syntaxhighlight lang="text">
nccompress ice_daily_0001.nc
</syntaxhighlight>
or using a wildcard expression: <syntaxhighlight lang="text">
nccompress ice*.nc
</syntaxhighlight>
You can also specify one or more directory names in combination with the recursive flag (-r) and the program will recursively descend into those directories and find all netCDF files contained therein. For example, a directory listing might look like so: <syntaxhighlight lang="text">
$ ls data/
output001 output003 output005 output007 output009 restart001 restart003 restart005 restart007 restart009
output002 output004 output006 output008 output010 restart002 restart004 restart006 restart008 restart010
</syntaxhighlight>
with a number of sub-directories, all containing netCDF files. It is a good idea to do a trial run and make sure it functions properly. For example, this will compress the netCDF files in just one of the directories: <syntaxhighlight lang="text">
nccompress -p -r data/output001
</syntaxhighlight>
Once completed there will be a new subdirectory called tmp.nc_compress inside the directory output001. It will contain compressed copies of all the netCDF files from the directory above. You can check the compressed copies to make sure they are correct. The paranoid option (-p) calls an nco command to check that the variables contained in the two files are the same. You can use the paranoid option routinely, thought it will make the process more time consuming. It is a good idea to use it in the testing phase. You should also check the compressed copies manually to make sure they look ok, and if so, re-run the command with the -o option (overwrite): <syntaxhighlight lang="text">
nccompress -r -o data/output001
</syntaxhighlight>

and it will find the already compressed files, copy them over the originals and delete the temporary directory tmp.nc_compress. It won’t try to compress the files again. It also won’t compress already compressed files, so, for example, if you were happy that the compression was working well you could compress the entire data directory, and the already compressed files in output001 will not be re-compressed.

So, by default, nccompress '''does not overwrite the original files'''. If you invoke it without the '-o' option it will create compressed copies in the tmp.nc_compress subdirectory and leave them there, which will consume more disk space! This is a feature, not a bug, but you need to be aware that this is how it functions.

With large variables, which usually means large files (> 1GB) it is a good idea to specify a larger buffer size with the '-b' option, as it will run faster. On raijin this may mean you need to run interactively with a higher memory (~10GB) or submit it as a copyq job. A typical buffer size might be 1000 -> 5000 (1->5 GB).

It is also possible to use wildcards type operations, e.g.

 
<syntaxhighlight lang="text">
nccompress -r -o output*

nccompress -r -o output00[1-5]

nccompress -r -o run[1-5]/output*/ocean*.nc random.nc ice*.nc
</syntaxhighlight>
The nccompress program just sorts out finding files/directories etc, it calls nc2nc to do the compression. Using the '--nccopy' forces nccompress to use the nccopy program in place of nc2nc, though the netcdf package must already be loaded for this to work. You can tell nccompress to work on multple files simultaneously with the '-pa' option. By default this will use all the physical processors on the machine, or you can specify how many simultaneous processes you want to with '-np', e.g. <syntaxhighlight lang="text">
nccompress -r -o -np 16 run[1-5]/output*/ocean*.nc random.nc ice*.nc
</syntaxhighlight>
will compress 16 netCDF files at a time (the -np option implies parallel option). As each directory is processed before beginning on a new directory there will be little reduction in execution time if there are few netCDF files in each directory.
== nc2nc ==

The nc2nc program was written because no existing tool had a generalised per variable chunking algorithm. The total chunk size is defined to be the file system block size (4096KB). The dimensions of the chunk are sized to be as close as possible to the same ratio as the dimensions of the data, with the limits that no dimension can be less than 1. This chunking scheme performs well for a wide range of data, but there will always be cases for certain types of access, or variable shape that this is not optimal. In those cases a different approach may be required.

Be aware that nc2nc takes at least twice as long to compress an equivalent file as nccopy. In some cases with large files containing many variables it can be up to five times slower.

You can use nc2nc “stand alone”. It has a couple of extra features that can only be accessed by calling it directly:
<syntaxhighlight lang="text">
$ nc2nc -h
usage: nc2nc [-h] [-d {1-9}] [-m MINDIM] [-b BUFFERSIZE] [-n] [-v] [-c] [-f]
[-va VARS] [-q QUANTIZE] [-o]
origin destination

Make a copy of a netCDF file with automatic chunk sizing

positional arguments:
origin netCDF file to be compressed
destination netCDF output file

optional arguments:
-h, --help show this help message and exit
-d {1-9}, --dlevel {1-9}
Set deflate level. Valid values 0-9 (default=5)
-m MINDIM, --mindim MINDIM
Minimum dimension of chunk. Valid values 1-dimsize
-b BUFFERSIZE, --buffersize BUFFERSIZE
Set size of copy buffer in MB (default=50)
-n, --noshuffle Don't shuffle on deflation (default is to shuffle)
-v, --verbose Verbose output
-c, --classic use NETCDF4_CLASSIC output instead of NETCDF4 (default
true)
-f, --fletcher32 Activate Fletcher32 checksum
-va VARS, --vars VARS
Specify variables to copy (default is to copy all)
-q QUANTIZE, --quantize QUANTIZE
Truncate data in variable to a given decimal
precision, e.g. -q speed=2 -q temp=0 causes variable
speed to be truncated to a precision of 0.01 and temp
to a precision of 1
-o, --overwrite Write output file even if already it exists (default
is to not overwrite)
</syntaxhighlight>
With the vars option (-va) it is possible to select out only a subset of variables to be copied to the destination file. By default the output file is netCDf4 classic, but this can be changed to netCDF4 using the '-c' option. It is also possible to specify a minimum dimension size for the chunks (-m). This may be desirable for a dataset that has one particularly long dimension,. The chunk dimensions would mirror this and be very large in this direction . If fast access is required from slices orthogonal to this direction performance might be improved setting this option to a number greater than 1.
== ncvarinfo ==

ncvarinfo is a convenient way to get a summary of the contents of a netCDF file.
<syntaxhighlight lang="text">
./ncvarinfo -h
usage: ncvarinfo [-h] [-v] [-t] [-d] [-a] [-va VARS] inputs [inputs ...]

Output summary information about a netCDF file

positional arguments:
inputs netCDF files

optional arguments:
-h, --help show this help message and exit
-v, --verbose Verbose output
-t, --time Show time variables
-d, --dims Show dimensions
-a, --aggregate Aggregate multiple netCDF files into one dataset
-va VARS, --vars VARS
Show info for only specify variables

</syntaxhighlight>
By default it prints out a simple summary of the variables in a netCDF file, but omitting dimensions and time related variables. e.g. <syntaxhighlight lang="text">
ncvarinfo output096/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

</syntaxhighlight>
If you specify more than one file it will print the information for each file in turn <syntaxhighlight lang="text">
ncvarinfo output09?/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output097/ocean_daily.nc
Time steps: 365 x 1.0 days
output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output099/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude
</syntaxhighlight>
If the files have the same structure it is possible to aggregate the data and display it as if it were contained in a single dataset: <syntaxhighlight lang="text">
ncvarinfo -a output09?/ocean_daily.nc

Time steps: 1460 x 1.0 days
tau_x :: (1460, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (1460, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude
</syntaxhighlight>
You can also just request variables you are interested in to be output: <syntaxhighlight lang="text">
ncvarinfo -va tau_x -va tau_y output09?/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output097/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output099/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
</syntaxhighlight>

 

[[Category:Data induction]]

NetCDF Compression Tools

2022-04-26T00:21:30Z

A.heerdegen: Re-create Claire's edits

== Why compress netCDF files? ==

Space. <ins>You are likely to store your netCDF files on a shared filesystem either at NCI or at your University, as such, it </ins>is your <ins>responsibility to manage your space correctly and avoid wasting storage </ins>space. Compressing your netCDF data files <ins>can </ins>shrink your data files to one third <ins>of </ins>the size. This is the equivalent of being given three times as much disk space.

NetCDF compression is lossless: the data is exactly as it was when read from disk. It can still be read using the same programming interface. As long as the program reading the data has been compiled with the latest netCDF library (version 4) then the task of decompressing the data is handled by the library and as far as the programs are concerned there is no difference in the data. The usual tools, such as ncdump, can be used to examine the variables contained within the netCDF file. However, if you rely on an old piece of software that you think may not have been compiled with netCDF4 you should test that it can read compressed netCDF4 files before converting all your data.

Compression with tools such as gzip is possible but not recommended except for archival purposes. It has the disadvantage that the file must be decompressed to be read and then recompressed again when you have finished, which can be time consuming and degrade your productivity, not to mention the data in question will take up much more room while it is being analysed.

= General guidelines =

The netCDF library has several options for compressing data, which all compression programs will use, as they all use the underlying library to perform the compression. There is a [http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf_compression more detailed explanation] if you wish to understand more, but briefly:

=== Deflate level ===

This is an integer value from ranging from 0 to 9. A value of 0 means no compression, and 9 is the highest level of compression possible. The higher this value the smaller your file will be once compressed. However, there is a trade-off, the higher the deflate level, the longer it will take to compress, particularly so with very high deflate levels. At deflate level 9 it can take six times longer to compress the data, with only a few percent improvement in compression. The recommended deflate level is 5. This combines good compression with a small increase in compression time.

=== Shuffle ===

Turn shuffle on. Simple. It usually results in a smaller compressed file with little performance overhead.

=== Chunking ===

The netCDF library writes the data to disk in "chunks". There is a [http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters very good description of chunking and how it works]. All you really have to know is that in order to use netCDF compression '''your data must be chunked'''.The question then is, do I care how the program I use chooses the size of my data chunks? The answer is almost certainly yes, but maybe not a lot. An optimal chunking strategy is [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes largely determined by the structure of your data and how you will access it].

Specific details about chunking strategies are largely dependent on the tool used to compress your data, and will be covered in more detail in the next section. However, all tools still utilise the underlying netCDF4 library, and so can implement the default chunking strategy, which has changed over time. For many versions the default strategy has been to create chunks that are simply the same size as the dimensions of the variable, which can be a disastrous choice in terms of performance if the data is also compressed. The entire variable must be read into memory to be uncompressed even if only a single slice is required.

= Compression tools =

There are some software packages available on raijin that can be used to compress netCDF data:

=== nco ===

The [http://nco.sourceforge.net netCDF Operator (NCO) program suite] can compress netCDF files and has recently [http://nco.sourceforge.net/nco.html#Compression included some ability to choose different chunking strategies]. It may be that for some cases this is a reasonable solution based on the available options, but a weakness is the inability to use their optimised chunking strategy for variables with four dimensions or more.

=== cdo ===

<span class="cmsl-10">[https://code.zmaw.de/projects/cdo Climate Data Operators] (cdo) can also compress netCDF and offers limited chunking options: </span>auto, grid or lines.

 

=== netcdf ===

One of the standard tools included in a netCDF installation is [https://docs.unidata.ucar.edu/nug/current/netcdf_utilities_guide.html#guide_nccopy nccopy]. nccopy can compress files and define the chunking using a command line argument (-c). nccopy is a good option if your data file structure changes little, so a chunking scheme can be decided upon and hard coded into scripts. It is not so useful if the dimensions and variables change. Another major limitation is that the chunking is defined by dimensions, not variables. If your data file has variables that share dimensions, but have different combinations or numbers of dimensions it is not possible to determine an optimal chunking strategy for each variable.

=== nccompress ===

The nccompress package is available on gadi in the CMS conda environment. At present it consists of three python programs, ncfind, nc2nc and nccompress, written and supported by [http://www.climatescience.org.au/staff/profile/AHeerdegen Aidan]. nc2nc can copy netCDF files with compression and an optimised chunking strategy that has reasonable performance for many datasets. This two main limitations: it is slower than the other programs, and it can only compress netCDF3 or netCDF4 classic format. There is more detail in the following sections.

=== ncvarinfo ===

The convenience utility ncvarinfo is also included, and though it has no direct relevance to compression, it is a convenient way to get a summary of the contents of a netCDF file.

== Identifying files to be compressed ==

ncfind, part of the nccompress package, can be used to find netCDF files and discriminate between compressed and uncompressed:

 
<syntaxhighlight lang="text">
$ ncfind -h
usage: ncfind [-h] [-r] [-u | -c] [inputs [inputs ...]]

Find netCDF files. Can discriminate by compression

positional arguments:
inputs netCDF files or directories (-r must be specified to
recursively descend directories). Can accept piped
arguments.

optional arguments:
-h, --help show this help message and exit
-r, --recursive Recursively descend directories to find netCDF
files (default False)
-u, --uncompressed Find only uncompressed netCDF files (default False)
-c, --compressed Find only compressed netCDF files (default False)

</syntaxhighlight>
There are other methods for finding files, namely the unix utility find utility. For example, to find all files in the directory "directoryname" which end in ".nc": <syntaxhighlight lang="text">
find directoryname -iname "*.nc"

</syntaxhighlight>
However, if your netCDF files do not use the convention of ending in ".nc" or cannot be systematically found based on filename, you can use the ncfind to recursively descend into a directory structure looking for netCDF files: <syntaxhighlight lang="text">
ncfind -r directoryname
</syntaxhighlight>
You can refine the search further by requesting to return only those files that are uncompressed: <syntaxhighlight lang="text">
ncfind -r -u directoryname
</syntaxhighlight>
If you want to find out how much space these uncompressed files occupy you can combine this command with other unix utilities such as xargs and du: <syntaxhighlight lang="text">
ncfind -r -u directoryname | xargs du -h
</syntaxhighlight>
du is the disk usage utility. The output looks something like this: <syntaxhighlight lang="text">
67M output212/ice__212_223.nc
1003M output212/ocean__212_223.nc
1.1G total

</syntaxhighlight>
It is even possible to combine the system find utility with ncfind, using a unix pipe (|). This command will find all files ending in ".nc", pipe the results to ncfind, and only those that are uncompressed will be printed to the screen: <syntaxhighlight lang="text">
find directoryname -iname "*.nc" | ncfind -u
</syntaxhighlight>

== Batch Compressing files ==

Having identified where the netCDF files you wish to compress are located, there is a convenience program, nccompress, which can be used to easily step through and compress each file in turn:
<syntaxhighlight lang="text">
$ nccompress -h
usage: nccompress [-h] [-d {1-9}] [-n] [-b BUFFERSIZE] [-t TMPDIR] [-v] [-r]
[-o] [-m MAXCOMPRESS] [-p] [-f] [-c] [-pa] [-np NUMPROC]
[--nccopy]
inputs [inputs ...]

Run nc2nc (or nccopy) on a number of netCDF files

positional arguments:
inputs netCDF files or directories (-r must be specified to
recursively descend directories)

optional arguments:
-h, --help show this help message and exit
-d {1-9}, --dlevel {1-9}
Set deflate level. Valid values 0-9 (default=5)
-n, --noshuffle Don't shuffle on deflation (default is to shuffle)
-b BUFFERSIZE, --buffersize BUFFERSIZE
Set size of copy buffer in MB (default=50)
-t TMPDIR, --tmpdir TMPDIR
Specify temporary directory to save compressed files
-v, --verbose Verbose output
-r, --recursive Recursively descend directories compressing all netCDF
files (default False)
-o, --overwrite Overwrite original files with compressed versions
(default is to not overwrite)
-m MAXCOMPRESS, --maxcompress MAXCOMPRESS
Set a maximum compression as a paranoid check on
success of nccopy (default is 10, set to zero for no
check)
-p, --paranoid Paranoid check : run nco ndiff on the resulting file
ensure no data has been altered
-f, --force Force compression, even if input file is already
compressed (default False)
-c, --clean Clean tmpdir by removing existing compressed files
before starting (default False)
-pa, --parallel Compress files in parallel
-np NUMPROC, --numproc NUMPROC
Specify the number of processes to use in parallel
operation
--nccopy Use nccopy instead of nc2nc (default False)

</syntaxhighlight>
The simplest way to invoke the program would be with a single file: <syntaxhighlight lang="text">
nccompress ice_daily_0001.nc
</syntaxhighlight>
or using a wildcard expression: <syntaxhighlight lang="text">
nccompress ice*.nc
</syntaxhighlight>
You can also specify one or more directory names in combination with the recursive flag (-r) and the program will recursively descend into those directories and find all netCDF files contained therein. For example, a directory listing might look like so: <syntaxhighlight lang="text">
$ ls data/
output001 output003 output005 output007 output009 restart001 restart003 restart005 restart007 restart009
output002 output004 output006 output008 output010 restart002 restart004 restart006 restart008 restart010
</syntaxhighlight>
with a number of sub-directories, all containing netCDF files. It is a good idea to do a trial run and make sure it functions properly. For example, this will compress the netCDF files in just one of the directories: <syntaxhighlight lang="text">
nccompress -p -r data/output001
</syntaxhighlight>
Once completed there will be a new subdirectory called tmp.nc_compress inside the directory output001. It will contain compressed copies of all the netCDF files from the directory above. You can check the compressed copies to make sure they are correct. The paranoid option (-p) calls an nco command to check that the variables contained in the two files are the same. You can use the paranoid option routinely, thought it will make the process more time consuming. It is a good idea to use it in the testing phase. You should also check the compressed copies manually to make sure they look ok, and if so, re-run the command with the -o option (overwrite): <syntaxhighlight lang="text">
nccompress -r -o data/output001
</syntaxhighlight>

and it will find the already compressed files, copy them over the originals and delete the temporary directory tmp.nc_compress. It won’t try to compress the files again. It also won’t compress already compressed files, so, for example, if you were happy that the compression was working well you could compress the entire data directory, and the already compressed files in output001 will not be re-compressed.

So, by default, nccompress '''does not overwrite the original files'''. If you invoke it without the '-o' option it will create compressed copies in the tmp.nc_compress subdirectory and leave them there, which will consume more disk space! This is a feature, not a bug, but you need to be aware that this is how it functions.

With large variables, which usually means large files (> 1GB) it is a good idea to specify a larger buffer size with the '-b' option, as it will run faster. On raijin this may mean you need to run interactively with a higher memory (~10GB) or submit it as a copyq job. A typical buffer size might be 1000 -> 5000 (1->5 GB).

It is also possible to use wildcards type operations, e.g.

 
<syntaxhighlight lang="text">
nccompress -r -o output*

nccompress -r -o output00[1-5]

nccompress -r -o run[1-5]/output*/ocean*.nc random.nc ice*.nc
</syntaxhighlight>
The nccompress program just sorts out finding files/directories etc, it calls nc2nc to do the compression. Using the '--nccopy' forces nccompress to use the nccopy program in place of nc2nc, though the netcdf package must already be loaded for this to work. You can tell nccompress to work on multple files simultaneously with the '-pa' option. By default this will use all the physical processors on the machine, or you can specify how many simultaneous processes you want to with '-np', e.g. <syntaxhighlight lang="text">
nccompress -r -o -np 16 run[1-5]/output*/ocean*.nc random.nc ice*.nc
</syntaxhighlight>
will compress 16 netCDF files at a time (the -np option implies parallel option). As each directory is processed before beginning on a new directory there will be little reduction in execution time if there are few netCDF files in each directory.
== nc2nc ==

The nc2nc program was written because no existing tool had a generalised per variable chunking algorithm. The total chunk size is defined to be the file system block size (4096KB). The dimensions of the chunk are sized to be as close as possible to the same ratio as the dimensions of the data, with the limits that no dimension can be less than 1. This chunking scheme performs well for a wide range of data, but there will always be cases for certain types of access, or variable shape that this is not optimal. In those cases a different approach may be required.

Be aware that nc2nc takes at least twice as long to compress an equivalent file as nccopy. In some cases with large files containing many variables it can be up to five times slower.

You can use nc2nc “stand alone”. It has a couple of extra features that can only be accessed by calling it directly:
<syntaxhighlight lang="text">
$ nc2nc -h
usage: nc2nc [-h] [-d {1-9}] [-m MINDIM] [-b BUFFERSIZE] [-n] [-v] [-c] [-f]
[-va VARS] [-q QUANTIZE] [-o]
origin destination

Make a copy of a netCDF file with automatic chunk sizing

positional arguments:
origin netCDF file to be compressed
destination netCDF output file

optional arguments:
-h, --help show this help message and exit
-d {1-9}, --dlevel {1-9}
Set deflate level. Valid values 0-9 (default=5)
-m MINDIM, --mindim MINDIM
Minimum dimension of chunk. Valid values 1-dimsize
-b BUFFERSIZE, --buffersize BUFFERSIZE
Set size of copy buffer in MB (default=50)
-n, --noshuffle Don't shuffle on deflation (default is to shuffle)
-v, --verbose Verbose output
-c, --classic use NETCDF4_CLASSIC output instead of NETCDF4 (default
true)
-f, --fletcher32 Activate Fletcher32 checksum
-va VARS, --vars VARS
Specify variables to copy (default is to copy all)
-q QUANTIZE, --quantize QUANTIZE
Truncate data in variable to a given decimal
precision, e.g. -q speed=2 -q temp=0 causes variable
speed to be truncated to a precision of 0.01 and temp
to a precision of 1
-o, --overwrite Write output file even if already it exists (default
is to not overwrite)
</syntaxhighlight>
With the vars option (-va) it is possible to select out only a subset of variables to be copied to the destination file. By default the output file is netCDf4 classic, but this can be changed to netCDF4 using the '-c' option. It is also possible to specify a minimum dimension size for the chunks (-m). This may be desirable for a dataset that has one particularly long dimension,. The chunk dimensions would mirror this and be very large in this direction . If fast access is required from slices orthogonal to this direction performance might be improved setting this option to a number greater than 1.
== ncvarinfo ==

ncvarinfo is a convenient way to get a summary of the contents of a netCDF file.
<syntaxhighlight lang="text">
./ncvarinfo -h
usage: ncvarinfo [-h] [-v] [-t] [-d] [-a] [-va VARS] inputs [inputs ...]

Output summary information about a netCDF file

positional arguments:
inputs netCDF files

optional arguments:
-h, --help show this help message and exit
-v, --verbose Verbose output
-t, --time Show time variables
-d, --dims Show dimensions
-a, --aggregate Aggregate multiple netCDF files into one dataset
-va VARS, --vars VARS
Show info for only specify variables

</syntaxhighlight>
By default it prints out a simple summary of the variables in a netCDF file, but omitting dimensions and time related variables. e.g. <syntaxhighlight lang="text">
ncvarinfo output096/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

</syntaxhighlight>
If you specify more than one file it will print the information for each file in turn <syntaxhighlight lang="text">
ncvarinfo output09?/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output097/ocean_daily.nc
Time steps: 365 x 1.0 days
output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output099/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude
</syntaxhighlight>
If the files have the same structure it is possible to aggregate the data and display it as if it were contained in a single dataset: <syntaxhighlight lang="text">
ncvarinfo -a output09?/ocean_daily.nc

Time steps: 1460 x 1.0 days
tau_x :: (1460, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (1460, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude
</syntaxhighlight>
You can also just request variables you are interested in to be output: <syntaxhighlight lang="text">
ncvarinfo -va tau_x -va tau_y output09?/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output097/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output099/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
</syntaxhighlight>

 

[[Category:Data induction]]

NetCDF Compression Tools

2022-04-26T00:16:28Z

A.heerdegen: Reverted edits by C.carouge (talk) to last revision by P.petrelli

== Why compress netCDF files? ==

Space. Your supervisor/CI/system administrator is telling you your files are taking too much space. Compressing your netCDF data files will shrink your data files to one third the size. This is the equivalent of being given three times as much disk space.

NetCDF compression is lossless: the data is exactly as it was when read from disk. It can still be read using the same programming interface. As long as the program reading the data has been compiled with the latest netCDF library (version 4) then the task of decompressing the data is handled by the library and as far as the programs are concerned there is no difference in the data. The usual tools, such as ncdump, can be used to examine the variables contained within the netCDF file. However, if you rely on an old piece of software that you think may not have been compiled with netCDF4 you should test that it can read compressed netCDF4 files before converting all your data.

It is possible to simply compress the entire file on disk with tools such as gzip. This has the disadvantage that the file must be decompressed to be read and then recompressed again when you have finished, which can be time consuming and degrade your productivity, not to mention the data in question will take up much more room while it is being analysed. This is not recommended.

= General guidelines =

The netCDF library has several options for compressing data, which all compression programs will use, as they all use the underlying library to perform the compression. There is a [http://www.unidata.ucar.edu/blogs/developer/en/entry/netcdf_compression more detailed explanation] if you wish to understand more, but briefly:

=== Deflate level ===

This is an integer value from ranging from 0 to 9. A value of 0 means no compression, and 9 is the highest level of compression possible. The higher this value the smaller your file will be once compressed. However, there is a trade-off, the higher the deflate level, the longer it will take to compress, particularly so with very high deflate levels. At deflate level 9 it can take six times longer to compress the data, with only a few percent improvement in compression. The recommended deflate level is 5. This combines good compression with a small increase in compression time.

=== Shuffle ===

Turn shuffle on. Simple. It usually results in a smaller compressed file with little performance overhead.

=== Chunking ===

The netCDF library writes the data to disk in "chunks". There is a [http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters very good description of chunking and how it works]. All you really have to know is that in order to use netCDF compression '''your data must be chunked'''.The question then is, do I care how the program I use chooses the size of my data chunks? The answer is almost certainly yes, but maybe not a lot. An optimal chunking strategy is [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes largely determined by the structure of your data and how you will access it].

Specific details about chunking strategies are largely dependent on the tool used to compress your data, and will be covered in more detail in the next section. However, all tools still utilise the underlying netCDF4 library, and so can implement the default chunking strategy, which has changed over time. For many versions the default strategy has been to create chunks that are simply the same size as the dimensions of the variable, which can be a disastrous choice in terms of performance if the data is also compressed. The entire variable must be read into memory to be uncompressed even if only a single slice is required.

= Compression tools =

There are some software packages available on raijin that can be used to compress netCDF data:

=== nco ===

The [http://nco.sourceforge.net netCDF Operator (NCO) program suite] can compress netCDF files and has recently [http://nco.sourceforge.net/nco.html#Compression included some ability to choose different chunking strategies]. It may be that for some cases this is a reasonable solution based on the available options, but a weakness is the inability to use their optimised chunking strategy for variables with four dimensions or more.

=== cdo ===

<span class="cmsl-10">[https://code.zmaw.de/projects/cdo Climate Data Operators] (cdo) can also compress netCDF and offers limited chunking options: </span>auto, grid or lines.

=== netcdf ===

One of the standard tools included in a netCDF installation is [https://docs.unidata.ucar.edu/nug/current/netcdf_utilities_guide.html#guide_nccopy nccopy]. nccopy can compress files and define the chunking using a command line argument (-c). nccopy is a good option if your data file structure changes little, so a chunking scheme can be decided upon and hard coded into scripts. It is not so useful if the dimensions and variables change. Another major limitation is that the chunking is defined by dimensions, not variables. If your data file has variables that share dimensions, but have different combinations or numbers of dimensions it is not possible to determine an optimal chunking strategy for each variable.

=== nccompress ===

The nccompress package is available on gadi in the CMS conda environment. At present it consists of three python programs, ncfind, nc2nc and nccompress, written and supported by [http://www.climatescience.org.au/staff/profile/AHeerdegen Aidan]. nc2nc can copy netCDF files with compression and an optimised chunking strategy that has reasonable performance for many datasets. This two main limitations: it is slower than the other programs, and it can only compress netCDF3 or netCDF4 classic format. There is more detail in the following sections.

=== ncvarinfo ===

The convenience utility ncvarinfo is also included, and though it has no direct relevance to compression, it is a convenient way to get a summary of the contents of a netCDF file.

== Identifying files to be compressed ==

ncfind, part of the nccompress package, can be used to find netCDF files and discriminate between compressed and uncompressed:

 
<syntaxhighlight lang="text">
$ ncfind -h
usage: ncfind [-h] [-r] [-u | -c] [inputs [inputs ...]]

Find netCDF files. Can discriminate by compression

positional arguments:
inputs netCDF files or directories (-r must be specified to
recursively descend directories). Can accept piped
arguments.

optional arguments:
-h, --help show this help message and exit
-r, --recursive Recursively descend directories to find netCDF
files (default False)
-u, --uncompressed Find only uncompressed netCDF files (default False)
-c, --compressed Find only compressed netCDF files (default False)

</syntaxhighlight>
There are other methods for finding files, namely the unix utility find utility. For example, to find all files in the directory "directoryname" which end in ".nc": <syntaxhighlight lang="text">
find directoryname -iname "*.nc"

</syntaxhighlight>
However, if your netCDF files do not use the convention of ending in ".nc" or cannot be systematically found based on filename, you can use the ncfind to recursively descend into a directory structure looking for netCDF files: <syntaxhighlight lang="text">
ncfind -r directoryname
</syntaxhighlight>
You can refine the search further by requesting to return only those files that are uncompressed: <syntaxhighlight lang="text">
ncfind -r -u directoryname
</syntaxhighlight>
If you want to find out how much space these uncompressed files occupy you can combine this command with other unix utilities such as xargs and du: <syntaxhighlight lang="text">
ncfind -r -u directoryname | xargs du -h
</syntaxhighlight>
du is the disk usage utility. The output looks something like this: <syntaxhighlight lang="text">
67M output212/ice__212_223.nc
1003M output212/ocean__212_223.nc
1.1G total

</syntaxhighlight>
It is even possible to combine the system find utility with ncfind, using a unix pipe (|). This command will find all files ending in ".nc", pipe the results to ncfind, and only those that are uncompressed will be printed to the screen: <syntaxhighlight lang="text">
find directoryname -iname "*.nc" | ncfind -u
</syntaxhighlight>

== Batch Compressing files ==

Having identified where the netCDF files you wish to compress are located, there is a convenience program, nccompress, which can be used to easily step through and compress each file in turn:
<syntaxhighlight lang="text">
$ nccompress -h
usage: nccompress [-h] [-d {1-9}] [-n] [-b BUFFERSIZE] [-t TMPDIR] [-v] [-r]
[-o] [-m MAXCOMPRESS] [-p] [-f] [-c] [-pa] [-np NUMPROC]
[--nccopy]
inputs [inputs ...]

Run nc2nc (or nccopy) on a number of netCDF files

positional arguments:
inputs netCDF files or directories (-r must be specified to
recursively descend directories)

optional arguments:
-h, --help show this help message and exit
-d {1-9}, --dlevel {1-9}
Set deflate level. Valid values 0-9 (default=5)
-n, --noshuffle Don't shuffle on deflation (default is to shuffle)
-b BUFFERSIZE, --buffersize BUFFERSIZE
Set size of copy buffer in MB (default=50)
-t TMPDIR, --tmpdir TMPDIR
Specify temporary directory to save compressed files
-v, --verbose Verbose output
-r, --recursive Recursively descend directories compressing all netCDF
files (default False)
-o, --overwrite Overwrite original files with compressed versions
(default is to not overwrite)
-m MAXCOMPRESS, --maxcompress MAXCOMPRESS
Set a maximum compression as a paranoid check on
success of nccopy (default is 10, set to zero for no
check)
-p, --paranoid Paranoid check : run nco ndiff on the resulting file
ensure no data has been altered
-f, --force Force compression, even if input file is already
compressed (default False)
-c, --clean Clean tmpdir by removing existing compressed files
before starting (default False)
-pa, --parallel Compress files in parallel
-np NUMPROC, --numproc NUMPROC
Specify the number of processes to use in parallel
operation
--nccopy Use nccopy instead of nc2nc (default False)

</syntaxhighlight>
The simplest way to invoke the program would be with a single file: <syntaxhighlight lang="text">
nccompress ice_daily_0001.nc
</syntaxhighlight>
or using a wildcard expression: <syntaxhighlight lang="text">
nccompress ice*.nc
</syntaxhighlight>
You can also specify one or more directory names in combination with the recursive flag (-r) and the program will recursively descend into those directories and find all netCDF files contained therein. For example, a directory listing might look like so: <syntaxhighlight lang="text">
$ ls data/
output001 output003 output005 output007 output009 restart001 restart003 restart005 restart007 restart009
output002 output004 output006 output008 output010 restart002 restart004 restart006 restart008 restart010
</syntaxhighlight>
with a number of sub-directories, all containing netCDF files. It is a good idea to do a trial run and make sure it functions properly. For example, this will compress the netCDF files in just one of the directories: <syntaxhighlight lang="text">
nccompress -p -r data/output001
</syntaxhighlight>
Once completed there will be a new subdirectory called tmp.nc_compress inside the directory output001. It will contain compressed copies of all the netCDF files from the directory above. You can check the compressed copies to make sure they are correct. The paranoid option (-p) calls an nco command to check that the variables contained in the two files are the same. You can use the paranoid option routinely, thought it will make the process more time consuming. It is a good idea to use it in the testing phase. You should also check the compressed copies manually to make sure they look ok, and if so, re-run the command with the -o option (overwrite): <syntaxhighlight lang="text">
nccompress -r -o data/output001
</syntaxhighlight>

and it will find the already compressed files, copy them over the originals and delete the temporary directory tmp.nc_compress. It won’t try to compress the files again. It also won’t compress already compressed files, so, for example, if you were happy that the compression was working well you could compress the entire data directory, and the already compressed files in output001 will not be re-compressed.

So, by default, nccompress '''does not overwrite the original files'''. If you invoke it without the '-o' option it will create compressed copies in the tmp.nc_compress subdirectory and leave them there, which will consume more disk space! This is a feature, not a bug, but you need to be aware that this is how it functions.

With large variables, which usually means large files (> 1GB) it is a good idea to specify a larger buffer size with the '-b' option, as it will run faster. On raijin this may mean you need to run interactively with a higher memory (~10GB) or submit it as a copyq job. A typical buffer size might be 1000 -> 5000 (1->5 GB).

It is also possible to use wildcards type operations, e.g.

 
<syntaxhighlight lang="text">
nccompress -r -o output*

nccompress -r -o output00[1-5]

nccompress -r -o run[1-5]/output*/ocean*.nc random.nc ice*.nc
</syntaxhighlight>
The nccompress program just sorts out finding files/directories etc, it calls nc2nc to do the compression. Using the '--nccopy' forces nccompress to use the nccopy program in place of nc2nc, though the netcdf package must already be loaded for this to work. You can tell nccompress to work on multple files simultaneously with the '-pa' option. By default this will use all the physical processors on the machine, or you can specify how many simultaneous processes you want to with '-np', e.g. <syntaxhighlight lang="text">
nccompress -r -o -np 16 run[1-5]/output*/ocean*.nc random.nc ice*.nc
</syntaxhighlight>
will compress 16 netCDF files at a time (the -np option implies parallel option). As each directory is processed before beginning on a new directory there will be little reduction in execution time if there are few netCDF files in each directory.
== nc2nc ==

The nc2nc program was written because no existing tool had a generalised per variable chunking algorithm. The total chunk size is defined to be the file system block size (4096KB). The dimensions of the chunk are sized to be as close as possible to the same ratio as the dimensions of the data, with the limits that no dimension can be less than 1. This chunking scheme performs well for a wide range of data, but there will always be cases for certain types of access, or variable shape that this is not optimal. In those cases a different approach may be required.

Be aware that nc2nc takes at least twice as long to compress an equivalent file as nccopy. In some cases with large files containing many variables it can be up to five times slower.

You can use nc2nc “stand alone”. It has a couple of extra features that can only be accessed by calling it directly:
<syntaxhighlight lang="text">
$ nc2nc -h
usage: nc2nc [-h] [-d {1-9}] [-m MINDIM] [-b BUFFERSIZE] [-n] [-v] [-c] [-f]
[-va VARS] [-q QUANTIZE] [-o]
origin destination

Make a copy of a netCDF file with automatic chunk sizing

positional arguments:
origin netCDF file to be compressed
destination netCDF output file

optional arguments:
-h, --help show this help message and exit
-d {1-9}, --dlevel {1-9}
Set deflate level. Valid values 0-9 (default=5)
-m MINDIM, --mindim MINDIM
Minimum dimension of chunk. Valid values 1-dimsize
-b BUFFERSIZE, --buffersize BUFFERSIZE
Set size of copy buffer in MB (default=50)
-n, --noshuffle Don't shuffle on deflation (default is to shuffle)
-v, --verbose Verbose output
-c, --classic use NETCDF4_CLASSIC output instead of NETCDF4 (default
true)
-f, --fletcher32 Activate Fletcher32 checksum
-va VARS, --vars VARS
Specify variables to copy (default is to copy all)
-q QUANTIZE, --quantize QUANTIZE
Truncate data in variable to a given decimal
precision, e.g. -q speed=2 -q temp=0 causes variable
speed to be truncated to a precision of 0.01 and temp
to a precision of 1
-o, --overwrite Write output file even if already it exists (default
is to not overwrite)
</syntaxhighlight>
With the vars option (-va) it is possible to select out only a subset of variables to be copied to the destination file. By default the output file is netCDf4 classic, but this can be changed to netCDF4 using the '-c' option. It is also possible to specify a minimum dimension size for the chunks (-m). This may be desirable for a dataset that has one particularly long dimension,. The chunk dimensions would mirror this and be very large in this direction . If fast access is required from slices orthogonal to this direction performance might be improved setting this option to a number greater than 1.
== ncvarinfo ==

ncvarinfo is a convenient way to get a summary of the contents of a netCDF file.
<syntaxhighlight lang="text">
./ncvarinfo -h
usage: ncvarinfo [-h] [-v] [-t] [-d] [-a] [-va VARS] inputs [inputs ...]

Output summary information about a netCDF file

positional arguments:
inputs netCDF files

optional arguments:
-h, --help show this help message and exit
-v, --verbose Verbose output
-t, --time Show time variables
-d, --dims Show dimensions
-a, --aggregate Aggregate multiple netCDF files into one dataset
-va VARS, --vars VARS
Show info for only specify variables

</syntaxhighlight>
By default it prints out a simple summary of the variables in a netCDF file, but omitting dimensions and time related variables. e.g. <syntaxhighlight lang="text">
ncvarinfo output096/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

</syntaxhighlight>
If you specify more than one file it will print the information for each file in turn <syntaxhighlight lang="text">
ncvarinfo output09?/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output097/ocean_daily.nc
Time steps: 365 x 1.0 days
output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude

output099/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude
</syntaxhighlight>
If the files have the same structure it is possible to aggregate the data and display it as if it were contained in a single dataset: <syntaxhighlight lang="text">
ncvarinfo -a output09?/ocean_daily.nc

Time steps: 1460 x 1.0 days
tau_x :: (1460, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (1460, 1080, 1440) :: j-directed wind stress forcing v-velocity
geolon_t :: (1080, 1440) :: tracer longitude
geolat_t :: (1080, 1440) :: tracer latitude
geolon_c :: (1080, 1440) :: uv longitude
geolat_c :: (1080, 1440) :: uv latitude
</syntaxhighlight>
You can also just request variables you are interested in to be output: <syntaxhighlight lang="text">
ncvarinfo -va tau_x -va tau_y output09?/ocean_daily.nc

output096/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output097/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output098/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity

output099/ocean_daily.nc
Time steps: 365 x 1.0 days
tau_x :: (365, 1080, 1440) :: i-directed wind stress forcing u-velocity
tau_y :: (365, 1080, 1440) :: j-directed wind stress forcing v-velocity
</syntaxhighlight>

 

[[Category:Data induction]]

Conda

2022-04-14T06:28:59Z

A.heerdegen:

 

== Conda Python Environments ==

CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.

You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.

To use any of the conda environments:

#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once)
#You must first run (to do at each session, a qsub job is a new session)

<code>module use /g/data/hh5/public/modules</code>

You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.

If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5
</syntaxhighlight>

Note that these <code><tt>conda</tt></code> environments will work on gadi login and compute nodes, and in the OOD cloud environment. They '''do not work '''on <code>accessdev</code>, as this has a very old system version that is no longer compatible.

=== Stable Environment ===

We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.

<code>module load conda/analysis3</code>

=== Unstable Environment ===

The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.

<code>module load conda/analysis3-unstable</code>

When we do our quarterly update the unstable environment becomes the new stable environment.

=== Removed Environments ===

Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].

== Creating personal environments ==

You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:

 
<syntaxhighlight lang="none">
auto_activate_base: false
envs_dirs:
- /scratch/$PROJECT/$USER/conda/envs
- /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
- /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
root-dir: /scratch/$PROJECT/$USER/conda/bld
</syntaxhighlight>

This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.

To create the conda environment, load the conda module, then deactivate it with

 
<syntaxhighlight lang="bash">
conda deactivate
conda env create ...
conda activate ...
</syntaxhighlight>

Just be aware that files on <code>scratch</code> might be deleted if no longer used, so it might be a good idea to export the <code>environment.yaml</code> file so that you can easily re-create the environment if that happens.

== Interactive Analysis / Jupyter ==

[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.

The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service [https://ood.nci.org.au https://ood.nci.org.au]. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:

*Module Directories: /g/data/hh5/public/modules
*Modules: conda/analysis3

The Centre has developed a script ('''<span style="font-family:Arial,Helvetica,sans-serif;">gadi_jupyter</span>'''<span style="font-family:Courier New,Courier,monospace;">)</span> that can run Jupyter on gadi compute nodes and display the notebook interface on your local computer. These scripts are available at [https://github.com/coecms/nci_scripts https://github.com/coecms/nci_scripts], see the instructions there for usage.

You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.

Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).

== Requesting new packages ==

You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.

It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment

As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.

== Update History ==

=== 22.01 (Current Stable) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://pygam.readthedocs.io/en/latest/ pygam] build Generalized Additive Models in Python
**[https://pyspectrum.readthedocs.io/en/latest/ spectrum] tools to estimate Power Spectral Densities based on Fourier transform, parametric methods or eigenvalues analysis
*'''Climate specific'''
**[https://xmhw.readthedocs.io/en/latest/ xmhw] xarray compatible Marine Heatwave Detection
**[https://argopy.readthedocs.io/en/latest/ argopy] library dedicated to [https://argopy.readthedocs.io/en/latest/what_is_argo.html#what-is-argo Argo] data access, manipulation and visualisation for standard users as well as Argo experts.
**[https://www.ilamb.org/doc/ ilamb] the International Land Model Benchmarking ([http://www.ilamb.org/ ILAMB]) project is a model-data intercomparison and integration project designed to improve the performance of land models
*'''Developer'''
**''''''[https://github.com/lidatong/dataclasses-json dataclasses-json] provides a simple API for encoding and decoding [https://docs.python.org/3/library/dataclasses.html dataclasses] to and from JSON
*'''Geospatial'''
**''''''[https://pysal.org pysal] spatial analysis library for open, cross platform geospatial data science

=== 21.10 (Unsupported) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays
**[https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset

*'''Geospatial'''
**[https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline
**[https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes
**[https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs

*'''Developer Tools'''
**[https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols
**[https://github.com/gorakhargosh/watchdog watchdog] monitor file system events
**[https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more
**[https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure
**[https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol
**[https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material
**[https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data
**[https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder

'''Known Issues'''

*MetPy is incompatible with the Matplotlib version in this environment [https://github.com/Unidata/MetPy/issues/2281 https://github.com/Unidata/MetPy/issues/2281]

=== <s>21.07</s> ===

Python has been updated to version 3.9. See what's new at [https://docs.python.org/3/whatsnew/3.9.html https://docs.python.org/3/whatsnew/3.9.html]

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning
***Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes
**[https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model

*'''Geospatial'''
**[https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models

*'''Visualisation'''
**[https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python

*'''Developer Tools'''
**[https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line

=== <s>21.04</s> ===

Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems
**[https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library
**[https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends

*'''Geospatial'''
**[https://xarray-spatial.org/ xarray-spatial] common raster analysis functions

*'''Visualisation'''
**[https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects

*'''Developer Tools'''
**[https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git
**[https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB
**[https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]
**[https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information

=== <s>21.01</s> ===

The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf

era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.

'''Notable new packages'''

*'''Statistics'''
**[https://ml.dask.org/index.html dask-ml] Scalable machine learning
**[https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit
**[https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data

*'''Geospatial'''
**[http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis
**[https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python

*'''Visualisation'''
**[https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation
**[https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation
**[https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps

*'''Developer Tools'''
**[https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup
**[https://github.com/psf/black black] Python code formatter
**[https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool

=== <s>20.10 (Unsupported)</s> ===

Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])

There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working

*pynio ([https://github.com/conda-forge/pynio-feedstock/issues/90 https://github.com/conda-forge/pynio-feedstock/issues/90])
*<s>cf_units</s> (was renamed to 'cf-units')
*<s>pymunge</s> (fixed)
*<s>ants</s> (fixed)

'''Notable new packages'''

*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib
*[https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries
*[https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution
*[https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios
*[https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data
*[https://github.com/jdowner/gist python-gist] a command line interface for working with github gists
*[https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo

=== <s>20.07 (Unsupported)</s> ===

'''Notable New Packages'''

*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles)
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output

=== <s>20.04 (Unsupported)</s> ===

Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes

Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI

'''Notable New Packages'''

*geopy: Locate lat/lon coordinates of places
*ninja: mom6 build system

=== <s>20.01 (Unsupported)</s> ===

Python has been updated to 3.7

'''Notable New Packages'''

*xlrd: Read excel files
*ants: Unified Model Ancillary tools
*climtas: Dask-aware Xarray timeseries processing

=== <s>19.10 (Unsupported)</s> ===

basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages

'''Notable New Packages'''

*addmeta
*[http://holoviews.org/ holoviews]: Fancy plots
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes

=== <s>19.07 (Unsupported)</s> ===

'''Notable New Packages'''

*pyferret
*pyngl
*pynio
*xgcm
*xrft

=== <s>19.04 (unsupported)</s> ===

'''Notable Changes'''

*arccssive has been renamed to clef
*A bug preventing Iris from opening UM files without a date has been fixed

'''Notable New Packages'''

*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py
*[https://github.com/intake/intake intake] Lightweight data catalogues
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays

[[Category:NCI]] [[Category:Python]]

Conda

2022-04-12T01:24:44Z

A.heerdegen: Updated conda versions

 

== Conda Python Environments ==

CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.

You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.

To use any of the conda environments:

#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once)
#You must first run (to do at each session, a qsub job is a new session)

<code>module use /g/data/hh5/public/modules</code>

You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.

If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5
</syntaxhighlight>

Note that these <code><tt>conda</tt></code> environments will work on gadi login and compute nodes, and in the OOD cloud environment. They '''do not work '''on <code>accessdev</code>, as this has a very old system version that is no longer compatible.

=== Stable Environment ===

We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.

<code>module load conda/analysis3</code>

=== Unstable Environment ===

The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.

<code>module load conda/analysis3-unstable</code>

When we do our quarterly update the unstable environment becomes the new stable environment.

=== Removed Environments ===

Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].

== Creating personal environments ==

You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:

 
<syntaxhighlight lang="none">
auto_activate_base: false
envs_dirs:
- /scratch/$PROJECT/$USER/conda/envs
- /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
- /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
root-dir: /scratch/$PROJECT/$USER/conda/bld
</syntaxhighlight>

This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.

To create the conda environment, load the conda module, then deactivate it with

 
<syntaxhighlight lang="bash">
conda deactivate
conda env create ...
conda activate ...
</syntaxhighlight>

Just be aware that files on <code>scratch</code> might be deleted if no longer used, so it might be a good idea to export the <code>environment.yaml</code> file so that you can easily re-create the environment if that happens.

== Interactive Analysis / Jupyter ==

[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.

The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service [https://ood.nci.org.au https://ood.nci.org.au]. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:

*Module Directories: /g/data/hh5/public/modules
*Modules: conda/analysis3

The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at [https://github.com/coecms/nci_scripts https://github.com/coecms/nci_scripts], see the instructions there for usage.

You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.

Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).

== Requesting new packages ==

You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.

It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment

As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.

== Update History ==

=== 22.01 (Current Stable) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://pygam.readthedocs.io/en/latest/ pygam] build Generalized Additive Models in Python
**[https://pyspectrum.readthedocs.io/en/latest/ spectrum] tools to estimate Power Spectral Densities based on Fourier transform, parametric methods or eigenvalues analysis
*'''Climate specific'''
**[https://xmhw.readthedocs.io/en/latest/ xmhw] xarray compatible Marine Heatwave Detection
**[https://argopy.readthedocs.io/en/latest/ argopy] library dedicated to [https://argopy.readthedocs.io/en/latest/what_is_argo.html#what-is-argo Argo] data access, manipulation and visualisation for standard users as well as Argo experts.
**[https://www.ilamb.org/doc/ ilamb] the International Land Model Benchmarking ([http://www.ilamb.org/ ILAMB]) project is a model-data intercomparison and integration project designed to improve the performance of land models
*'''Developer'''
**''''''[https://github.com/lidatong/dataclasses-json dataclasses-json] provides a simple API for encoding and decoding [https://docs.python.org/3/library/dataclasses.html dataclasses] to and from JSON
*'''Geospatial'''
**''''''[https://pysal.org pysal] spatial analysis library for open, cross platform geospatial data science

=== 21.10 (Unsupported) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays
**[https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset

*'''Geospatial'''
**[https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline
**[https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes
**[https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs

*'''Developer Tools'''
**[https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols
**[https://github.com/gorakhargosh/watchdog watchdog] monitor file system events
**[https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more
**[https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure
**[https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol
**[https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material
**[https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data
**[https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder

'''Known Issues'''

*MetPy is incompatible with the Matplotlib version in this environment [https://github.com/Unidata/MetPy/issues/2281 https://github.com/Unidata/MetPy/issues/2281]

=== <s>21.07</s> ===

Python has been updated to version 3.9. See what's new at [https://docs.python.org/3/whatsnew/3.9.html https://docs.python.org/3/whatsnew/3.9.html]

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning
***Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes
**[https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model

*'''Geospatial'''
**[https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models

*'''Visualisation'''
**[https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python

*'''Developer Tools'''
**[https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line

=== <s>21.04</s> ===

Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems
**[https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library
**[https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends

*'''Geospatial'''
**[https://xarray-spatial.org/ xarray-spatial] common raster analysis functions

*'''Visualisation'''
**[https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects

*'''Developer Tools'''
**[https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git
**[https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB
**[https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]
**[https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information

=== <s>21.01</s> ===

The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf

era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.

'''Notable new packages'''

*'''Statistics'''
**[https://ml.dask.org/index.html dask-ml] Scalable machine learning
**[https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit
**[https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data

*'''Geospatial'''
**[http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis
**[https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python

*'''Visualisation'''
**[https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation
**[https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation
**[https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps

*'''Developer Tools'''
**[https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup
**[https://github.com/psf/black black] Python code formatter
**[https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool

=== <s>20.10 (Unsupported)</s> ===

Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])

There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working

*pynio ([https://github.com/conda-forge/pynio-feedstock/issues/90 https://github.com/conda-forge/pynio-feedstock/issues/90])
*<s>cf_units</s> (was renamed to 'cf-units')
*<s>pymunge</s> (fixed)
*<s>ants</s> (fixed)

'''Notable new packages'''

*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib
*[https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries
*[https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution
*[https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios
*[https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data
*[https://github.com/jdowner/gist python-gist] a command line interface for working with github gists
*[https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo

=== <s>20.07 (Unsupported)</s> ===

'''Notable New Packages'''

*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles)
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output

=== <s>20.04 (Unsupported)</s> ===

Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes

Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI

'''Notable New Packages'''

*geopy: Locate lat/lon coordinates of places
*ninja: mom6 build system

=== <s>20.01 (Unsupported)</s> ===

Python has been updated to 3.7

'''Notable New Packages'''

*xlrd: Read excel files
*ants: Unified Model Ancillary tools
*climtas: Dask-aware Xarray timeseries processing

=== <s>19.10 (Unsupported)</s> ===

basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages

'''Notable New Packages'''

*addmeta
*[http://holoviews.org/ holoviews]: Fancy plots
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes

=== <s>19.07 (Unsupported)</s> ===

'''Notable New Packages'''

*pyferret
*pyngl
*pynio
*xgcm
*xrft

=== <s>19.04 (unsupported)</s> ===

'''Notable Changes'''

*arccssive has been renamed to clef
*A bug preventing Iris from opening UM files without a date has been fixed

'''Notable New Packages'''

*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py
*[https://github.com/intake/intake intake] Lightweight data catalogues
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays

[[Category:NCI]] [[Category:Python]]

Conda

2022-04-04T01:58:32Z

A.heerdegen: Updated packages in 22.01

 

== Conda Python Environments ==

CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.

You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.

To use any of the conda environments:

#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once)
#You must first run (to do at each session, a qsub job is a new session)

<code>module use /g/data/hh5/public/modules</code>

You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.

If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5
</syntaxhighlight>

Note that these <code><tt>conda</tt></code> environments will work on gadi login and compute nodes, and in the OOD cloud environment. They '''do not work '''on <code>accessdev</code>, as this has a very old system version that is no longer compatible.

=== Stable Environment ===

We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.

<code>module load conda/analysis3</code>

=== Unstable Environment ===

The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.

<code>module load conda/analysis3-unstable</code>

When we do our quarterly update the unstable environment becomes the new stable environment.

=== Removed Environments ===

Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].

== Creating personal environments ==

You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:

 
<syntaxhighlight lang="none">
auto_activate_base: false
envs_dirs:
- /scratch/$PROJECT/$USER/conda/envs
- /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
- /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
root-dir: /scratch/$PROJECT/$USER/conda/bld
</syntaxhighlight>

This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.

To create the conda environment, load the conda module, then deactivate it with

 
<syntaxhighlight lang="bash">
conda deactivate
conda env create ...
conda activate ...
</syntaxhighlight>

Just be aware that files on <code>scratch</code> might be deleted if no longer used, so it might be a good idea to export the <code>environment.yaml</code> file so that you can easily re-create the environment if that happens.

== Interactive Analysis / Jupyter ==

[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.

The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service [https://ood.nci.org.au https://ood.nci.org.au]. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:

*Module Directories: /g/data/hh5/public/modules
*Modules: conda/analysis3

The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at [https://github.com/coecms/nci_scripts https://github.com/coecms/nci_scripts], see the instructions there for usage.

You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.

Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).

== Requesting new packages ==

You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.

It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment

As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.

== Update History ==

=== 22.01 (Current Unstable) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://pygam.readthedocs.io/en/latest/ pygam] build Generalized Additive Models in Python
**[https://pyspectrum.readthedocs.io/en/latest/ spectrum] tools to estimate Power Spectral Densities based on Fourier transform, parametric methods or eigenvalues analysis
*'''Climate specific'''
**''''''[https://xmhw.readthedocs.io/en/latest/ xmhw] xarray compatible Marine Heatwave Detection
**[https://argopy.readthedocs.io/en/latest/ argopy] library dedicated to [https://argopy.readthedocs.io/en/latest/what_is_argo.html#what-is-argo Argo] data access, manipulation and visualisation for standard users as well as Argo experts.
**[https://www.ilamb.org/doc/ ilamb] the International Land Model Benchmarking ([http://www.ilamb.org/ ILAMB]) project is a model-data intercomparison and integration project designed to improve the performance of land models
*'''Developer'''
**''''''[https://github.com/lidatong/dataclasses-json dataclasses-json] provides a simple API for encoding and decoding [https://docs.python.org/3/library/dataclasses.html dataclasses] to and from JSON
*'''Geospatial'''
**''''''[https://pysal.org pysal] spatial analysis library for open, cross platform geospatial data science

=== 21.10 (Current Stable) ===

'''Notable new packages'''

*'''Statistics/ML'''
**[https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays
**[https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset

*'''Geospatial'''
**[https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline
**[https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes
**[https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs

*'''Developer Tools'''
**[https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols
**[https://github.com/gorakhargosh/watchdog watchdog] monitor file system events
**[https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more
**[https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure
**[https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol
**[https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material
**[https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data
**[https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder

'''Known Issues'''

*MetPy is incompatible with the Matplotlib version in this environment [https://github.com/Unidata/MetPy/issues/2281 https://github.com/Unidata/MetPy/issues/2281]

=== 21.07 (Unsupported) ===

Python has been updated to version 3.9. See what's new at [https://docs.python.org/3/whatsnew/3.9.html https://docs.python.org/3/whatsnew/3.9.html]

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning
***Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes
**[https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model

*'''Geospatial'''
**[https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models

*'''Visualisation'''
**[https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python

*'''Developer Tools'''
**[https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line

=== <s>21.04</s> ===

Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.

'''Notable new packages'''

*'''Statistics/ML'''
**[https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems
**[https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library
**[https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends

*'''Geospatial'''
**[https://xarray-spatial.org/ xarray-spatial] common raster analysis functions

*'''Visualisation'''
**[https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects

*'''Developer Tools'''
**[https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git
**[https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB
**[https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]
**[https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information

=== <s>21.01</s> ===

The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf

era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.

'''Notable new packages'''

*'''Statistics'''
**[https://ml.dask.org/index.html dask-ml] Scalable machine learning
**[https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit
**[https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data

*'''Geospatial'''
**[http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis
**[https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python

*'''Visualisation'''
**[https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation
**[https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation
**[https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps

*'''Developer Tools'''
**[https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup
**[https://github.com/psf/black black] Python code formatter
**[https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool

=== <s>20.10 (Unsupported)</s> ===

Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])

There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working

*pynio ([https://github.com/conda-forge/pynio-feedstock/issues/90 https://github.com/conda-forge/pynio-feedstock/issues/90])
*<s>cf_units</s> (was renamed to 'cf-units')
*<s>pymunge</s> (fixed)
*<s>ants</s> (fixed)

'''Notable new packages'''

*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib
*[https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries
*[https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution
*[https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios
*[https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data
*[https://github.com/jdowner/gist python-gist] a command line interface for working with github gists
*[https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo

=== <s>20.07 (Unsupported)</s> ===

'''Notable New Packages'''

*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles)
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output

=== <s>20.04 (Unsupported)</s> ===

Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes

Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI

'''Notable New Packages'''

*geopy: Locate lat/lon coordinates of places
*ninja: mom6 build system

=== <s>20.01 (Unsupported)</s> ===

Python has been updated to 3.7

'''Notable New Packages'''

*xlrd: Read excel files
*ants: Unified Model Ancillary tools
*climtas: Dask-aware Xarray timeseries processing

=== <s>19.10 (Unsupported)</s> ===

basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages

'''Notable New Packages'''

*addmeta
*[http://holoviews.org/ holoviews]: Fancy plots
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes

=== <s>19.07 (Unsupported)</s> ===

'''Notable New Packages'''

*pyferret
*pyngl
*pynio
*xgcm
*xrft

=== <s>19.04 (unsupported)</s> ===

'''Notable Changes'''

*arccssive has been renamed to clef
*A bug preventing Iris from opening UM files without a date has been fixed

'''Notable New Packages'''

*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py
*[https://github.com/intake/intake intake] Lightweight data catalogues
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays

[[Category:NCI]] [[Category:Python]]

Conda

2022-03-02T06:15:43Z

A.heerdegen: Add note that conda envs don't work on accessdev

== Conda Python Environments ==

CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.

You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.

To use any of the conda environments:

#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once)
#You must first run (to do at each session, a qsub job is a new session)

<code>module use /g/data/hh5/public/modules</code>

You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.

If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5
</syntaxhighlight>

Note that these <code><tt>conda</tt></code> environments will work on gadi login and compute nodes, and in the OOD cloud environment. They '''do not work '''on <code>accessdev</code>, as this has a very old system version that is no longer compatible.

=== Stable Environment ===

We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.

<code>module load conda/analysis3</code>

=== Unstable Environment ===

The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.

<code>module load conda/analysis3-unstable</code>

When we do our quarterly update the unstable environment becomes the new stable environment.

=== Removed Environments ===

Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].

== Creating personal environments ==

You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:

 
<syntaxhighlight lang="none">
auto_activate_base: false
envs_dirs:
- /scratch/$PROJECT/$USER/conda/envs
- /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
- /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
root-dir: /scratch/$PROJECT/$USER/conda/bld
</syntaxhighlight>

This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.

== Interactive Analysis / Jupyter ==

[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.

The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:

* Module Directories: /g/data/hh5/public/modules
* Modules: conda/analysis3

The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.

You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.

Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).

== Requesting new packages ==

You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.

It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment

As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.

== Update History ==

=== 22.01 (Current Unstable) ===

=== 21.10 (Current Stable) ===

'''Notable new packages'''

* '''Statistics/ML'''
** [https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays
** [https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset

* '''Geospatial'''
** [https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline
** [https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes
** [https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs

* '''Developer Tools'''
** [https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols
** [https://github.com/gorakhargosh/watchdog watchdog] monitor file system events
** [https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more
** [https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure
** [https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol
** [https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material
** [https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data
** [https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder

'''Known Issues'''

* MetPy is incompatible with the Matplotlib version in this environment https://github.com/Unidata/MetPy/issues/2281

=== 21.07 (Unsupported) ===

Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html

'''Notable new packages'''

* '''Statistics/ML'''
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model

* '''Geospatial'''
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models

* '''Visualisation'''
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python

* '''Developer Tools'''
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line

=== <s>21.04</s> ===

Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.

'''Notable new packages'''

* '''Statistics/ML'''
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends

* '''Geospatial'''
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions

* '''Visualisation'''
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects

* '''Developer Tools'''
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information

=== <s>21.01</s> ===

The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf

era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.

'''Notable new packages'''

* '''Statistics'''
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data

* '''Geospatial'''
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python

* '''Visualisation'''
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps

* '''Developer Tools'''
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup
** [https://github.com/psf/black black] Python code formatter
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool

=== <s>20.10 (Unsupported)</s> ===

Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])

There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working

*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)
*<s>cf_units</s> (was renamed to 'cf-units')
*<s>pymunge</s> (fixed)
*<s>ants</s> (fixed)

'''Notable new packages'''

*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo

=== <s>20.07 (Unsupported)</s> ===

'''Notable New Packages'''

*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles)
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output

=== <s>20.04 (Unsupported)</s> ===

Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes

Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI

'''Notable New Packages'''

*geopy: Locate lat/lon coordinates of places
*ninja: mom6 build system

=== <s>20.01 (Unsupported)</s> ===

Python has been updated to 3.7

'''Notable New Packages'''

*xlrd: Read excel files
*ants: Unified Model Ancillary tools
*climtas: Dask-aware Xarray timeseries processing

=== <s>19.10 (Unsupported)</s> ===

basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages

'''Notable New Packages'''

*addmeta
*[http://holoviews.org/ holoviews]: Fancy plots
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes

=== <s>19.07 (Unsupported)</s> ===

'''Notable New Packages'''

*pyferret
*pyngl
*pynio
*xgcm
*xrft

=== <s>19.04 (unsupported)</s> ===

'''Notable Changes'''

*arccssive has been renamed to clef
*A bug preventing Iris from opening UM files without a date has been fixed

'''Notable New Packages'''

*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py
*[https://github.com/intake/intake intake] Lightweight data catalogues
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays

[[Category:NCI]] [[Category:Python]]

Migrate experiment projects

2022-02-16T03:18:10Z

A.heerdegen:

This page contains information on what to do if you want to change project to work at NCI.

= Setting your default project =

To change the default project that jobs get submitted to when you run 'qsub', edit the file on Gadi '~/.config/gadi-login.conf'

If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'

In either case you will need to log out and back in again for the change to take effect

= Climate model experiments =

*[[Migrate_payu_experiment_project|Payu experiment]]
*[[Migrate_umui_experiment_project|UMUI experiment]]
*[[Migrate_rose_experiment_project|Rose/Cylc experiment]]
*[[Migrate_NUWRF_experiment_project|NUWRF experiment]]

WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.

= Tips for other scripts =

In your script, you may have used w35 or w48 in the following lines:

*PBS directive -P to specify the project
*PBS directive -l storage to load the storage space for these projects.
*Full paths of input or output data

It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.

It is important for you to make sure you know what changes you are doing, that's why we do not provide a magic <code>sed</code> one-liner command to change all the project IDs in all your files at once.

== Find files with the project listed in ==

To find all the files that contain the mention of <samp>w35</samp>, you can use the following command:
<pre>grep --exclude=*.nc -irlI w35 <directory_name> > files_to_update.txt</pre>

<samp>files_to_update.txt </samp>will contain a list of files where <samp>w35</samp> appear in at least one line. The command will not search in binary files and netCDF files and you can exclude more files from the search (e.g. figures) with the <code>--exclude=</code> option. Replace <samp><directory_name></samp> with the directory you want to search through.

At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only these:
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt</pre>

== Backup before modifications ==

You will want to create some sort of backup of your text files before changing them. <span style="color:#c0392b">DO NOT create copies. </span>We do not have heaps of free storage around to allow for it. Our recommended method is to use Git. If your files are not under the Git version control yet, here is a simple workflow that allows to record the state of your files before change. And some tips on comparing the modified and initial versions of the files.

Let's say, you want to change some files under a directory called <tt>my_scripts</tt>. You will first need to create a Git repository from that directory, record the current state of your files, do the changes you want to do, test the modified files work:
<pre># Create repository. Go in your my_scripts directory
git init

# Record current state
git add .
git commit -m "State at time of merge"

# Change your files as you require, test the new files</pre>

If something does not work, you can then use the following commands depending on what you want to do:
<pre># Difference between modified file and previous version
git diff <filename>

# Re-instate the previous version and delete the modified file
git checkout <filename></pre>

== Find all and replace all ==

All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality, including reviewing the changes before making the change, for your favourite text editor.

CLEX projects at NCI

2021-12-08T23:34:23Z

A.heerdegen:

The projects are:

 

{| style="width: 465px;"
|-
! Project code
! style="width: 214px;" | Research Program (RP)
! style="width: 151px;" | Lead CI
|-
| [https://my.nci.org.au/mancini/project/v45 v45]
| style="width: 214px;" | Ocean Extremes
| style="width: 151px;" | Andy Hogg
|-
| [https://my.nci.org.au/mancini/project/w40 w40]
| style="width: 214px;" | Weather and Climate interactions
| style="width: 151px;" | Martin Jucker
|-
| [https://my.nci.org.au/mancini/project/w42 w42]
| style="width: 214px;" | Attribution and Risk
| style="width: 151px;" | Abhnil Prasad
|-
| [https://my.nci.org.au/mancini/project/w97 w97]
| style="width: 214px;" | Drought
| style="width: 151px;" | Jason Evans
|}

w35 has been merged into w97

w48 has been merged into w40

Pre - 2022 project assignments:

{|
|-
! Project code
! Research Program (RP)
! Lead CI
|-
| v45
| Extra-tropical variability
| Andy Hogg
|-
| w35
| Drought
| Claire Carouge
|-
| w40
| Extreme Rainfall
| Todd Lane
|-
| w42
| Extreme Rainfall
| Steven Sherwood
|-
| w48
| Tropical Variability
| Dietmar Dommenget
|-
| w97
| Heatwaves
| Jason Evans
|}

=== Which project to join if part of several RPs ===

We recommend people join all relevant projects but choose one project to work from. This will help people to collaborate with researchers from all the RPs they are part of.

 

=== Setup default project on Gadi ===

*Check your default project. Connect to Gadi. Then type:

 
<syntaxhighlight lang="bash">echo $PROJECT
</syntaxhighlight>

*If the output from the previous step is not what you want you can [[Gadi#Swapping_Projects|change your default project]].

[[Category:Admin]]

Getting Help

2021-12-06T01:13:07Z

A.heerdegen:

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left\ hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*The CLEX slack workspace [https://arccss.slack.com arccss.slack.com]  
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The slack workspace is a good way to contact other CLEX members as well as the CMS team. It is best suited to shorter less complex queries in general. There are a number of channels focussing on different topics. When asking for help it may be that someone else from the Centre will have some useful feedback. It is best not to contact team members directly: we rotate support duties to allow time for other tasks, so asking in a general channel means the team member tasked with support, or who is best placed to answer the question, can assist.

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences. Simply outline your issue in an email and send it to [mailto:cws_help@nci.org.au cws_help@nci.org.au] which will open a help ticket on our helpdesk.

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Include all relevant information:
**what machine are you running on: your machine (what OS), Gadi at NCI, VDI at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working at NCI was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. Tell us if you load modules in your .bashrc or .bash_profile files (unrecommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

== What can users expect ==

Users can expect:

*to receive a reply to a query within 2 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

Getting Help

2021-11-10T22:24:23Z

A.heerdegen:

= Introduction =

Part of the role of the CMS team is to support members of CLEX with computational isues such as those arising with running models, accessing data and analysing data.

The CMS team supports a range of models (listed in the navigation bar on the left\ hand side of the page). We can provide assistance for other models, but this will be limited in scope and dependent on the availability of team resources. We also support analysis and other general computing tasks on NCI machines. We can provide general advice for activities undertaken at other sites, or on home machines, but this is necessarily limited as we have no control over, or familiarity with other systems.

There are a number of different ways to contact the CMS team to get support:  

*Climate Weather Science helpdesk [mailto:cws_help@nci.org.au cws_help@nci.org.au]
*The CLEX slack workspace [https://arccss.slack.com arccss.slack.com]  
*[[One-on-one_calls|One-on-one videocall]]
*In person with local [[CMS|CMS team member]]

The slack workspace is a good way to contact other CLEX members as well as the CMS team. It is best suited to shorter less complex queries in general. There are a number of channels focussing on different topics. When asking for help it may be that someone else from the Centre will have some useful feedback. It is best not to contact team members directly: we rotate support duties to allow time for other tasks, so asking in a general channel means the team member tasked with support, or who is best placed to answer the question, can assist.

The help desk email is the preferred method for requesting help, as it allows the CMS to triage any queries to reserve time of the CMS team members for other activities. It also allows better visibility of open requests for the CMS team especially for queries that take several days to solve. We also cover for team absences.

== Guidelines for help requests ==

First of all, do not apologise or belittle yourself for asking a question! We all had to start one day. We all have different backgrounds with different areas of expertise. And most of all, we are all humans! None of us will know everything, all of us will be forgetful. Asking someone else is the best way to learn what we don't know or to realise what we have forgotten.

When seeking assistance, there are some steps that should be taken to make the most efficient use of everyone's time:

*Include all relevant information:
**what machine are you running on: your machine (what OS), Gadi at NCI, VDI at NCI, accessdev at NCI or another server (give name)
**if working at NCI please supply your NCI user name
**if working at NCI was the issue in a login session or on the PBS compute nodes?
**upload any code to [https://gist.github.com gist.github.com] or for more complex examples [https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/create-a-repo create a GitHub repository]. If you cannot put the code on Github because of licensing, put it on NCI and give us the path. '''Do not send copies of your code.'''
**if relevant, specify the modules that were loaded, or environment variables that were set. Tell us if you load modules in your .bashrc or .bash_profile files (unrecommended practice!)
**if a command produces an error make sure to include the exact command run, including all arguments. The simplest is copy-paste.
**if a script produces an error make sure to include the exact name of the script and the exact way you execute the script. Copy-paste is the simplest here again.
**Add any relevant reference. It is unlikely you are referring to a paper-only resource, so include the link (or at least the URL), a quote from emails and webpage (with the address to the relevant page), forward emails, link to a paper... For example, if you say: "As I've seen on the wiki", "As per the article X" etc. you need to put the reference.
*Please check to make sure any specified directories and/or files [[Tips:_Custom_file_permissions_at_creation|have group permissions set]] such that they can be accessed. Assume we can access the shared filesystems at NCI: /scratch and /g/data for your group. If we can't, we'll tell you what to do. '''/home directories are per default personal! You will need to change permissions for us to access!'''
*Where appropriate, reduce your query to a minimal reproducible example (Stack Overflow has a [https://stackoverflow.com/help/minimal-reproducible-example nice description of what this means], note that the formatting requirements are redundant)
*Please '''do not''' send code or error messages as screenshots or poorly formatted text 
*Please submit new help requests for substantially new queries and not just append to existing queries. This makes sure any new query is visible to all team members, and can be triaged appropriately. We also have the satisfaction of completing a task rather than have it drag on interminably. Additionally, it provides us with a more organised database of queries and solutions.

To foster best practice software development for code housed in a GitHub repository (or similar) the following steps are preferred: 

*Create an issue in the repo for the bug/problem
*Add a test for the specific bug where appropriate
*Document solution in the issue and refer to commits/pull requests: this creates a tight link between discussion of the problem and potential solutions with the code changes made to address points raised in the discussion. This could be done by the CMS team member helping you when appropriate.
*Resolve issue through a pull request mechanism. This could be done by the CMS team member helping you when appropriate.
*Close issue when fix has been committed

== What can users expect ==

Users can expect:

*to receive a reply to a query within 2 hours during normal business hours
*have their problem solved if it is within scope, or a work-around suggested, and if not then an explanation of why their problem is currently not solvable
*where an update is required for a current ticket to wait no longer than a week during a non-holiday period

If any of the above expectations are not being met, contact the CMS team. We are only human and sometimes might inadvertently let slip some requests.

== Resources ==

Blog post on [https://codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question/ Writing the perfect question]  

   

Migrate payu experiment project

2021-10-28T01:21:58Z

A.heerdegen:

Instructions to migrate a [https://payu.readthedocs.io/en/latest/ payu] experiment from one project to another on [[Gadi|gadi]].  

Definitions:

*new project: the NCI project code for the project to which the experiment has been migrated (these instructions use <tt>w40</tt> as an example, replace <tt>w40</tt> with the appropriate project)
*control directory: location where the model is run (contains <tt>config.yaml</tt> file)
*model: the name of the model in your experiment (these instructions use <tt>access-esm</tt> as an example, replace <tt>access-esm</tt> with the appropriate model name)
*laboratory directory: location of the archive and work directories, e.g. <tt>/scratch/w40/user/access-esm</tt>, where <tt>user</tt> is your NCI username.

These instructions assume that the laboratory directory has already been copied from the <tt>/scratch</tt> directory of the original project to the scratch directory of the new project.

'''If''' the laboratory directory is at the top level of your <tt>/scratch</tt> directory, e.g. <tt>/scratch/w40/user/access-esm:</tt>
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">Edit the <tt>config.yaml</tt> in the control directory and add 
<code>shortpath: /scratch/w40</code>
 if there is no shortpath option, or change it to the new project code if there is already a <tt>shortpath</tt> option defined.</div>
'''Otherwise''', if the laboratory directory is located at some sub-directory path in your <tt>/scratch</tt> directory:
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;">Edit the <tt>config.yaml</tt> in the control directory and add the full path to the model laboratory directory 
<code>laboratory: /scratch/w40/user/subdirectorypath/access-esm</code> 
if there is already a shortpath option remove it</div>
Now remove the symbolic link to the archive directory in your control directory: 
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><code>rm archive</code></div>
 (make sure to not have a trailing <tt>/</tt> because the operating system thinks this refers to the directory it is linking to rather than the link itself) and run
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><code>payu setup --archive</code></div>
to test your paths are correct and create a new <tt>archive</tt>. If this is successful then
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><code>payu sweep</code></div>
  and the model is ready to run.  

Migrate payu experiment project

2021-10-28T01:04:00Z

A.heerdegen:

Migrate payu experiment project

2021-10-28T00:58:54Z

A.heerdegen: Initial version

Out of space

2021-10-18T01:28:57Z

A.heerdegen: Initial edit

You're out of space in your project on <tt>gadi</tt>, what now?   Broadly there are four options: delete, compress, move or archive

=== Delete ===

The best option is to delete files you no longer require e.g.

*failed model runs
*duplicated data (stored in another location) 
*intermediate data in analysis (delete intermediates but keep scripts to recreate if required)
*model fields not required (many models default to writing out many more fields than most people require or use)
*code (should be in a version controlled repository, e.g. [https://github.com GitHub])

=== Compress ===

All netCDF files you need to keep should be compressed. There is a detailed page on [[NetCDF_Compression_Tools|compression tools]].

If the number of inodes (files) is the quota that needs to be reduced then using tools like [[TAR_guidelines|<tt>tar</tt>]] and <tt>zip</tt> can be used to collect smaller files, such as text files, into a single archive file.

=== Move ===

Some projects have smaller allocations than others, or more free space. Assuming it is appropriate to do so then it may be legitimate to change the group to a different project using the <tt>chgrp</tt> command. Note that for the purposes of accounting at NCI the group of a file is used to determine which quota it counts against, not the physical location on disk.

There is some short term storage available under the Climate LIEF grant that is administed by the CMS Team. All elgible researchers associated with Climate research at NCI can [[Storage#Temporary_storage|apply for short term storage]].

Most home institutions have local storage options for data. This is also an option if it is not essential for the data to remain on NCI systems for further processing or analysis.

Note that when copying data from one location to another the best tool to use is rsync. Note that when using rsync the normal archive option (-a) will also change the group of the files after copying to be the same as they are in the original location. On NCI systems this can be undesirable dur to the way the accounting system works, so the best combination of options is
<pre>rsync -vrltoD --safe-links srcdir targetdir
</pre>

=== Archive ===

NCI has a tape-based data store (MDSS). This is by far the cheapest storage option at NCI, so also boasts the largest available capacity. In general this is where data is stored when it is no longer required for analysis. It is possible to retrieve archived data from MDSS, so it is feasible to archive data before completely finished analysing it, and keep only the fraction that is being actively analysed. It is not a suitable location for data files that need to be regularly accessed, e.g. model inputs, or for data that is likely to still be modified or updated.

There are [[Archiving_data|detailed instructions]] about archiving your data to the Mass Data Store on gadi.

Storage

2021-10-17T23:15:58Z

A.heerdegen: Updated to nci-files-report commands

<span style="font-size:medium;">During your work at the Centre, you are likely to produce, use and share data on different systems. You will probably have access to two different systems: your University system and NCI.</span>

= Storage at NCI =

<span style="font-size:medium;">NCI provides two types of storage: tape and disk. Tape is for long term storage while disk is more suited to store data you need to access often.</span>

== Tape at NCI ==

<span style="font-size:medium;">The tape system at NCI is called /massdata. Please read the [[Archiving_data|archiving data wiki page]] to learn how to use this system. Here are a few important points to keep in mind:</span>

*<span style="font-size:medium;">Tape is mostly appropriate for archiving data.</span>
*<span style="font-size:medium;">You should only store big files on tape. If you want to migrate a lot of small files, you should first archive them together. To learn how to do that please have a look at "File compression and archiving" section below and [mailto:climate_help@nci.org.au| email] your questions to us.</span>
*<span style="font-size:medium;">/massdata is only accessible from the login nodes (interactive) or via a script submitted to the copyq queue. It is generally recommended to use the copyq queue as you then have a much longer run time.</span>
*<span style="font-size:medium;">Tape access (writing and reading) is slow.</span>
*<span style="font-size:medium;">Considering data on /massdata are likely to be un-used for a long time it is quite essential to document your data. For example adding a detailed README file to your data folder can help a lot.</span>
*<span style="font-size:medium;">Storage update for quota is only updated daily, overnight for massdata because of the size of it. It is then recommended to act quickly for clean up and if possible before breaching the quota.</span>

=== '''<span style="font-size:medium;">Additional storage</span>''' ===

<span style="font-size:medium;">It might be possible to add quota on massdata for your project to do so please send an [mailto:climate_help@nci.org.au| email to us] detailing how much additional space you would like and which NCI project it is for. Again, being tape storage the request might take some days to be processed so please plan ahead.</span>

== '''Disk at NCI''' ==

<span style="font-size:medium;">There are three different disk filesystems at NCI, each with a slightly different purpose. NCI also has a [http://nci.org.au/user-support/getting-help/filesystem-user-guide/ Users' Guide]. All the disk filesystems are accessible from the login nodes and the compute nodes hence you can read/write to one while currently being in an other filesystem. And all these filesystems have access to massdata either through login nodes connection or sending a script to copyq queue.</span>

=== <span style="font-size:medium;">/home</span> ===

*<span style="font-size:medium;">This is your home directory.</span>
*<span style="font-size:medium;">This space is strictly limited at 10GB for each user but it is backed up.</span>
*<span style="font-size:medium;">It is most suitable for storing source code rather than model outputs or observation datasets.</span>
*<span style="font-size:medium;">You can monitor your use on home with the "quota" command.</span>

=== <span style="font-size:medium;">/scratch</span> ===

*<span style="font-size:medium;">All projects have some storage on /scratch.</span>
*<span style="font-size:medium;">The amount of space varies from a project to another.</span>
*<span style="font-size:medium;">The management of the space is left to the responsibility of the members of each project. .</span>
*<span style="font-size:medium;">When a project fills its quota on /short, the project's members will not be able to use the computing queues except for copyq queues to help with moving data around.</span>
*<span style="font-size:medium;">To monitor the overall usage, please use the "nci_account" command. To monitor the usage per user, please use "nci-files-report -f scratch".</span>
*<span style="font-size:medium;">Increasing the quota on /scratch for a project might be possible but is left to the decision of NCI staff. An extension can only be requested by the Lead CI for the project. You can check who the Lead CI is on my.nci.org.au.</span>

=== <span style="font-size:medium;">/g/data</span> ===

*<span style="font-size:medium;">Most projects now have some storage on /g/data</span>
*<span style="font-size:medium;">As for /scratch, the quota on /g/data is per project with management of the usage the sole responsibility of the project's members.</span>
*<span style="font-size:medium;">/g/data can be less stable than /scratch.</span>
*<span style="font-size:medium;">For compute nodes to have read and/or write access to this filesystem, you need to mount the specific area you need with the "-l storage" PBS flag</span>
*<span style="font-size:medium;">A project's quota on /g/data cannot be increased with a simple request. There is a review of some of the quotas every year, at which point some projects might be granted an increase. If you need additional storage before then, please consider deleting old or incorrect data, archiving old data to /massdata, using [[Storage#temporary|temporary storage]] or your [[Storage#local|University system]]. If you still want an increase to be considered at review time, please make sure to discuss it with the Lead CI of your project who will be part of the review.</span>
*<span style="font-size:medium;">To monitor usage, please use "nci_account". To see a per-user summary run the command "nci-files-report -f gdata"</span>

== '''Temporary storage''' ==

----

<span style="font-size:medium;">The CMS is also managing two projects on NCI that can be used for temporary storage of data. Both are mounted on /g/data and have the same characteristic as other /g/data storage space as explained above.</span>

<span style="font-size:medium;">To use any space on these projects''', you need to:'''</span>

*<span style="font-size:medium;">request [https://my.nci.org.au/mancini/login?next=/mancini/project/ connection to the project] if you are not yet a member. You can check which projects you are part of with the "groups" command.</span>
*<span style="font-size:medium;">fill a storage request using the [https://clex.dmponline.cloud.edu.au/ CLEX DMPonline tool]. If you do not yet have an account on this tool, please be patient for the account creation. To avoid robots and unauthorised access, the account creation requires human verification on our end. Please [mailto:cws_help@nci.org.au email us] if you have any question about filling in the form. Note this form is principally to enable us to monitor the space used, requested and available. It also enables us to prepare a folder for you with appropriate permissions. The forms are very short and quick to fill and the storage is usually ready for use within a few hours. See [[Storage-request|this page]] for more detailed instructions on how to fill the form. Note:</span>
**<span style="font-size:medium;">you do not need to email us, please just fill in the form to request the allocation you would like.</span>
**<span style="font-size:medium;">also, you can request space for use by a whole group instead of per user, but all users of the group '''must''' request connection to the project.</span>

<span style="font-size:medium;">The temporary storage projects are:</span>

*<span style="font-size:medium;">/g/data/hh5: this project is for short temporary use (~3 months). It could be used for example to print your raw model outputs, then you would save a subset or a reformatted version to your project's space and move the raw outputs to /massdata for safekeeping.</span>
*<span style="font-size:medium;">/g/data/ua8: the main purpose of this project is to store replica of datasets which do not have a specific data project assigned. However, the free space in this project can be used as temporary storage for data that is being processed for publication.</span>

= Storage at Universities =

----

<span style="font-size:medium;"><span style="line-height: 1.5;">at ANU</span></span>

<span style="font-size:medium;">at Monash</span>

<span style="font-size:medium;">at UMelb</span>

<span style="font-size:medium;">at UNSW</span>

<span style="font-size:medium;">at UTAS</span>

= File compression and archiving =

----

<span style="font-size:medium;">For an efficient use of storage, there are a few rules to keep in mind:</span>

*<span style="font-size:medium;">it is more efficient to store a few larger files than lots of smaller files. It is hard to define large and small but files of several tens of gigabytes are absolutely acceptable. The size of the files should clearly also take into account how you or others are going to use them. File nearing 100 GB become unmanageable and should be produced only if there is no other option. In that case you should read about ...</span>
*<span style="font-size:medium;">It is always best practice to compress your data when possible. Netcdf files are now easily compressible, see [[NetCDF_Compression_Tools|this wiki page]] for detailed explanations on tools available at NCI.</span>

<span style="font-size:medium;">To store small setup files that define your experiments, think about using the "tar" command. This is a shell command with a manual accessible through</span>
<syntaxhighlight lang="bash">
man tar
</syntaxhighlight>
<span style="font-size:medium;">This command will save many files together in a single archive, it can be used on a directory tree and will restore the directory structure when restoring the files from the archive. This means if you have several experiments you need to save the setup of, the best way might be to create a directory tree containing the setup files of all the experiments then create one single archive file for all. The archive files can also easily be compressed/uncompressed using the gzip utility either at the archive creation time or afterwards: see the [https://drive.google.com/drive/folders/0B1KncyYETiZqNGdHZ0VZSk02cUE 2017 training material] for usage of these commands and acls and tar cheat sheets. For more details see [[Archiving_data|Archiving Data]]  </span>
[[Category:Data induction]]

CMS

2021-10-17T22:50:53Z

A.heerdegen: Removed Annette from team

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Claire-Carouge-2-440x440.jpg</div> <div style="width:80%">
==== Claire Carouge ====

*Atmosphere and Land Surface Modelling
*[[WRF|WRF]]
*[[CABLE|CABLE]]
*[[JULES|JULES]]
</div> </div> <div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div>
=== Monash University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div> <div style="width:80%">
==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Scott-Wales-3-440x440.jpg</div> <div style="width:80%">
==== Scott Wales ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

CMS

2021-10-17T22:48:39Z

A.heerdegen: Reverted edits by A.heerdegen (talk) to last revision by C.carouge

== Our People ==

We are a distributed team with members sitting at each CLEX node

https://www.climateextremes.org.au/people/technical-staff/

=== Australian National University ===

<div style="display:flex; flex-direction:row">
<div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Claire-Carouge-2-440x440.jpg</div>
<div style="width:80%">
====Claire Carouge====
* Atmosphere and Land Surface Modelling
* [[WRF]]
* [[CABLE]]
* [[JULES]]
</div>
</div>

<div style="display:flex; flex-direction:row">
<div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div>
<div style="width:80%">

====Aidan Heerdegen====
* Ocean Modelling
* [[MOM]]
* [[Payu]]
</div>
</div>

<div style="display:flex; flex-direction:row">
<div style="width:20%; transform:scale(0.15); height:100px">https://climateextremes.org.au/wp-content/uploads/2018/03/annette-hirsch-sq_1_orig.png</div>
<div style="width:80%">
====Annette Hirsch====
* Land Surface Modelling
* [[WRF]]
* [[CABLE]]
</div>
</div>

=== Monash University ===

<div style="display:flex; flex-direction:row">
<div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Holger-Wolff-1-440x440.jpg</div>
<div style="width:80%">
====Holger Wolff====
* Atmosphere Modelling
* [[Unified Model]]
</div>
</div>

=== University of Melbourne ===

<div style="display:flex; flex-direction:row">
<div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Scott-Wales-3-440x440.jpg</div>
<div style="width:80%">
====Scott Wales====
* Atmosphere Modelling
* [[Unified Model]]
</div>
</div>

=== University of Tasmania ===

<div style="display:flex; flex-direction:row">
<div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div>
<div style="width:80%">
====Paola Petrelli====
* [[Data]]
</div>
</div>

== Contact ==
General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one calls | One-on-one videocall]]
[[Category:Admin]]

CMS

2021-10-17T22:47:58Z

A.heerdegen: Removed Annette from team

== Our People ==

We are a distributed team with members sitting at each CLEX node

[https://www.climateextremes.org.au/people/technical-staff/ https://www.climateextremes.org.au/people/technical-staff/]

=== Australian National University ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Claire-Carouge-2-440x440.jpg</div> <div style="width:80%">
==== Claire Carouge ====

*Atmosphere and Land Surface Modelling
*[[WRF|WRF]]
*[[CABLE|CABLE]]
*[[JULES|JULES]]
</div> </div> <div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Aidan-Heerdegen-1-1-440x440.jpg</div> <div style="width:80%">
==== Aidan Heerdegen ====

*Ocean Modelling
*[[MOM|MOM]]
*[[Payu|Payu]]
</div> </div> <div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.15); height:100px">https://climateextremes.org.au/wp-content/uploads/2018/03/annette-hirsch-sq_1_orig.png</div> <div style="width:80%">

==== Holger Wolff ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Melbourne ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.2); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Scott-Wales-3-440x440.jpg</div> <div style="width:80%">
==== Scott Wales ====

*Atmosphere Modelling
*[[Unified_Model|Unified Model]]
</div> </div>
=== University of Tasmania ===
<div style="display:flex; flex-direction:row"><div style="width:20%; transform:scale(0.5); height:100px">https://www.climateextremes.org.au/wp-content/uploads/2018/04/Paola-Petrelli-headshot.jpg</div> <div style="width:80%">
==== Paola Petrelli ====

*[[Data|Data]]
</div> </div>
== Contact ==

General enquiries and help questions: [mailto:cws_help@nci.org.au cws_help@nci.org.au]

Quick help addressed to CMS and the Centre's community: [https://arccss.slack.com arccss.slack.com]

[[One-on-one_calls|One-on-one videocall]]

[[Category:Admin]]

Accounting at NCI

2021-10-06T00:35:02Z

A.heerdegen: Added in ncdu and other info from deleted NCI Guidelines Monitoring page

This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by NCI and the CMS team.

= Grafana =

CMS has put in place a Grafana server for visualising a range of accounting statistics for CLEx. You can access this server using your NCI credentials at: [https://accessdev.nci.org.au/grafana/login https://accessdev.nci.org.au/grafana/login]

Unfortunately, we haven't yet put in place the collection of all the statistics for Gadi. Currently, you can see timeseries of your storage and SU usage on the [https://accessdev.nci.org.au/grafana/d/toeLAYDWz/user-report?orgId=1 User Report].

We will update this section as more statistics become available.

= Computing resources =

== nci_account ==

You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. Using the <code>-v</code> option, you can see the usage per user:

nci_account -v

== nqstat ==

<code>qstat</code> allows you to monitor your jobs at NCI. <code>nqstat</code> allows you to see the jobs currently submitted:

*for a given project
*for a given queue and project
*for a given user and project

== uqstat ==

The information output by <code>qstat</code> and <code>nqstat</code> isn't always enough. CMS has developed <code>uqstat</code> to output more information by default such as the job efficiency (cpu%), the queueing time, the walltime, the cost in SU etc.

To use this command, please load the <code>nci-scripts</code> module:

module use /g/data/hh5/public/modules
module load nci-scripts

Contrary to what we usually recommend, it should be safe to load this module in your .bashrc file so this module is loaded by default.

uqstat

The <code>-x</code> option allows you to see the same information on current jobs and jobs finished up to 24 hours previously.

== qcost ==

CMS has also developed <code>qcost</code> a tool to help you choose the best PBS configuration for your job. <code>qcost </code>calculates the cost, in SU, for a job submitted to the PBS system. The PBS [https://opus.nci.org.au/display/Help/Queue+Limits queue information is provided by NCI] but it can be tedious to determine which configuration of queue and memory request should be used to minimise job cost. qcost was created to make this process easier.

To use this command, please load the <code>nci-scripts</code> module (see above). Usage:

qcost -h
 usage: qcost [-h] -q QUEUE -n NCPUS -m MEM [-t TIME]

Return the cost (in SUs) for a PBS job submitted on gadi. No checking is done to ensure requests are within queue limits.

optional arguments:
-h, --help show this help message and exit
-q QUEUE, --queue QUEUE PBS queue
-n NCPUS, --ncpus NCPUS Number of cpus
-m MEM, --mem MEM Requested memory
-t TIME, --time TIME Job walltime (hours)

e.g. qcost -q normal -n 4 -m 60GB -t 3:00:00

Note that if no time is specified it defaults to 1 hour (1:00:00). Walltime can be specified as <tt>H:M</tt> or <tt>H:M:S</tt>. Memory must be specified in units of bytes (B), e.g. <tt>160GB</tt>, <tt>2000MB</tt>. For example:
<pre> qcost -q normal -n 4 -m 60GB -t 3:00:00
90 SU

qcost -q express -n 4 -m 60GB -t 3:00:00
270 SU
</pre>

= Storage resources =

== lquota ==

This command will list the quota and usage for all the projects you are a member of on both /scratch and /g/data. It will give usage and quota for both space and number of files.

== nci-files-report ==

This command will list the usage in both space and number of files along different dimensions depending on options.

=== Usage owned by a project per user ===

If you want to know who owns files in your project or where those files are, you should use:

nci-files-report --group w35 --filesystem gdata

This is probably the most useful options for the project's managers to find who has the most data owned by a project.

=== Usage in a project directory per user ===

If you want to know who owns files in the main directory of one of your projects, you should use:

nci-files-report --project w35 --filesystem gdata

This is probably the least useful option for this function.

=== Usage per user ===

If you want to know your data footprint across all your projects in a filesystem, you should use:

nci-files-report --user --filesystem gdata

'''Why are the totals different in <code>nci-files-report</code> and Grafana?'''

CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project.

= ncfind and nccompress =

----

These utilities are maintained by the CMS team. These allow you to find uncompressed netcdf files and to easily compress them. You'll find a complete documentation on [[NetCDF_Compression_Tools|this wiki page]]

= du, find and wc =

----

These commands can help you identify where in your area are the large files or which directories contain lots of files. These are Unix commands and all have man pages for information.

Useful options:

*Check the sizes of sub-directories:

du -shc *

*Find files larger than X

find . -type f -size +100M

*Count number of files and subdirectories in directory

ls -1 | wc -l

Note: the Internet is the perfect source for commands and options variations to find this type of information on your files and directories.

= ncdu =

[https://dev.yorhel.nl/ncdu ncdu] is a disk usage analyzer with an ncurses interface. It is designed to find space hogs and aims to be fast, simple and easy to use.

This command is part of the conda environment maintained by CMS. To use:

module purge
module use /g/data3/hh5/public/modules
module load conda/analysis3

To invoke run ncdu directory_path and replace directory_path with a path to a directory you wish to check.

Some example output:

ncdu /g/data3/hh5/public/apps

Gives a screen items in the directory sorted by size:

--- /g/data3/hh5/public/apps ----------------------------------------------------
64.0 GiB [##########] /miniconda3
799.6 MiB [ ] /lrose
152.3 MiB [ ] /easybuild
4.0 KiB [ ] README

Total disk usage: 64.9 GiB Apparent size: 64.6 GiB Items: 2461484

Pushing <tt><g></tt> twice changes the way the usage is displayed

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /g/data3/hh5/public/apps ----------------------------------------------------
64.0 GiB [ 98.6% ##########] /miniconda3
799.6 MiB [ 1.2% ] /lrose
152.3 MiB [ 0.2% ] /easybuild
4.0 KiB [ 0.0% ] README

Total disk usage: 64.9 GiB Apparent size: 64.6 GiB Items: 2461484

Select a directory, push <tt><return></tt> and it will show you a view of that directory sorted by size

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /g/data3/hh5/public/apps/miniconda3 ------------------------------------------
/..
33.2 GiB [ 51.9% ##########] /pkgs
22.6 GiB [ 35.3% ###### ] /envs
11.8 GiB [ 18.5% ### ] /old-envs
4.6 GiB [ 7.1% # ] /conda-bld
236.6 MiB [ 0.4% ] /lib
8.7 MiB [ 0.0% ] /share
6.1 MiB [ 0.0% ] /bin
5.1 MiB [ 0.0% ] /include
4.9 MiB [ 0.0% ] /conda-meta
4.3 MiB [ 0.0% ] /ssl
1.9 MiB [ 0.0% ] /compiler_compat
28.0 KiB [ 0.0% ] /etc
12.0 KiB [ 0.0% ] /x86_64-conda_cos6-linux-gnu
4.0 KiB [ 0.0% ] /locks
4.0 KiB [ 0.0% ] LICENSE.txt

Total disk usage: 64.0 GiB Apparent size: 63.7 GiB Items: 2446705

Select <tt>/..</tt> at the top to move back up the directory tree. In this way it is possible to navigate the directory structure and pinpoint the largest users of disk space. The disk usage information is cached, so once it has analysed a directory tree it is very fast to navigate.

Press <q>to quit the program.</q>

[[Category:NCI]]

Training

2021-09-06T05:11:32Z

A.heerdegen:

== Training Needs ==

=== General ===

*[[NCI|Accessing and using NCI]]
*[[Programming|Programming]]
*[[ProgrammingTools|ProgrammingTools]]

== CMS Presentations ==

CMS used to run weekly training sessions presenting computational and data related topics. Now we run a fortnightly [[CodeBreak|Code Break sessions]].

To get updates on our next training sessions '''[https://www.lists.unsw.edu.au/mailman/listinfo/training-coe sign up]''' to our email list

All our training videos are available at our '''[https://www.youtube.com/channel/UCSmoK6oWV9O0Hmyt9UdDNsQ Youtube channel]'''

*[https://www.youtube.com/watch?v=aYukDeA-8hE Python Basics] ([https://www.youtube.com/watch?v=-UktTJDh_48 Part 2]) ([https://www.youtube.com/watch?v=pCabwrUa26E Part 3])
*[https://www.youtube.com/watch?v=2GmFf9x1qg0 Introduction to NCI]
*[https://youtu.be/5z75iHJyT8Q Large Dataset Analysis with Jupyter]
*[https://www.youtube.com/watch?v=aNUe-ukitpU Jupyter Introduction]
*[https://www.youtube.com/watch?v=pk-UD_4ZAcE WRF at NCI]
*[https://www.youtube.com/watch?v=DzbtG3UiMUw WRF Q&A]
*[[Training:NCI_Storage_Monitoring|NCI Storage Monitoring]] 2019-10-02
*[https://youtu.be/4tzug0DMZVI Licensing for Data and Software] 2019-07-10
*[[Training:_Basic_Python|Basic Python]] 2019-06-05 - 2019-07-03
*[[Training:Big_Data|Analysing Big Data]] 2019-05-15 - 2019-05-22
*[https://youtu.be/gYKc1mJitEw Data at NCI] 2019-05-08
*[[Git_Introduction_old|Git Introduction old]] 2019-04-10 - 2019-04-17
*[[Git_Introduction|Git Introduction]] 2020-09-09 - 2020-09-16
*[[Bash_Introduction|Bash Introduction]] 2019-03-13 - 2019-04-03
*[[WRF_Introduction|WRF Introduction]] 2019-10-23 - 2019-10-30
*[http://climate-cms.wikis.unsw.edu.au/Category:Setting_up Setting up]

== Software Carpentry ==

For general programming training we recommend the material from [https://software-carpentry.org/ Software Carpentry], a group whos focus is teaching researchers programming skills. Check with your University if they run Software Carpentry Workshops.

[[Category:Training]]