Scratch file expiry

2022-04-26T07:00:31Z

C.carouge: Created page with "On 17th May 2022, NCI is introducing an automatic system to purge unused data from the scratch filesystem.   Below we are giving some steps on how best to prepare for tha..."

On 17th May 2022, NCI is introducing an automatic system to purge unused data from the scratch filesystem.   Below we are giving some steps on how best to prepare for that change. We also give some information on some useful tools to prepare a good data workflow going forward.
= Preparation =

Your best course of action is to start preparing now to avoid having any useful data going into quarantine on the 17th May. Below are the steps we recommend you to follow:

*Read the information provided by NCI: [https://opus.nci.org.au/pages/viewpage.action?pageId=156434436]
*Clean up /g/data: delete what you can, archive to tape or outside NCI what you can.
*Clean up /scratch: delete what you can, move to /g/data or tape or outside NCI if you need long term storage without accessing it.
*Run “nci-file-expiry list-warnings -p <project> > expiry_warning_<project>.txt” for all the projects you are a member.
*Check the output of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Run “nci-file-expiry list-warnings” : this will catch any file you own in any project you may have forgotten about.
*Check the output of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Rethink your data pipeline: do not leave data you are not using anymore in /scratch. Decide what you need to do with it when you stop using it: delete, move to /g/data or outside NCI or archive to tape.
*Manage your data under /g/data regularly: review your data, archive to tape or outside NCI as necessary.

= Additional information =

Below is a list of resources you might find useful to prepare for the automatic file expiry and build a good data workflow for the future.

[[Archiving_data|Archiving data at NCI]]

[https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html Blog on building a sustainable data workflow]

[[Storage|Description of the various filesystems at NCI]]

/scratch file expiry

2022-04-26T06:52:21Z

C.carouge: Created page with "On 17th May 2022, NCI is introducing an automatic process to delete files from /scratch that have not been accessed in a long time.   In preparation for this, we share he..."

On 17th May 2022, NCI is introducing an automatic process to delete files from /scratch that have not been accessed in a long time.   In preparation for this, we share here some relevant information.
= Preparation =

Your best course of action is to prepare before the 17th May so you avoid files you want to keep being placed into quarantine. Below are steps we recommend for you to take:

*Read the information provided by NCI: [https://opus.nci.org.au/pages/viewpage.action?pageId=156434436]
*Clean up /g/data: delete what you can, archive to tape or outside NCI what you can.
*Clean up /scratch: delete what you can, move to /g/data or tape or outside NCI if you need long term storage without accessing it.
*Run “nci-file-expiry list-warnings -p <project> > expiry_warning_<project>.txt” for all the projects you are a member.
*Check the output of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Run “nci-file-expiry list-warnings” : this will catch any file you own in any project you may have forgotten about.
*Check the output of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
*Rethink your data pipeline: do not leave data you are not using anymore in /scratch. Decide what you need to do with it when you stop using it: delete, move to /g/data or outside NCI or archive to tape.
*Manage your data under /g/data regularly: review your data, archive to tape or outside NCI as necessary.

= Additional resources =

[[Archiving_data|Archiving]]

[https://climate-cms.org/posts/2022-04-26-storage-where-what-why-how.html Blog on creating a sustainable data workflow]

[[Storage|Description of filesystems at NCI]]

Storage

2022-04-26T06:38:45Z

C.carouge:

Storage

2022-04-26T02:08:33Z

C.carouge:

How to use MDSS tape storage at NCI

2022-04-26T01:58:28Z

C.carouge:

All universities have their own archives where you can store your data, for more information contact your Library or IT department. Here we will focus on archiving data using the NCI archive storage, for all cases where using a university archive is not applicable.

Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation.

= MDSS proper usage =

MDSS is designed for medium to long-term archive of large files, this means it is optimised for storing big amounts of data, but not for retrieving this data frequently. This means it is most suitable for:

*Files you are required to keep for a long term, like data underlining published datasets, publications, PhD thesis etc.
*Files that you or someone else are likely to reuse or analyse again in the future but not in the next few months. For example, restart files or other model output you are not immediately using should be moved from disk to massdata as soon as possible.
*MDSS is suitable for backup of big data projects, like model output which could not be backed up elsewhere.

= Preparing your data for mdss =

#Organise your files and delete anything which you will not be re-using. Do not transfer data before organising it. It is difficult to get a list of what is stored on massdata, let alone to list what is in a tarred file once it is uploaded. 
#Big files: use tools like <tt>tar</tt> to bundle files together into archive files. Create reasonably big archive files but also think of how you might want to access the data later. There is no point of tarring together two different simulations if you would want to access them separately, as then you would need to transferred back a big amount of data you do not need. Your upload will fail if any of your files are less than 20MB or the average size is less than 250 MB.
#Files should be group readable, with group execute permissions for directories. This helps with long term maintenance, allowing administrators to track the type and size of archived data. You can change the permissions on data you own with the <tt>chmod</tt> unix utility.

While you are preparing your data to be moved it is an opportunity to also document it, if you have not done so already. You should document what you are archiving and how you are archiving it. Even a simple readme file added to your main directory can help others and your future self. If you are archiving data underlining a publication or published dataset then it is important to have a summary of what is stored in /massdata. This is part of the [[Data_Management_Plan|dataset management plan]] and/or [[Data_Availability_Statement|data availiability statement]].

Useful tools:

         [[TAR_guidelines|TAR- to create archives]] cheatsheet

         [[NetCDF_Compression_Tools|Compressing tools]]

= Accessing MDSS =

Massdata cannot be accessed directly via a directory path. All access of MDSS is via the command '''<tt>mdss</tt>.'''

Users connected to the project have read, write and execute permissions in the corresponding directory on mdss and so may create their own files in it.

<tt>mdss</tt> has several sub-commands and options to see all of them use either:
<syntaxhighlight lang="bash"> $mdss --help
or
$man mdss
</syntaxhighlight>

If you don't specify a project, it will use your default project. Then you add a sub-command and the path of the files and directories you want to upload, list etc.

mdss -P <project-id> + <sub-command> + <path>

Most useful sub-commands are:
<syntaxhighlight lang="bash">mdss put - upload files
mdss get - retrieve files
mdss ls - list directories and files
mdss dmdu - get the size of a directory/file
mdss dmls - show what is on cache and what is on tape

NB "mdss du" will also work but only return the size of what is still cached, dmdu will give the full size of what is on tape regardless if it is cached or not.
</syntaxhighlight>

Please note <tt>mdss</tt> sub-commands work only interactively or on the <tt>copyq </tt>queue. To use it on <tt>copyq</tt> remember to set the storage flag as
<pre>-l storage=massdata/<project_code></pre>

= Monitoring MDSS usage =

Unfortunately, there is no command to check the usage by user-id as for /g/data and /scratch. The only way to get this information currently is to ask help<at>nci.org.au. The NCI administrators can access this information for any CI of the group.

= Transferring data to and from MDSS =

NCI also supports the <tt>netmv</tt> and <tt>netcp</tt> commands to work with MDSS. These commands create a copyq job to transfer multiple files. Files can be automatically tarred and compressed as part of the copy process.

'''Warning: The automatic archiving and compression of these tools can use a lot of storage on /scratch if you're moving lots of data!'''

For more info run `man netmv`.

The CMS team has also developed a utility called [https://github.com/coecms/mdssdiff mdssdiff] available from [[Conda|our conda environments]]. This utility allows users to compare the contents of the local directory and a directory under /massdata. It will also recursively update the content on the massdata directory to copy the local directory or vice versa.

= Modifications to MDSS datasets =

Ask the Lead CI of the project to contact NCI at help@nci.org.au if large metadata operations are needed on massdata, such as changing ownership, project code, permissions etc. of existing datasets

[[Category:NCI Guidelines]] [[Category:Data induction]]

2022-04-25T23:21:17Z

C.carouge:

Storage

2022-04-25T23:18:21Z

C.carouge:

Storage

2022-04-22T00:58:17Z

C.carouge:

During your work at the Centre, you are likely to produce, use and share data on different systems. You will probably have access to two different systems: your University system and NCI.

= Storage on Gadi =

On Gadi, NCI has four different filesystems: $HOME, /scratch, /g/data and mdss. mdss is a tape-based filesystem while $HOME, /scratch and /g/data are disk-based.

There are also specific filesystems linked to the Open On Demand system which are less used. On this page, we will only talk about the storage available from Gadi.

== Hierarchy of filesystems ==

The different filesystems are best thought of as a hierarchy, each with a specific usecase:

*$HOME: this is a small disk space that is backed up by NCI. It is best suited for important, hard to reproduce files, e.g. data analysis codes.
*/scratch: this is a temporary disk space. Files are automatically deleted on some condition. It is best suited for raw output from climate models.
*/g/data: this is a permanent disk space. It is best suited for data that is used over a long time, e.g. climate models inputs, climate models outputs that is being analysed, climate models source code that needs to be compiled.
*mdss: this is a tape space. It is best suited as a backup of important files or an archiving system.

=== $HOME ===

*This is your home directory on Gadi.
*Unaccessible from OOD
*This space is strictly limited at 10GB for each user but it is backed up by NCI.
*It is most suitable for storing source code rather than model outputs or observation datasets. We still encourage you to also use Git to manage your source code and to save it to Github.
*You can monitor your use on home with the "quota" command. You may have errors at log in when your home directory is full.

=== /scratch ===

{| style="height: 50px; width: 800px" cellspacing="4" cellpadding="4" border="3" align="center"
|-
| Warning: this is a temporary space with [https://opus.nci.org.au/pages/viewpage.action?pageId=156434436 automatic deletion]. All users are responsible to learn about the deletion process.
|}

*All projects have some storage on /scratch.
*Unaccessible from OOD
*The amount of space varies from a project to another.
*The total space allocation is shared between all project members.
*The management of the space is left to the responsibility of the members of each project.
*There is [https://opus.nci.org.au/pages/viewpage.action?pageId=156434436 an automated file management system] in place. It will automatically remove any file that has not been accessed in the last 100 days.
*When a project fills its quota on /scratch, the project's members will not be able to use the computing queues except for copyq queues to help with moving data around.
*To monitor the overall usage, please use the <tt>nci_account</tt> command. To monitor the usage per user, please use <tt>nci-files-report -f scratch</tt>.

'''Additional storage'''

Increasing the quota on /scratch for a project might be possible but is left to the decision of NCI staff. An extension can only be requested by the Lead CI for the project. You can check who the Lead CI is on [https://my.nci.org.au/ my.nci.org.au].

=== /g/data ===

*The quota on /g/data is per project with management of the usage the sole responsibility of the project's members.
*Accessible from OOD
*For compute nodes to have read and/or write access to this filesystem, you need to mount the specific area you need with the "-l storage" PBS flag
*To monitor usage, please use <tt>nci_account</tt>. To see a per-user summary run the command <tt>nci-files-report -f gdata</tt>

'''Additional storage'''

A project's quota on /g/data cannot be increased with a simple request. There is a review of some of the quotas every year, at which point some projects might be granted an increase. If you need additional storage before then, please consider deleting old or incorrect data, archiving old data to /massdata, using [[Storage#temporary|temporary storage]] or your [[Storage#local|University system]]. If you still want an increase to be considered at review time, please make sure to discuss it with the Lead CI of your project who will be part of the review.

=== Tape at NCI ===

The tape system at NCI is called /massdata. Please read the [[Archiving_data|archiving data wiki page]] to learn how to use this system.

*Tape access (writing and reading) is slow.
*Unaccessible from OOD
*Tape is mostly appropriate for backing up or archiving data.
*You should only store big files on tape. If you want to migrate a lot of small files, you should first archive them together. To learn how to do that please have a look at "File compression and archiving" section below and [mailto:climate_help@nci.org.au| email] your questions to us.
*/massdata is only accessible from the login nodes (interactive) or via a script submitted to the copyq queue. It is generally recommended to use the copyq queue as you then have a much longer run time.
*Considering data on /massdata are likely to be unused for a long time it is quite essential to document your data. For example adding a detailed README file to your data folder can help a lot.

 

== '''Disk at NCI''' ==

There are three different disk filesystems at NCI, each with a slightly different purpose. NCI also has a [http://nci.org.au/user-support/getting-help/filesystem-user-guide/ Users' Guide]. All the disk filesystems are accessible from the login nodes and the compute nodes hence you can read/write to one while currently being in an other filesystem. And all these filesystems have access to massdata either through login nodes connection or sending a script to copyq queue.  

== '''Temporary storage''' ==

The CMS is also managing two projects on NCI that can be used for temporary storage of data. Both are mounted on /g/data and have the same characteristic as other /g/data storage space as explained above.

To use any space on these projects''', you need to:'''

*request [https://my.nci.org.au/mancini/login?next=/mancini/project/ connection to the project] if you are not yet a member. You can check which projects you are part of with the "groups" command.
*fill a storage request using the [https://clex.dmponline.cloud.edu.au/ CLEX DMPonline tool]. If you do not yet have an account on this tool, please be patient for the account creation. To avoid robots and unauthorised access, the account creation requires human verification on our end. Please [mailto:cws_help@nci.org.au email us] if you have any question about filling in the form. Note this form is principally to enable us to monitor the space used, requested and available. It also enables us to prepare a folder for you with appropriate permissions. The forms are very short and quick to fill and the storage is usually ready for use within a few hours. See [[Storage-request|this page]] for more detailed instructions on how to fill the form. Note:
**you do not need to email us, please just fill in the form to request the allocation you would like.
**also, you can request space for use by a whole group instead of per user, but all users of the group '''must''' request connection to the project.

The temporary storage projects are:

*/g/data/hh5: this project is for short temporary use (~3 months). It could be used for example to print your raw model outputs, then you would save a subset or a reformatted version to your project's space and move the raw outputs to /massdata for safekeeping.
*/g/data/ua8: the main purpose of this project is to store replica of datasets which do not have a specific data project assigned. However, the free space in this project can be used as temporary storage for data that is being processed for publication.

= Storage at Universities =

----

at ANU

at Monash

at UMelb

at UNSW

at UTAS

= File compression and archiving =

----

For an efficient use of storage, there are a few rules to keep in mind:

*it is more efficient to store a few larger files than lots of smaller files. It is hard to define large and small but files of several tens of gigabytes are absolutely acceptable. The size of the files should clearly also take into account how you or others are going to use them. File nearing 100 GB become unmanageable and should be produced only if there is no other option. In that case you should read about ...
*It is always best practice to compress your data when possible. Netcdf files are now easily compressible, see [[NetCDF_Compression_Tools|this wiki page]] for detailed explanations on tools available at NCI.

To store small setup files that define your experiments, think about using the "tar" command. This is a shell command with a manual accessible through
<syntaxhighlight lang="bash">
man tar
</syntaxhighlight>
This command will save many files together in a single archive, it can be used on a directory tree and will restore the directory structure when restoring the files from the archive. This means if you have several experiments you need to save the setup of, the best way might be to create a directory tree containing the setup files of all the experiments then create one single archive file for all. The archive files can also easily be compressed/uncompressed using the gzip utility either at the archive creation time or afterwards: see the [https://drive.google.com/drive/folders/0B1KncyYETiZqNGdHZ0VZSk02cUE 2017 training material] for usage of these commands and acls and tar cheat sheets. For more details see [[Archiving_data|Archiving Data]]  
[[Category:Data induction]]

CodeBreak 13/04/2022

2022-04-20T00:38:37Z

C.carouge: Created page with " = <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal;..."

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Access files on OOD from remote computer =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Files saved under the home directory on OOD are not accessible except from the OOD, but the requirement was to copy them to a remote (home) computer. The solution is to save the files to a <span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">/g/data <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">location, which is accessible from the OOD, but also accessible remotely (using <span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">rsync <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">or <span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">scp <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">for example).

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">To make this even easier, create a symbolic link in your OOD home directory to the location on <span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">/g/data<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">, so that it is easy to navigate to that location using the file explorer tools and other methods for accessing the file system in OOD. e.g.  
<div style="background:#eeeeee; border:1px solid #cccccc; padding:5px 10px"><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">ln -s /g/data/$PROJECT/$USER gdata  </div>
= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Using hh5 conda environment in OOD jupyterlab =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">There was an issue with loading the CLEX conda environment in OOD JupyterLab. This is [http://climate-cms.wikis.unsw.edu.au/Conda#Interactive_Analysis_.2F_Jupyter <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">covered in our wiki]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">but the proximate problem was a specific, out-of-date, environment was being specified.

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Convert model longitude to value between 0 and 360 degrees =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The ACCESS-OM2 model longitude has a minimum of -270 and a maximum of 90 degrees. Using functions from the GSW toolbox for the equation of state of seawater requires longitude values between 0 and 360 degrees. The question revolved around the best way to do this.

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The answer given for this case was to create a small function (e.g. <span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">lon_360<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">) which will take a value between -270 and 90 and return a value between 0 and 360. The function can then be used in the call to the gsw function directly, e.g.  
<div style="background:#eeeeee; border:1px solid #cccccc; padding:5px 10px"><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">gsw.some_function(... , lon_360(lon_value), ...)</div>
= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Interpolate density profiles to new vertical grid =

  <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Use [https://xarray.pydata.org/en/v0.12.1/interpolation.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">xarray interpolation]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">.

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">General Matplotlib Questions =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The student had questions about the various different ways in which plots are created with matplotlib, especially when searching online and different people suggest different methods.

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">We discussed the difference between object oriented plotting and function plotting, and how we usually use the object oriented way as it's easier to take full control of the plot features.

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">We compared the figure object to the canvas on which the items are to be drawn, and the axes object the coordinate system inside the figure.

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">How to run a tested analysis on a bigger dataset in Python =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The student had an analysis written in a Jupyter Notebook in Python. The problem was that when moving to a bigger dataset, the kernel would die. 

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">As the student had no previous knowledge of any parallel programming with Python, we advised them to follow [https://coecms-training.github.io/parallel/README.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">our latest parallel training]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">and then contact us again for follow-up questions.

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">We also gave information about the [https://climtas.readthedocs.io/en/latest/ <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">climtas]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">package. This package makes it simpler to open a distributed cluster for Dask. Additionally, the student was doing some resampling of data without missing data and always using blocks of the same length. In that specific case, climtas contains a resampling function that will speed up the process compared to the general function from xarray.

CodeBreak

2022-04-20T00:10:31Z

C.carouge:

= Introduction =

The CMS team is running CodeBreak sessions every fortnight.

Those sessions are an opportunity to come and have your questions answered by the CMS team, or to listen in to learn something new. Queries can be on some specific code you are writing, about a training you are trying to understand or broader questions about programming or scientific modelling. At the start of each session, we will ask you to provide a brief summary of your query so we can organise break-out rooms themed on topics, in order to make it possible to cover as many questions as possible and keep each room focussed on related problems. When appropriate we will take a few minutes at the beginning of a session to present useful new software or language features, interesting blogs or highlight new developments in modelling. These topics will be announced beforehand as they arise.

= CodeBreak sessions synopsis =

== 2022 ==

=== [[CodeBreak_13/04/2022|13/04/2022]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Accessing files on OOD from remote computer</li>
<li>Loading hh5 conda environment in OOD jupyterlab</li>
<li>Convert model longitude to value between 0 and 360 degrees</li>
<li>Interpolate density profiles to new vertical grid</li>
<li>General Matplotlib Questions</li>
<li>How to apply an analysis to a bigger dataset in Python</li>
</ul>

=== [[CodeBreak_30/03/2022|30/03/2022]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Understanding error in Python</li>
<li>Avoiding repetition in Python through code organisation</li>
<li>Count the number of elements that follow a given condition</li>
</ul>

=== [[CodeBreak_16/03/2022|16/03/2022]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Copying global attributes from a NetCDF file</li>
<li>Calculating differential in xarray</li>
<li>Regression analysis on spatial data</li>
</ul>

=== [[CodeBreak_02/03/2022|02/03/2022]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Problem installing the climextRemes Python package</li>
<li>Understand and use a Fortran program</li>
</ul>

=== [[CodeBreak_16/02/2022|16/02/2022]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Looping in Python and code organisation</li>
<li>Memory Consumption and optimisation in Python.</li>
<li>UM Model Crash explanation</li>
<li>How to use [https://github.com/TEOS-10/GSW-Python Gibbs-SeaWater (GSW) Oceanographic Toolbox] to calculate the dynamic height</li>
<li>Code organisation with NCL and more generally.</li>
</ul>

== 2021 ==

=== [[CodeBreak_1/9/2021|1/9/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>How to organise plotting large numbers of plots from heterogenous models in python</li>
<li>Layout of subplots with matplotlib in python</li>
<li>Fast percentile calculation in python</li>
</ul>

 

=== [[CodeBreak_15/9/2021|15/9/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>How to analyse data weekly and creating weekly climatology</li>
<li>Curl calculations</li>
<li>Weighted function in xarray</li>
<li>How to organise calculations dynamically with python data structure (dict) e.g. looping over seasons</li>
</ul>

 

=== [[CodeBreak_29/9/2021|29/9/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>What to do when variables have coordinates that are not quite equal.</li>
<li>Calculate daily means of ERA5 pressure-level data</li>
<li>Conversion of UM outputs between pressure and model levels</li>
<li>Writing a Xarray DataArray sub-class</li>
<li>Incompatible datetime and cftime error</li>
<li>Reprojecting data from metres to lat/lon in python</li>
</ul>

 

=== [[CodeBreak_13/10/2021|13/10/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Applying land masking to CMIP data</li>
<li>netCDF compression</li>
<li>Bathymetry data</li>
</ul>

=== [[CodeBreak_27/10/2021|27/10/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Panel plot or facet plot along the time axis</li>
</ul>

=== [[CodeBreak_24/11/2021|24/11/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Using dask when calculating Eddy Kinetic Energy for high resolution ocean model</li>
<li>How to calculate antecedent precipitation index efficiently for entire AGCD dataset</li>
</ul>

=== [[CodeBreak_08/12/2021|08/12/2021]] ===
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li>Particle trajectory data analysis.</li>
<li>Fortran code strange behaviour</li>
<li>Old Python code crash due to user-installed package needing update</li>
<li>Using ERA5 data as input for the HySplit trajectory model</li>
<li>ACCESS-OM2 output: selection of time slice</li>
<li>Understanding xarray, chunking, queue</li>
</ul>

[[Category:Training]]

Training

2022-04-12T05:29:34Z

C.carouge:

== Training Needs ==

=== General ===

*[[NCI|Accessing and using NCI]]
*[[Programming|Programming]]
*[[ProgrammingTools|ProgrammingTools]]

== CodeBreak ==

CMS runs fortnightly Q&A virtual sessions for the Centre's researchers. We publish a synopsis of those sessions shortly afterwards. You can find all the information on [[CodeBreak|this page]].

== CMS Presentations ==

CMS used to run weekly training sessions presenting computational and data related topics. Now we run a fortnightly [[CodeBreak|Code Break sessions]].

To get updates on our next training sessions '''[https://mailmanlists.co/mailman/listinfo/training-coe sign up]''' to our email list

All our training videos are available at our '''[https://www.youtube.com/channel/UCSmoK6oWV9O0Hmyt9UdDNsQ Youtube channel]'''

*[https://www.youtube.com/watch?v=aYukDeA-8hE Python Basics] ([https://www.youtube.com/watch?v=-UktTJDh_48 Part 2]) ([https://www.youtube.com/watch?v=pCabwrUa26E Part 3])
*[https://www.youtube.com/watch?v=2GmFf9x1qg0 Introduction to NCI]
*[https://youtu.be/5z75iHJyT8Q Large Dataset Analysis with Jupyter]
*[https://www.youtube.com/watch?v=aNUe-ukitpU Jupyter Introduction]
*[https://www.youtube.com/watch?v=pk-UD_4ZAcE WRF at NCI]
*[https://www.youtube.com/watch?v=DzbtG3UiMUw WRF Q&A]
*[[Training:NCI_Storage_Monitoring|NCI Storage Monitoring]] 2019-10-02
*[https://youtu.be/4tzug0DMZVI Licensing for Data and Software] 2019-07-10
*[[Training:_Basic_Python|Basic Python]] 2019-06-05 - 2019-07-03
*[[Training:Big_Data|Analysing Big Data]] 2019-05-15 - 2019-05-22
*[https://youtu.be/gYKc1mJitEw Data at NCI] 2019-05-08
*[[Git_Introduction_old|Git Introduction old]] 2019-04-10 - 2019-04-17
*[[Git_Introduction|Git Introduction]] 2020-09-09 - 2020-09-16
*[[Bash_Introduction|Bash Introduction]] 2019-03-13 - 2019-04-03
*[[WRF_Introduction|WRF Introduction]] 2019-10-23 - 2019-10-30
*[http://climate-cms.wikis.unsw.edu.au/Category:Setting_up Setting up]

== Software Carpentry ==

For general programming training we recommend the material from [https://software-carpentry.org/ Software Carpentry], a group whos focus is teaching researchers programming skills. Check with your University if they run Software Carpentry Workshops.

[[Category:Training]]

CodeBreak 30/03/2022

2022-04-12T01:04:53Z

C.carouge:

= Error in Python =

The person was trying to reproduce an example Python program. It output the error:
<pre>AttributeError: 'function' object has no attribute ‘compute’ </pre>

This happened because the person had omitted the parentheses on a previous call to another function. For example, the code below will return the same error:
<pre>a=xr.DataArray([1,2])
b=a.mean
b.mean()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'mean'</pre>

On the second line, <code>b=a.mean</code> will return a reference to the meanfunction associated with the DataArray <code>a</code>. In other words, <code>b</code> is the mean function, not its result. That is why the error message starts with <code>‘function’ object</code>. 

To get the results from the function, you need to call the function which is done with the <code>()</code> and any argument you need to give.

= Avoiding repetition in python through code organisation =

It is quite common when starting to write a code to just copy and paste a group of lines if we want, for example, to execute the same calculation on a different variable. If you find that you are doing that a lot, it might be time to go back and re-organise your code. A better organised code is not only more readable but is easier to fix should you find a bug. A well-organised code will limit the number of copied lines so you only have to fix your code in one point.

One simple step is to try to create a function for every block of code you are repeating, using only the variables and parameters that might change between calls as arguments.    
<pre>def myfunction( row, array ):

param = … # this doesn’t change no need to use it as an argument for the function
… Your calculation …

return newarray </pre>

In the case we looked at, one function was applied to each row of a pandas dataframe. To do so we can use the pandas dataframe <code>itterrows()</code> method and append each result to a list.   
<pre>results = []
for index, row in df.iterrows():
results.append( myfunction(row, array) ) </pre>

Basically <code><kbd>iterrows()</kbd></code> loops across each row of the dataframe and returns the index and the row itself. 

The results list can then be concatenated into one dataArray
<pre>final = xarray.concat(results, dim='mydim') </pre>

Another example we looked at was a function to create a subplot, which was then called on different variables, each representing a different region. In such a case, to handle different titles and parameters it can be handy to use a dictionary.  
<pre>regions = {'east': [mslp_east, 1, 'Eastern Australian NRM region'],
‘north’: [mslp_north, 2, 'Northern Australian NRM region'], … } </pre>

Then you can use the key/values pair to call the same function on each region:  
<pre>for k in regions.keys():
fig = subplot(regions[k][0], regions[k][1], regions[k][2]) </pre>

The dictionary method <code>keys()</code> here works in a similar way to <code>iterrows()</code> but returns only the key. Use <code>items() </code>if you want the key,value pairs and <code>values()</code> if you want only the values.

= Count the number of elements that follow a given condition =

A question was to count how many events with positive values there are for each grid point given an array like <code>anom(event, lat, lon)</code>. xarray has a function <code>count()</code>. This will count the number of valid values along the given dimensions. So we need to filter out the negative values using <code>where()</code> and then <code>count()</code> the number of values left along the "event" dimension.
<pre>positive_count = anom.where(anom > 0).count(dim=’event’)</pre>