http://climate-cms.wikis.unsw.edu.au/api.php?action=feedcontributions&user=S.wales&feedformat=atomclimate-cms wikis.unsw.edu.au - User contributions [en]2024-03-28T14:25:40ZUser contributionsMediaWiki 1.31.0http://climate-cms.wikis.unsw.edu.au/index.php?title=Migrate_experiment_projects&diff=3439Migrate experiment projects2022-02-10T23:14:31Z<p>S.wales: /* Setting your default project */</p>
<hr />
<div><br />
This page contains information on what to do if you want to change project to work at NCI.<br />
<br />
= Setting your default project =<br />
<br />
To change the default project that jobs get submitted to when you run 'qsub', edit the file on Gadi '~/.config/gadi-login.conf'<br />
<br />
If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'<br />
<br />
In either case you will need to log out and back in again for the change to take effect<br />
<br />
= Climate model experiments =<br />
<br />
*[[Migrate_payu_experiment_project|Payu experiment]] <br />
*[[Migrate_umui_experiment_project|UMUI experiment]] <br />
*[[Migrate_rose_experiment_project|Rose/Cylc experiment]] <br />
*[[Migrate_NUWRF_experiment_project|NUWRF experiment]] <br />
<br />
WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.<br />
<br />
= Tips for other scripts =<br />
<br />
In your script, you may have used w35 or w48 in the following lines:<br />
<br />
*PBS directive -P to specify the project <br />
*PBS directive -l storage to load the storage space for these projects. <br />
*Full paths of input or output data <br />
<br />
It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.<br />
<br />
== Find files with the project listed in ==<br />
<br />
To find all the files that contain the mention of <samp>w35</samp>, you can use the following command:<br />
<pre>grep --exclude=*.nc -irlI w35 <directory_name> > files_to_update.txt</pre><br />
<br />
<samp>files_to_update.txt </samp>will contain a list of files where <samp>w35</samp> appear in at least one line. The command will not search in binary files and netCDF files and you can exclude more files from the search (e.g. figures) with the <code>--exclude=</code> option. Replace <samp><directory_name></samp> with the directory you want to search through.<br />
<br />
At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only those:<br />
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt</pre><br />
<br />
== Find all and replace all ==<br />
<br />
All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality for your favourite text editor.</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Migrate_experiment_projects&diff=3438Migrate experiment projects2022-02-10T23:09:59Z<p>S.wales: </p>
<hr />
<div><br />
This page contains information on what to do if you want to change project to work at NCI.<br />
<br />
= Setting your default project =<br />
<br />
To change the default project that jobs get submitted to when you run 'qsub', edit the file on Gadi '~/.config/gadi-login.conf'<br />
<br />
If you also run models from accessdev you can change the default project by editing the file '~/.rashrc'<br />
<br />
In either case you will need to log out and back in again for the change to take effect<br />
<br />
= Climate model experiments =<br />
<br />
*[[Migrate_payu_experiment_project|Payu experiment]] <br />
*[[Migrate_umui_experiment_project|UMUI experiment]] <br />
*[[Migrate_rose_experiment_project|Rose/Cylc experiment]] <br />
*[[Migrate_NUWRF_experiment_project|NUWRF experiment]] <br />
<br />
WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.<br />
<br />
= Tips for other scripts =<br />
<br />
In your script, you may have used w35 or w48 in the following lines:<br />
<br />
*PBS directive -P to specify the project <br />
*PBS directive -l storage to load the storage space for these projects. <br />
*Full paths of input or output data <br />
<br />
It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.<br />
<br />
== Find files with the project listed in ==<br />
<br />
To find all the files that contain the mention of <samp>w35</samp>, you can use the following command:<br />
<pre>grep --exclude=*.nc -irlI w35 <directory_name> > files_to_update.txt</pre><br />
<br />
<samp>files_to_update.txt </samp>will contain a list of files where <samp>w35</samp> appear in at least one line. The command will not search in binary files and netCDF files and you can exclude more files from the search (e.g. figures) with the <code>--exclude=</code> option. Replace <samp><directory_name></samp> with the directory you want to search through.<br />
<br />
At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only those:<br />
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt</pre><br />
<br />
== Find all and replace all ==<br />
<br />
All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality for your favourite text editor.</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3424Conda2022-01-18T22:17:25Z<p>S.wales: /* 21.10 (Current Stable) */</p>
<hr />
<div><br />
<br />
<br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session, a qsub job is a new session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 22.01 (Current Unstable) ===<br />
<br />
=== 21.10 (Current Stable) ===<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays<br />
** [https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset<br />
<br />
* '''Geospatial'''<br />
** [https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline<br />
** [https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes<br />
** [https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs <br />
<br />
* '''Developer Tools'''<br />
** [https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols<br />
** [https://github.com/gorakhargosh/watchdog watchdog] monitor file system events<br />
** [https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more <br />
** [https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure<br />
** [https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol<br />
** [https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material<br />
** [https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data<br />
** [https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder<br />
<br />
'''Known Issues'''<br />
<br />
* MetPy is incompatible with the Matplotlib version in this environment https://github.com/Unidata/MetPy/issues/2281<br />
<br />
=== 21.07 (Unsupported) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== <s>21.04</s> ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== <s>21.01</s> ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3423Conda2022-01-18T00:37:27Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
<br />
<br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session, a qsub job is a new session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 22.01 (Current Unstable) ===<br />
<br />
=== 21.10 (Current Stable) ===<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays<br />
** [https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset<br />
<br />
* '''Geospatial'''<br />
** [https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline<br />
** [https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes<br />
** [https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs <br />
<br />
* '''Developer Tools'''<br />
** [https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols<br />
** [https://github.com/gorakhargosh/watchdog watchdog] monitor file system events<br />
** [https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more <br />
** [https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure<br />
** [https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol<br />
** [https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material<br />
** [https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data<br />
** [https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder<br />
<br />
=== 21.07 (Unsupported) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== <s>21.04</s> ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== <s>21.01</s> ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3422Conda2022-01-10T23:51:38Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
<br />
<br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session, a qsub job is a new session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 22.01 ===<br />
<br />
=== 21.10 ===<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays<br />
** [https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset<br />
<br />
* '''Geospatial'''<br />
** [https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline<br />
** [https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes<br />
** [https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs <br />
<br />
* '''Developer Tools'''<br />
** [https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols<br />
** [https://github.com/gorakhargosh/watchdog watchdog] monitor file system events<br />
** [https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more <br />
** [https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure<br />
** [https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol<br />
** [https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material<br />
** [https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data<br />
** [https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder<br />
<br />
=== 21.07 (Current Stable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== 21.04 (Unsupported) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3421Conda2022-01-10T23:37:29Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
<br />
<br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session, a qsub job is a new session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.bashrc</code> file. It can go in the <code>if in_interactive_shell; then</code> section of the file.<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at [https://github.com/coecms/conda-history https://github.com/coecms/conda-history].<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.10 ===<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://github.com/josuemtzmo/xarrayMannKendall xarraymannkendall] compute linear trends over 2D and 3D arrays<br />
** [https://hdbscan.readthedocs.io/en/latest/ hdbscan] tools to use unsupervised learning to find clusters, or dense regions, of a dataset<br />
<br />
* '''Geospatial'''<br />
** [https://github.com/jannikmi/timezonefinder timezonefinder] looking up the corresponding timezone for given coordinates on earth entirely offline<br />
** [https://iris-grib.readthedocs.io/en/stable/ iris-grib] converting between weather and climate datasets that are stored as GRIB files and Iris cubes<br />
** [https://github.com/NCAR/intake-thredds intake-thredds] Intake interface to THREDDS data catalogs <br />
<br />
* '''Developer Tools'''<br />
** [https://lftp.yar.ru/ lftp] sophisticated file transfer program supporting a number of network protocols<br />
** [https://github.com/gorakhargosh/watchdog watchdog] monitor file system events<br />
** [https://github.com/ml31415/numpy-groupies numpy_groupies] Optimised tools for group-indexing operations: aggregated sum and more <br />
** [https://anytree.readthedocs.io/en/latest/ anytree] Simple, lightweight and extensible Tree data structure<br />
** [https://github.com/hansec/fortran-language-server fortran-language-server] Fortran implementation of the Language Server Protocol<br />
** [https://jupyterbook.org/intro.html jupyter-book] building beautiful, publication-quality books and documents from computational material<br />
** [https://github.com/fsspec/kerchunk kerchunk] Cloud-friendly access to archival data<br />
** [https://github.com/ultrajson/ultrajson ujson] ultra fast JSON encoder and decoder<br />
<br />
=== 21.07 (Current Stable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== 21.04 (Unsupported) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=CLEX_projects_at_NCI&diff=3406CLEX projects at NCI2021-12-08T22:57:52Z<p>S.wales: </p>
<hr />
<div><br />
The projects are:<br />
<br />
<br />
{|<br />
|-<br />
! Project code<br />
! Research Program (RP)<br />
! Lead CI<br />
|-<br />
| [https://my.nci.org.au/mancini/project/v45 v45]<br />
| Ocean Extremes<br />
| Andy Hogg<br />
|-<br />
| [https://my.nci.org.au/mancini/project/w40 w40]<br />
| Weather and Climate interactions<br />
| Martin Jucker<br />
|-<br />
| [https://my.nci.org.au/mancini/project/w42 w42]<br />
| Attribution and Risk<br />
| Abhnil Prasad<br />
|-<br />
| [https://my.nci.org.au/mancini/project/w97 w97]<br />
| Drought<br />
| Jason Evans<br />
|}<br />
<br />
w35 has been merged into w97<br />
<br />
w48 has been merged into w40<br />
<br />
Pre - 2022 project assignments:<br />
{|<br />
|-<br />
! Project code<br />
! Research Program (RP)<br />
! Lead CI<br />
|-<br />
| v45<br />
| Extra-tropical variability<br />
| Andy Hogg<br />
|-<br />
| w35<br />
| Drought<br />
| Claire Carouge<br />
|-<br />
| w40<br />
| Extreme Rainfall<br />
| Todd Lane<br />
|-<br />
| w42<br />
| Extreme Rainfall<br />
| Steven Sherwood<br />
|-<br />
| w48<br />
| Tropical Variability<br />
| Dietmar Dommenget<br />
|-<br />
| w97<br />
| Heatwaves<br />
| Jason Evans<br />
|}<br />
<br />
=== Which project to join if part of several RPs ===<br />
<br />
We recommend people join all relevant projects but choose one project to work from. This will help people to collaborate with researchers from all the RPs they are part of.<br />
<br />
&nbsp;<br />
<br />
=== Setup default project on Gadi ===<br />
<br />
*Check your default project. Connect to Gadi. Then type: <br />
<br />
<syntaxhighlight lang="bash">echo $PROJECT<br />
</syntaxhighlight><br />
<br />
*If the output from the previous step is not what you want you can [[Gadi#Swapping_Projects|change your default project]]. <br />
<br />
[[Category:Admin]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=CLEX_projects_at_NCI&diff=3405CLEX projects at NCI2021-12-08T22:56:26Z<p>S.wales: </p>
<hr />
<div><br />
The projects are:<br />
<br />
<br />
{|<br />
|-<br />
! Project code<br />
! Research Program (RP)<br />
! Lead CI<br />
|-<br />
| v45<br />
| Ocean Extremes<br />
| Andy Hogg<br />
|-<br />
| w40<br />
| Weather and Climate interactions<br />
| Martin Jucker<br />
|-<br />
| w42<br />
| Attribution and Risk<br />
| Abhnil Prasad<br />
|-<br />
| w97<br />
| Drought<br />
| Jason Evans<br />
|}<br />
<br />
w35 has been merged into w97<br />
<br />
w48 has been merged into w40<br />
<br />
Pre - 2022 project assignments:<br />
{|<br />
|-<br />
! Project code<br />
! Research Program (RP)<br />
! Lead CI<br />
|-<br />
| v45<br />
| Extra-tropical variability<br />
| Andy Hogg<br />
|-<br />
| w35<br />
| Drought<br />
| Claire Carouge<br />
|-<br />
| w40<br />
| Extreme Rainfall<br />
| Todd Lane<br />
|-<br />
| w42<br />
| Extreme Rainfall<br />
| Steven Sherwood<br />
|-<br />
| w48<br />
| Tropical Variability<br />
| Dietmar Dommenget<br />
|-<br />
| w97<br />
| Heatwaves<br />
| Jason Evans<br />
|}<br />
<br />
=== Which project to join if part of several RPs ===<br />
<br />
We recommend people join all relevant projects but choose one project to work from. This will help people to collaborate with researchers from all the RPs they are part of.<br />
<br />
&nbsp;<br />
<br />
=== Setup default project on Gadi ===<br />
<br />
*Check your default project. Connect to Gadi. Then type: <br />
<br />
<syntaxhighlight lang="bash">echo $PROJECT<br />
</syntaxhighlight><br />
<br />
*If the output from the previous step is not what you want you can [[Gadi#Swapping_Projects|change your default project]]. <br />
<br />
[[Category:Admin]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=CLEX_projects_at_NCI&diff=3404CLEX projects at NCI2021-12-08T22:54:47Z<p>S.wales: </p>
<hr />
<div><br />
The projects are:<br />
<br />
<br />
{|<br />
|-<br />
! Project code<br />
! Research Program (RP)<br />
! Lead CI<br />
|-<br />
| v45<br />
| Ocean Extremes<br />
| Andy Hogg<br />
|-<br />
| w40<br />
| Weather and Climate interactions<br />
| Martin Jucker<br />
|-<br />
| w42<br />
| Attribution and Risk<br />
| Abhnil Prasad<br />
|-<br />
| w97<br />
| Drought<br />
| Jason Evans<br />
|}<br />
<br />
<br />
Pre - 2021 project assignments:<br />
{|<br />
|-<br />
! Project code<br />
! Research Program (RP)<br />
! Lead CI<br />
|-<br />
| v45<br />
| Extra-tropical variability<br />
| Andy Hogg<br />
|-<br />
| w35<br />
| Drought<br />
| Claire Carouge<br />
|-<br />
| w40<br />
| Extreme Rainfall<br />
| Todd Lane<br />
|-<br />
| w42<br />
| Extreme Rainfall<br />
| Steven Sherwood<br />
|-<br />
| w48<br />
| Tropical Variability<br />
| Dietmar Dommenget<br />
|-<br />
| w97<br />
| Heatwaves<br />
| Jason Evans<br />
|}<br />
<br />
=== Which project to join if part of several RPs ===<br />
<br />
We recommend people join all relevant projects but choose one project to work from. This will help people to collaborate with researchers from all the RPs they are part of.<br />
<br />
&nbsp;<br />
<br />
=== Setup default project on Gadi ===<br />
<br />
*Check your default project. Connect to Gadi. Then type: <br />
<br />
<syntaxhighlight lang="bash">echo $PROJECT<br />
</syntaxhighlight><br />
<br />
*If the output from the previous step is not what you want you can [[Gadi#Swapping_Projects|change your default project]]. <br />
<br />
[[Category:Admin]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Migrate_rose_experiment_project&diff=3378Migrate rose experiment project2021-11-02T01:00:58Z<p>S.wales: Created page with "UM Rose/Cylc experiments will by default run under your default project, which you can change on Accessdev in the '''~/.rashrc''' file and on Gadi in the '''~/.config/gadi-log..."</p>
<hr />
<div>UM Rose/Cylc experiments will by default run under your default project, which you can change on Accessdev in the '''~/.rashrc''' file and on Gadi in the '''~/.config/gadi-login.conf''' file.<br />
<br />
You can also manually set the project for an experiment in the run's '''suite.rc''' file, adding e.g. a '-P = w40' flag in the '[[[directives]]]' section alongside the other PBS flags.<br />
<br />
Output directories of a Cylc job are configured in the file on Accessdev '''~/.cylc/global.rc'''. Options that may be useful are the [https://cylc.github.io/cylc-doc/stable/html/appendices/site-user-config-ref.html#hosts-host-work-directory 'host work directory'], e.g.<br />
<pre><br />
[hosts]<br />
[[gadi.*]]<br />
work directory = '/scratch/w40/$USER/cylc-run'<br />
</pre><br />
will put your job's 'work' and 'share' directories in w40's /scratch space. You can confirm it ran correctly with 'cylc get-global-config', which will show the current settings of all configuration options.</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3358Conda2021-10-11T22:43:44Z<p>S.wales: /* Interactive Analysis / Jupyter */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3357Conda2021-10-11T22:43:21Z<p>S.wales: /* Interactive Analysis / Jupyter */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the ''''Open on Demand' (OOD)''' service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3348ACCESS-ESM 1.52021-10-01T00:10:03Z<p>S.wales: /* Checking */</p>
<hr />
<div><br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
* [https://github.com/coecms/access-esm-amip AMIP run (non-coupled atmosphere only)]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
1. Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
2. Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
3. Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
4. Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
<br />
== AMIP ==<br />
<br />
Science Contacts: Mustapha Adamu, Shayne McGregor<br />
<br />
An uncoupled, atmosphere only version of ACCESS-ESM is provided at [https://github.com/coecms/access-esm-amip this Github repository].<br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
= Fixing Ice Restart Dates (Payu) =<br />
<br />
In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).<br />
<br />
Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')<br />
<br />
We'll need to work out some numbers to put into the restart directory, which can be done in Python<br />
<syntaxhighlight lang="python"><br />
import cftime<br />
<br />
init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml<br />
dt = 3600 # Model timestep as set in ice/input_ice.nml<br />
<br />
this_run_start = cftime.datetime(1996,1,1) # Date we want to start at<br />
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at<br />
<br />
runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run<br />
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run<br />
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run<br />
istep0 = time / dt # Model steps between init_date and start of this run<br />
<br />
print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")<br />
</syntaxhighlight><br />
<br />
== Restart namelists ==<br />
<br />
Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.<br />
<br />
In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.<br />
<br />
== Restart file ==<br />
<br />
Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like<br />
<syntaxhighlight><br />
scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000<br />
</syntaxhighlight><br />
<br />
Give the output file a bigger number than any other restart to make sure Payu picks it up.<br />
<br />
== Checking ==<br />
<br />
As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.<br />
<br />
Next, check work/ice/RESTART/ice.restart_file contains the name of the new ice restart file, e.g. 'iced.99990101'<br />
<br />
Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like<br />
<pre><br />
istep0 = 52584<br />
istep1: 52584 idate: 18611231 sec: 0<br />
Restart read at istep= 52584 189302400.000000<br />
istep1: 52584 idate: 18560101 sec: 0<br />
istep1: 52608 idate: 18560102 sec: 0<br />
istep1: 52632 idate: 18560103 sec: 0<br />
istep1: 52656 idate: 18560104 sec: 0<br />
istep1: 52680 idate: 18560105 sec: 0<br />
istep1: 52704 idate: 18560106 sec: 0<br />
istep1: 52728 idate: 18560107 sec: 0<br />
...<br />
</pre><br />
<br />
After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.<br />
<br />
After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3344Conda2021-09-28T00:38:08Z<p>S.wales: /* Interactive Analysis / Jupyter */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The preferred method of running Jupyter at NCI is through the 'Open on Demand' (OOD) service https://ood.nci.org.au. This runs a Jupyter instance in NCI's cloud that you can access directly from your browser. To use CLEX Conda in OOD, start Jupyter with the advanced options:<br />
<br />
* Module Directories: /g/data/hh5/public/modules<br />
* Modules: conda/analysis3<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3343Conda2021-09-28T00:20:02Z<p>S.wales: /* 21.07 (Current Unstable) */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3342Conda2021-09-28T00:19:30Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.tensorflow.org/ tensorflow] TensorFlow is an end-to-end open source platform for machine learning<br />
*** Note: You may need to load the NCI modules 'cuda' and 'cudnn' to use tensorflow on GPU nodes<br />
** [https://shap.readthedocs.io/en/latest/index.html shap] a game theoretic approach to explain the output of any machine learning model<br />
<br />
* '''Geospatial'''<br />
** [https://gcm-filters.readthedocs.io/en/latest/ gcm_filters] Diffusion-based Spatial Filtering of Gridded Data from General Circulation Models<br />
<br />
* '''Visualisation'''<br />
** [https://docs.enthought.com/mayavi/mayavi/ mayavi] 3D scientific data visualization and plotting in Python<br />
<br />
* '''Developer Tools'''<br />
** [https://github.com/danielwhatmuff/gh gh] a tool to open Github projects in a browser from the command line<br />
** gcm_filters<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3319ACCESS-ESM 1.52021-09-15T03:34:00Z<p>S.wales: /* Restart file */</p>
<hr />
<div><br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
1. Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
2. Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
3. Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
4. Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
= Fixing Ice Restart Dates (Payu) =<br />
<br />
In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).<br />
<br />
Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')<br />
<br />
We'll need to work out some numbers to put into the restart directory, which can be done in Python<br />
<syntaxhighlight lang="python"><br />
import cftime<br />
<br />
init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml<br />
dt = 3600 # Model timestep as set in ice/input_ice.nml<br />
<br />
this_run_start = cftime.datetime(1996,1,1) # Date we want to start at<br />
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at<br />
<br />
runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run<br />
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run<br />
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run<br />
istep0 = time / dt # Model steps between init_date and start of this run<br />
<br />
print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")<br />
</syntaxhighlight><br />
<br />
== Restart namelists ==<br />
<br />
Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.<br />
<br />
In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.<br />
<br />
== Restart file ==<br />
<br />
Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like<br />
<syntaxhighlight><br />
scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000<br />
</syntaxhighlight><br />
<br />
Give the output file a bigger number than any other restart to make sure Payu picks it up.<br />
<br />
== Checking ==<br />
<br />
As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.<br />
<br />
Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like<br />
<pre><br />
istep0 = 52584<br />
istep1: 52584 idate: 18611231 sec: 0<br />
Restart read at istep= 52584 189302400.000000<br />
istep1: 52584 idate: 18560101 sec: 0<br />
istep1: 52608 idate: 18560102 sec: 0<br />
istep1: 52632 idate: 18560103 sec: 0<br />
istep1: 52656 idate: 18560104 sec: 0<br />
istep1: 52680 idate: 18560105 sec: 0<br />
istep1: 52704 idate: 18560106 sec: 0<br />
istep1: 52728 idate: 18560107 sec: 0<br />
...<br />
</pre><br />
<br />
After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.<br />
<br />
After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3318ACCESS-ESM 1.52021-09-15T03:33:37Z<p>S.wales: </p>
<hr />
<div><br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
1. Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
2. Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
3. Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
4. Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
= Fixing Ice Restart Dates (Payu) =<br />
<br />
In configuration file 'ice/input_ice.nml' make a note of the value of 'init_date' - this is the date that the model thinks it started at (ignore the similarly named 'inidate' value here).<br />
<br />
Make a copy of the restart directory that you want to re-run the model from (e.g. 'cp -r archive/restart1995 archive/restart11995')<br />
<br />
We'll need to work out some numbers to put into the restart directory, which can be done in Python<br />
<syntaxhighlight lang="python"><br />
import cftime<br />
<br />
init_date = cftime.datetime(1850,1,1) # Initial date as set in ice/input_ice.nml<br />
dt = 3600 # Model timestep as set in ice/input_ice.nml<br />
<br />
this_run_start = cftime.datetime(1996,1,1) # Date we want to start at<br />
prev_run_start = cftime.datetime(1995,1,1) # Date the previous run started at<br />
<br />
runtime0 = (prev_run_start - init_date).total_seconds() # Time between init_date and start of previous run<br />
runtime = (this_run_start - prev_run_start).total_seconds() # Model runtime of previous run<br />
time = (this_run_start - init_date).total_seconds() # Time between init_date and start of this run<br />
istep0 = time / dt # Model steps between init_date and start of this run<br />
<br />
print(f"{runtime0=}, {runtime=}, {time=}, {istep0=}")<br />
</syntaxhighlight><br />
<br />
== Restart namelists ==<br />
<br />
Payu also checks the namelist files in the restart directory, so we'll have to edit those as well.<br />
<br />
In the restart directory, edit ice/input_ice.nml (e.g. archive/restart11995/ice/input_ice.nml), setting 'runtime0' and 'runtime' to the values just calculated. Also still in the restart directory edit ice/cice_in.nml to set 'istep0'.<br />
<br />
== Restart file ==<br />
<br />
Now we'll set up the restart file. Payu will start from the highest numbered restart file in 'archive/restart11995/ice', and we need to add the 'time' and 'istep0' values previously calculated to that. 'scripts/cicedumpdatemodify.py' is provided to do this modification, use it like<br />
<syntaxhiglight><br />
scripts/cicedumpdatemodify.py -v -i archive/restart11995/ice/iced.40990601 -o archive/restart11995/ice/iced.99990101 --istep0 1279800 --time 4607280000<br />
</syntaxhighlight><br />
<br />
Give the output file a bigger number than any other restart to make sure Payu picks it up.<br />
<br />
== Checking ==<br />
<br />
As a first check, run 'payu setup' in the configuration directory, then check work/ice/input_ice.nml. 'inidate' should be set to the date you're starting this run at. 'runtime' should be set to the number of seconds this run will cover, 31536000 for a normal year or 31622400 for a leap year.<br />
<br />
Clean up the work directory with 'payu sweep', then submit the run with 'payu run'. After the model has run for a little, check the output of 'grep istep work/ice/ice_diag.d', which should look something like<br />
<pre><br />
istep0 = 52584<br />
istep1: 52584 idate: 18611231 sec: 0<br />
Restart read at istep= 52584 189302400.000000<br />
istep1: 52584 idate: 18560101 sec: 0<br />
istep1: 52608 idate: 18560102 sec: 0<br />
istep1: 52632 idate: 18560103 sec: 0<br />
istep1: 52656 idate: 18560104 sec: 0<br />
istep1: 52680 idate: 18560105 sec: 0<br />
istep1: 52704 idate: 18560106 sec: 0<br />
istep1: 52728 idate: 18560107 sec: 0<br />
...<br />
</pre><br />
<br />
After the 'Restart read' line the values of idate should be dates starting from the target start time, incrementing by one day for each line. If idate isn't incrementing something has gone wrong and the run should be stopped.<br />
<br />
After at least a month has run check the output in work/ice/HISTORY to make sure that the time and time_bounds values are correct.<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3216ACCESS-ESM 1.52021-07-21T23:39:37Z<p>S.wales: /* Post-Processing */</p>
<hr />
<div><br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
1. Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
2. Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
3. Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
4. Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS&diff=3215ACCESS2021-07-21T23:39:19Z<p>S.wales: /* Post Processing */</p>
<hr />
<div>=Overview= <br />
<br />
The Australian Community Climate and Earth-System Simulator (ACCESS) is a coupled climate and earth system simulator to be developed as a joint initiative of the Bureau of Meteorology and CSIRO in cooperation with the university community in Australia.<br />
<br />
There are multiple configurations of ACCESS<br />
<br />
Our goal is to assist researchers in using ACCESS coupled model as provided on the computing cluster at NCI. If you have any corrections, comments or contributions, please feel free to email us on [mailto:cws_help@nf.nci.org.au cws_help@nf.nci.org.au] so that we can continue to update and upgrade the guides over time.<br />
<br />
For the full list of ACCESS guides see [[:Category:ACCESS]]<br />
<br />
<br />
== Model Versions ==<br />
<br />
=== Coupled ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 N48|ACCESS-CM 2.0 N48]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 Rescalable|ACCESS-CM 2.0 Rescalable]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48, N96 or N216 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS-ESM|ACCESS-ESM 1.5]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[CABLE|CABLE]] with carbon cycle<br />
|-<br />
| [[ACCESS-CM_1.4|ACCESS-CM 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_CMIP|ACCESS-CM 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_CMIP|ACCESS-CM 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[MOSES|MOSES]]<br />
|}<br />
<br />
<br />
&nbsp;<br />
<br />
=== Atmosphere-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS-AMIP_1.4|ACCESS-AMIP 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_AMIP|ACCESS-AMIP 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_AMIP|ACCESS-AMIP 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOSES|MOSES]]<br />
|}<br />
<br />
=== Ocean-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS-OM_2.0|ACCESS-OM2]]<br />
| &nbsp;<br />
| [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-025|ACCESS-OM2-025]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.25° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-01|ACCESS-OM2-01]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.1° + [[CICE|CICE]]<br />
|-<br />
| [[AusCOM|AusCOM]]<br />
| &nbsp;<br />
| [[MOM|MOM]] + [[CICE|CICE]]<br />
|}<br />
<br />
=Guides= <br />
<br />
For information on accessing the code and setup of the environment, see the [[Unified Model | UM page]]<br />
<br />
[https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin How to setup Pre-Industrial Control and Historical runs for CMIP5 Simulations] (ACCESS 1.0)<br />
<br />
* [[How to setup Historical Runs for CMIP5 Simulations in ACCESS 1.3]]<br />
* [[Building ACCESS]]<br />
* [[How to build ACCESS v1.4]]<br />
* [[ACCESS CMIP5 Forcings]]<br />
* [[DataMover]]<br />
<br />
== Coupled Model Crashes ==<br />
<br />
If the model becomes unstable it may crash with a stack trace mentioning 'bi_linear_h', like<br />
<syntaxhighlight><br />
um7.3x 0000000001111A08 mpl_recv_ 67 mpl_recv.F90<br />
um7.3x 000000000110AF66 gc_rrecv_ 168 gc_rrecv.F90<br />
um7.3x 0000000000989C5D bi_linear_h_ 613 bi_linear_h.f90<br />
um7.3x 0000000000CC92BC ritchie_ 2557 ritchie.f90<br />
um7.3x 00000000009DFB7D departure_point_ 382 departure_point.f90<br />
um7.3x 00000000008AA46F sl_thermo_ 681 sl_thermo.f90<br />
um7.3x 00000000006EC376 ni_sl_thermo_ 778 ni_sl_thermo.f90<br />
</syntaxhighlight><br />
<br />
Standard practice in this case is to add a small perturbation to the theta field in the most recent atmosphere restart dump and then restart the simulation. This perturbation normally allows the model to proceed past the crash.<br />
<br />
First make a backup of the atmosphere restart file, then run the script '~access/apps/pythonlib/umfile_utils/perturbIC.py' on the restart file, e.g.<br />
<syntaxhighlight><br />
module purge<br />
module use ~access/modules<br />
module load pythonlib/umfile_utils<br />
<br />
# Make a backup<br />
cp restart_dump.astart restart_dump.astart.bak<br />
<br />
# Perturb the initial conditions<br />
~access/apps/pythonlib/umfile_utils/perturbIC.py restart_dump.astart<br />
</syntaxhighlight><br />
<br />
For Payu runs (ACCESS-ESM):<br />
* The error message can be found in the log file 'access.err' in the experiment directory<br />
* Restarts are in the highest numbered archive/restartNNN/atmosphere directory<br />
* The restart file is named 'restart_dump.astart'<br />
<br />
= Post Processing =<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://git.nci.org.au/cm2704/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
= Unsupported Configurations =<br />
<br />
These configurations have been used previously but are not presently being supported<br />
<br />
* [[ACCESS0.X Slab Reosc]]<br />
<br />
[[Category:ACCESS]][[Category:Coupled Models]][[Category:Atmosphere Models]][[Category:Ocean Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS&diff=3214ACCESS2021-07-20T00:11:45Z<p>S.wales: </p>
<hr />
<div>=Overview= <br />
<br />
The Australian Community Climate and Earth-System Simulator (ACCESS) is a coupled climate and earth system simulator to be developed as a joint initiative of the Bureau of Meteorology and CSIRO in cooperation with the university community in Australia.<br />
<br />
There are multiple configurations of ACCESS<br />
<br />
Our goal is to assist researchers in using ACCESS coupled model as provided on the computing cluster at NCI. If you have any corrections, comments or contributions, please feel free to email us on [mailto:cws_help@nf.nci.org.au cws_help@nf.nci.org.au] so that we can continue to update and upgrade the guides over time.<br />
<br />
For the full list of ACCESS guides see [[:Category:ACCESS]]<br />
<br />
<br />
== Model Versions ==<br />
<br />
=== Coupled ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 N48|ACCESS-CM 2.0 N48]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 Rescalable|ACCESS-CM 2.0 Rescalable]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48, N96 or N216 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS-ESM|ACCESS-ESM 1.5]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[CABLE|CABLE]] with carbon cycle<br />
|-<br />
| [[ACCESS-CM_1.4|ACCESS-CM 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_CMIP|ACCESS-CM 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_CMIP|ACCESS-CM 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[MOSES|MOSES]]<br />
|}<br />
<br />
<br />
&nbsp;<br />
<br />
=== Atmosphere-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS-AMIP_1.4|ACCESS-AMIP 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_AMIP|ACCESS-AMIP 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_AMIP|ACCESS-AMIP 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOSES|MOSES]]<br />
|}<br />
<br />
=== Ocean-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS-OM_2.0|ACCESS-OM2]]<br />
| &nbsp;<br />
| [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-025|ACCESS-OM2-025]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.25° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-01|ACCESS-OM2-01]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.1° + [[CICE|CICE]]<br />
|-<br />
| [[AusCOM|AusCOM]]<br />
| &nbsp;<br />
| [[MOM|MOM]] + [[CICE|CICE]]<br />
|}<br />
<br />
=Guides= <br />
<br />
For information on accessing the code and setup of the environment, see the [[Unified Model | UM page]]<br />
<br />
[https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin How to setup Pre-Industrial Control and Historical runs for CMIP5 Simulations] (ACCESS 1.0)<br />
<br />
* [[How to setup Historical Runs for CMIP5 Simulations in ACCESS 1.3]]<br />
* [[Building ACCESS]]<br />
* [[How to build ACCESS v1.4]]<br />
* [[ACCESS CMIP5 Forcings]]<br />
* [[DataMover]]<br />
<br />
== Coupled Model Crashes ==<br />
<br />
If the model becomes unstable it may crash with a stack trace mentioning 'bi_linear_h', like<br />
<syntaxhighlight><br />
um7.3x 0000000001111A08 mpl_recv_ 67 mpl_recv.F90<br />
um7.3x 000000000110AF66 gc_rrecv_ 168 gc_rrecv.F90<br />
um7.3x 0000000000989C5D bi_linear_h_ 613 bi_linear_h.f90<br />
um7.3x 0000000000CC92BC ritchie_ 2557 ritchie.f90<br />
um7.3x 00000000009DFB7D departure_point_ 382 departure_point.f90<br />
um7.3x 00000000008AA46F sl_thermo_ 681 sl_thermo.f90<br />
um7.3x 00000000006EC376 ni_sl_thermo_ 778 ni_sl_thermo.f90<br />
</syntaxhighlight><br />
<br />
Standard practice in this case is to add a small perturbation to the theta field in the most recent atmosphere restart dump and then restart the simulation. This perturbation normally allows the model to proceed past the crash.<br />
<br />
First make a backup of the atmosphere restart file, then run the script '~access/apps/pythonlib/umfile_utils/perturbIC.py' on the restart file, e.g.<br />
<syntaxhighlight><br />
module purge<br />
module use ~access/modules<br />
module load pythonlib/umfile_utils<br />
<br />
# Make a backup<br />
cp restart_dump.astart restart_dump.astart.bak<br />
<br />
# Perturb the initial conditions<br />
~access/apps/pythonlib/umfile_utils/perturbIC.py restart_dump.astart<br />
</syntaxhighlight><br />
<br />
For Payu runs (ACCESS-ESM):<br />
* The error message can be found in the log file 'access.err' in the experiment directory<br />
* Restarts are in the highest numbered archive/restartNNN/atmosphere directory<br />
* The restart file is named 'restart_dump.astart'<br />
<br />
= Post Processing =<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://github.com/chloemackallah/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information. <br />
<br />
= Unsupported Configurations =<br />
<br />
These configurations have been used previously but are not presently being supported<br />
<br />
* [[ACCESS0.X Slab Reosc]]<br />
<br />
[[Category:ACCESS]][[Category:Coupled Models]][[Category:Atmosphere Models]][[Category:Ocean Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS&diff=3213ACCESS2021-07-19T02:32:42Z<p>S.wales: </p>
<hr />
<div>=Overview= <br />
<br />
The Australian Community Climate and Earth-System Simulator (ACCESS) is a coupled climate and earth system simulator to be developed as a joint initiative of the Bureau of Meteorology and CSIRO in cooperation with the university community in Australia.<br />
<br />
There are multiple configurations of ACCESS<br />
<br />
Our goal is to assist researchers in using ACCESS coupled model as provided on the computing cluster at NCI. If you have any corrections, comments or contributions, please feel free to email us on [mailto:cws_help@nf.nci.org.au cws_help@nf.nci.org.au] so that we can continue to update and upgrade the guides over time.<br />
<br />
For the full list of ACCESS guides see [[:Category:ACCESS]]<br />
<br />
<br />
== Model Versions ==<br />
<br />
=== Coupled ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 N48|ACCESS-CM 2.0 N48]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 Rescalable|ACCESS-CM 2.0 Rescalable]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48, N96 or N216 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS-ESM|ACCESS-ESM 1.5]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[CABLE|CABLE]] with carbon cycle<br />
|-<br />
| [[ACCESS-CM_1.4|ACCESS-CM 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_CMIP|ACCESS-CM 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_CMIP|ACCESS-CM 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[MOSES|MOSES]]<br />
|}<br />
<br />
<br />
&nbsp;<br />
<br />
=== Atmosphere-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS-AMIP_1.4|ACCESS-AMIP 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_AMIP|ACCESS-AMIP 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_AMIP|ACCESS-AMIP 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOSES|MOSES]]<br />
|}<br />
<br />
=== Ocean-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS-OM_2.0|ACCESS-OM2]]<br />
| &nbsp;<br />
| [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-025|ACCESS-OM2-025]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.25° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-01|ACCESS-OM2-01]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.1° + [[CICE|CICE]]<br />
|-<br />
| [[AusCOM|AusCOM]]<br />
| &nbsp;<br />
| [[MOM|MOM]] + [[CICE|CICE]]<br />
|}<br />
<br />
=Guides= <br />
<br />
For information on accessing the code and setup of the environment, see the [[Unified Model | UM page]]<br />
<br />
[https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin How to setup Pre-Industrial Control and Historical runs for CMIP5 Simulations] (ACCESS 1.0)<br />
<br />
* [[How to setup Historical Runs for CMIP5 Simulations in ACCESS 1.3]]<br />
* [[Building ACCESS]]<br />
* [[How to build ACCESS v1.4]]<br />
* [[ACCESS CMIP5 Forcings]]<br />
* [[DataMover]]<br />
<br />
== Coupled Model Crashes ==<br />
<br />
If the model becomes unstable it may crash with a stack trace mentioning 'bi_linear_h', like<br />
<syntaxhighlight><br />
um7.3x 0000000001111A08 mpl_recv_ 67 mpl_recv.F90<br />
um7.3x 000000000110AF66 gc_rrecv_ 168 gc_rrecv.F90<br />
um7.3x 0000000000989C5D bi_linear_h_ 613 bi_linear_h.f90<br />
um7.3x 0000000000CC92BC ritchie_ 2557 ritchie.f90<br />
um7.3x 00000000009DFB7D departure_point_ 382 departure_point.f90<br />
um7.3x 00000000008AA46F sl_thermo_ 681 sl_thermo.f90<br />
um7.3x 00000000006EC376 ni_sl_thermo_ 778 ni_sl_thermo.f90<br />
</syntaxhighlight><br />
<br />
Standard practice in this case is to add a small perturbation to the theta field in the most recent atmosphere restart dump and then restart the simulation. This perturbation normally allows the model to proceed past the crash.<br />
<br />
First make a backup of the atmosphere restart file, then run the script '~access/apps/pythonlib/umfile_utils/perturbIC.py' on the restart file, e.g.<br />
<syntaxhighlight><br />
module purge<br />
module use ~access/modules<br />
module load pythonlib/umfile_utils<br />
<br />
# Make a backup<br />
cp restart_dump.astart restart_dump.astart.bak<br />
<br />
# Perturb the initial conditions<br />
~access/apps/pythonlib/umfile_utils/perturbIC.py restart_dump.astart<br />
</syntaxhighlight><br />
<br />
For Payu runs (ACCESS-ESM):<br />
* The error message can be found in the log file 'access.err' in the experiment directory<br />
* Restarts are in the highest numbered archive/restartNNN/atmosphere directory<br />
* The restart file is named 'restart_dump.astart'<br />
<br />
= Unsupported Configurations =<br />
<br />
These configurations have been used previously but are not presently being supported<br />
<br />
* [[ACCESS0.X Slab Reosc]]<br />
<br />
[[Category:ACCESS]][[Category:Coupled Models]][[Category:Atmosphere Models]][[Category:Ocean Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS&diff=3212ACCESS2021-07-19T02:31:17Z<p>S.wales: </p>
<hr />
<div>=Overview= <br />
<br />
The Australian Community Climate and Earth-System Simulator (ACCESS) is a coupled climate and earth system simulator to be developed as a joint initiative of the Bureau of Meteorology and CSIRO in cooperation with the university community in Australia.<br />
<br />
There are multiple configurations of ACCESS<br />
<br />
Our goal is to assist researchers in using ACCESS coupled model as provided on the computing cluster at NCI. If you have any corrections, comments or contributions, please feel free to email us on [mailto:cws_help@nf.nci.org.au cws_help@nf.nci.org.au] so that we can continue to update and upgrade the guides over time.<br />
<br />
For the full list of ACCESS guides see [[:Category:ACCESS]]<br />
<br />
<br />
== Model Versions ==<br />
<br />
=== Coupled ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 N48|ACCESS-CM 2.0 N48]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 Rescalable|ACCESS-CM 2.0 Rescalable]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48, N96 or N216 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS-ESM|ACCESS-ESM 1.5]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[CABLE|CABLE]] with carbon cycle<br />
|-<br />
| [[ACCESS-CM_1.4|ACCESS-CM 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_CMIP|ACCESS-CM 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_CMIP|ACCESS-CM 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[MOSES|MOSES]]<br />
|}<br />
<br />
<br />
&nbsp;<br />
<br />
=== Atmosphere-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS-AMIP_1.4|ACCESS-AMIP 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_AMIP|ACCESS-AMIP 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_AMIP|ACCESS-AMIP 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOSES|MOSES]]<br />
|}<br />
<br />
=== Ocean-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS-OM_2.0|ACCESS-OM2]]<br />
| &nbsp;<br />
| [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-025|ACCESS-OM2-025]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.25° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-01|ACCESS-OM2-01]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.1° + [[CICE|CICE]]<br />
|-<br />
| [[AusCOM|AusCOM]]<br />
| &nbsp;<br />
| [[MOM|MOM]] + [[CICE|CICE]]<br />
|}<br />
<br />
=Guides= <br />
<br />
For information on accessing the code and setup of the environment, see the [[Unified Model | UM page]]<br />
<br />
[https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin How to setup Pre-Industrial Control and Historical runs for CMIP5 Simulations] (ACCESS 1.0)<br />
<br />
* [[How to setup Historical Runs for CMIP5 Simulations in ACCESS 1.3]]<br />
* [[Building ACCESS]]<br />
* [[How to build ACCESS v1.4]]<br />
* [[ACCESS CMIP5 Forcings]]<br />
* [[DataMover]]<br />
<br />
== Coupled Model Crashes ==<br />
<br />
If the model becomes unstable it may crash with a stack trace mentioning 'bi_linear_h', like<br />
<syntaxhighlight><br />
um7.3x 0000000001111A08 mpl_recv_ 67 mpl_recv.F90<br />
um7.3x 000000000110AF66 gc_rrecv_ 168 gc_rrecv.F90<br />
um7.3x 0000000000989C5D bi_linear_h_ 613 bi_linear_h.f90<br />
um7.3x 0000000000CC92BC ritchie_ 2557 ritchie.f90<br />
um7.3x 00000000009DFB7D departure_point_ 382 departure_point.f90<br />
um7.3x 00000000008AA46F sl_thermo_ 681 sl_thermo.f90<br />
um7.3x 00000000006EC376 ni_sl_thermo_ 778 ni_sl_thermo.f90<br />
</syntaxhighlight><br />
<br />
Standard practice in this case is to add a small perturbation to the theta field in the most recent atmosphere restart dump and then restart the simulation. This perturbation normally allows the model to proceed past the crash.<br />
<br />
First make a backup of the atmosphere restart file, then run the script '~access/apps/pythonlib/umfile_utils/perturbIC.py' on the restart file, e.g.<br />
<syntaxhighlight><br />
module purge<br />
module use ~access/modules<br />
module load pythonlib/umfile_utils<br />
<br />
# Make a backup<br />
cp restart_dump.astart restart_dump.astart.bak<br />
<br />
# Perturb the initial conditions<br />
~access/apps/pythonlib/umfile_utils/perturbIC.py restart_dump.astart<br />
</syntaxhighlight><br />
<br />
For Payu runs (ACCESS-ESM):<br />
- The error message can be found in the log file 'access.err' in the experiment directory<br />
- Go to the most recent archive/restartNNN directory<br />
- The restart file is named 'restart_dump.astart'<br />
<br />
= Unsupported Configurations =<br />
<br />
These configurations have been used previously but are not presently being supported<br />
<br />
* [[ACCESS0.X Slab Reosc]]<br />
<br />
[[Category:ACCESS]][[Category:Coupled Models]][[Category:Atmosphere Models]][[Category:Ocean Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3201ACCESS-ESM 1.52021-07-16T01:58:43Z<p>S.wales: </p>
<hr />
<div><br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
1. Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
2. Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
3. Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
4. Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://github.com/chloemackallah/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3200ACCESS-ESM 1.52021-07-16T01:58:04Z<p>S.wales: </p>
<hr />
<div>{{Needs Update|Needs update for Gadi}}<br />
{{Stub}}<br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
1. Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
2. Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
3. Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
4. Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://github.com/chloemackallah/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS-ESM_1.5&diff=3199ACCESS-ESM 1.52021-07-16T01:57:31Z<p>S.wales: </p>
<hr />
<div>{{Needs Update|Needs update for Gadi}}<br />
{{Stub}}<br />
This page describes the different configurations of ACCESS-ESM 1.5 currently available and how to access them. <br />
<br />
ESM stands for Earth System Model. ACCESS-ESM is a coupled model with the following components:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Submodel !! Model !! Version !! Notes<br />
|-<br />
| Atmosphere || [[UM]] || 7.3 ||<br />
|-<br />
| Land Surface || [[CABLE]] || Part of UM ||<br />
|-<br />
| Sea Ice || CICE || ||<br />
|-<br />
| Ocean || [[MOM]] || 5 ||<br />
|-<br />
| Coupler || OASIS-MCT || 3 ||<br />
|}<br />
<br />
__TOC__<br />
<br />
= Payu Experiments =<br />
<br />
We strongly recommend that you run ACCESS-ESM 1.5 using the payu driver. CSIRO is running the CMIP6 submission runs using a setup based on KSH scripts. The CMS team is working to adapt those configurations to payu. <br />
<br />
Please contact [mailto:cws_help@nci.org.au the CMS team] if you would like a configuration that isn't currently available in payu or if you have any problems running a ACCESS-ESM 1.5 simulation with payu.<br />
<br />
* [https://github.com/coecms/esm-pre-industrial piControl]<br />
* [https://github.com/coecms/esm-historical historical]<br />
* [https://github.com/coecms/esm-ssp585 ssp585]<br />
* [https://github.com/coecms/esm-pmip-last-mil last millenium]<br />
* [https://github.com/coecms/esm-mh mid holocene]<br />
* [https://github.com/coecms/esm-lig last interglacial]<br />
<br />
== Model Spinup ==<br />
<br />
If starting from scratch the model will take some time to spin up. It's recommended that you branch off of piControl, which has already been spun up, rather than starting a run from scratch (except for SSP runs, which will branch from historical)<br />
<br />
The script 'warm-start.sh' in the Payu experiment directory will set up restart files based on an existing run<br />
<br />
== Using Payu ==<br />
<br />
# Load the [[Conda]] environment to access Payu<br />
<syntaxhighlight><br />
module use /g/data/hh5/public/modules<br />
module load conda/analysis3<br />
</syntaxhighlight><br />
<br />
# Download the Payu experiment configuration<br />
<syntaxhighlight><br />
git clone https://github.com/coecms/esm-historical<br />
</syntaxhighlight><br />
<br />
# Edit and run the warm-start script if needed<br />
<syntaxhighlight><br />
./warm-start.sh<br />
</syntaxhighlight><br />
<br />
# Run the model<br />
<syntaxhighlight><br />
payu run<br />
</syntaxhighlight><br />
<br />
Each 'payu run' submission will run the model for one model year. You can run multiple years with e.g. 'payu run -n 25' - Payu will automatically resubmit the run after each year has finished.<br />
<br />
We recommend inspecting the model output after running for a year or so to make sure it's behaving as desired, especially if it's a non-standard configuration<br />
<br />
== Post-Processing ==<br />
<br />
Post-processing scripts to convert atmosphere output to NetCDF format and move the output to /g/data are available at https://github.com/chloemackallah/ACCESS-Archiver. This can be run at any stage of the run, and will only process files that haven't yet been archived.<br />
<br />
Publication of runs to ESGF requires further processing to put the data in CMOR format. We recommend working with CSIRO for this so all ACCESS runs are consistent, contact us at cws_help@nci.org.au for more information.<br />
<br />
== Experiment Details ==<br />
<br />
=== piControl ===<br />
You can download the configuration by typing<br />
<syntaxhighlight lang="bash">$ git clone https://github.com/coecms/esm-pre-industrial.git<br />
</syntaxhighlight><br />
<br />
[https://github.com/coecms/esm-pre-industrial The website] of the configuration also contains detailed instructions on how to run the model.<br />
<br />
Default configuration, based off CSIRO ksh scripts<br />
<br />
=== PMIP4 Runs ===<br />
Main Page:[[ACCESS-ESM-PMIP4|ACCESS-ESM-PMIP4]]<br />
<br />
Currently contains two models from the PMIP4 setting: Last Interglacial and Mid-Holocene. <br />
<br />
Please refer to the main page to know where to download the configurations from.<br />
<br />
=== ''Last Interglacial'' ===<br />
Science Contacts: Nick Yeung, Laurie Menviel<br />
<br />
Note: In addition to changed MMRs for certain gases, it also needs an updated UM build with different orbital parameters. This updated UM build is on gadi at<br />
<syntaxhighlight lang="bash">/g/data/access/payu/access-esm-pmip/pmip-li/bin/um_hg3_20200706_pmip-li_r344.exe<br />
</syntaxhighlight><br />
<br />
=== ''Mid-Holocene'' ===<br />
Science Contacts: Josephine Brown<br />
<br />
Note: In addition to changed MMRs for certain trace gasses, it also needs and updated UM build with different orbital parameters. This updated UM build is at<br />
<div style="background:#eeeeee;border:1px solid #cccccc;padding:5px 10px;"><tt>/g/data/access/payu/access-esm-pmip/pmip-mh/bin/um_hg3_20200706_pmip-mh_r345.exe</tt></div><br />
<br />
= CSIRO KSH Scripts =<br />
<br />
CSIRO is running the ACCESS-ESM model using ksh shell scripts. If you need to know, you can look [[ACCESS-ESM-CSIRO|here]].<br />
<br />
<br />
[[Category:ACCESS 1.5]] [[Category:Coupled Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3120Conda2021-07-13T02:21:03Z<p>S.wales: /* 20.10 */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data3/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3119Conda2021-07-13T02:20:38Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data3/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 (Current Unstable) ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
=== 21.04 (Current Stable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Unsupported) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== <s>20.10</s> ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3051Conda2021-07-05T06:22:42Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data3/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.07 ===<br />
<br />
Python has been updated to version 3.9. See what's new at https://docs.python.org/3/whatsnew/3.9.html<br />
<br />
=== 21.04 (Current Unstable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Current Stable) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== 20.10 (Unsupported) ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS_2_Rescalable&diff=3049ACCESS 2 Rescalable2021-07-02T01:04:49Z<p>S.wales: Created page with " ACCESS-CM 2.0, with a changeable atmosphere resolution === Experiments: === *u-ce694: N216 Historical, based off of u-bm652 (CSIRO Gadi Historical) === Performance ===..."</p>
<hr />
<div><br />
[[ACCESS-CM 2.0]], with a changeable atmosphere resolution<br />
<br />
=== Experiments: ===<br />
<br />
*u-ce694: N216 Historical, based off of u-bm652 (CSIRO Gadi Historical)<br />
<br />
=== Performance ===<br />
<br />
The experiment hasn't yet been optimised for performance, please contact CMS before running the model so it can be optimised.<br />
<br />
Disabling UKCA chemistry and EasyAerosol aerosols should improve performance at the cost of no longer matching N96 behaviour<br />
<br />
=== Changing Resolutions ===<br />
<br />
In the Rose editor:<br />
* In um->namelist->Reconfiguration and Ancillary Control->Output dump grid sizes set 'global_row_length' and 'global_rows' to 2*N and 3*N/2 respectively (e.g. for N216 values are 432, 324)<br />
* In regrid_coupling->namelist->regrid_cm2 set atmosphere_endgame_n to the resolution to use for this run<br />
* Make sure reconfiguration is enabled<br />
<br />
=== Setting up new experiments ===<br />
<br />
The 'regrid_coupling' task regrids any input files to the target resolution, placing the newly regridded files in 'output_dir'.<br />
<br />
* 'ancils' is UM format ancillary files (e.g. ozone)<br />
* 'netcdf_ancils' is NetCDF format files (e.g. UKCA emissions)<br />
* 'set_flexible_calendar' is needed if you get error messages that UKCA oxidant files have the wrong calendar set<br />
* 'easyaerosol_dir' is the directory with EasyAerosol files, as set in um->namelist->easy_aerosol<br />
<br />
The directories for these files in the um section then needs to be changed to the 'output_dir' of 'regrid_coupling', by default this is '$WEIGHTS_DIR' (share/data/weights)<br />
<br />
The Oasis file paths shouldn't need to be changed unless the ocean model grid changes<br />
<br />
=== How it works ===<br />
<br />
Task 'regrid_coupling' regrids the original N96 resolution data files to the target resolution, and creates the Oasis weights files for the model coupling<br />
<br />
Task 'reconfiguration' regrids the UM astart dump to the target resolution, however it skips the CABLE fields<br />
<br />
Task 'regrid_cable' takes the output of the reconfiguration and adds regridded CABLE fields from the original astart file.</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS&diff=3048ACCESS2021-07-02T00:46:46Z<p>S.wales: /* Coupled */</p>
<hr />
<div>=Overview= <br />
<br />
The Australian Community Climate and Earth-System Simulator (ACCESS) is a coupled climate and earth system simulator to be developed as a joint initiative of the Bureau of Meteorology and CSIRO in cooperation with the university community in Australia.<br />
<br />
There are multiple configurations of ACCESS<br />
<br />
Our goal is to assist researchers in using ACCESS coupled model as provided on the computing cluster at NCI. If you have any corrections, comments or contributions, please feel free to email us on [mailto:cws_help@nf.nci.org.au cws_help@nf.nci.org.au] so that we can continue to update and upgrade the guides over time.<br />
<br />
For the full list of ACCESS guides see [[:Category:ACCESS]]<br />
<br />
<br />
== Model Versions ==<br />
<br />
=== Coupled ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-CM 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 N48|ACCESS-CM 2.0 N48]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS 2 Rescalable|ACCESS-CM 2.0 Rescalable]]<br />
| <br />
| [[Unified_Model|UM]] 10.6 @ N48, N96 or N216 resolution + [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS-ESM|ACCESS-ESM 1.5]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] 5 @ 1°+ [[CICE|CICE]] + [[CABLE|CABLE]] with carbon cycle<br />
|-<br />
| [[ACCESS-CM_1.4|ACCESS-CM 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_CMIP|ACCESS-CM 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_CMIP|ACCESS-CM 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOM|MOM]] + [[CICE|CICE]] + [[MOSES|MOSES]]<br />
|}<br />
<br />
<br />
&nbsp;<br />
<br />
=== Atmosphere-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS_2.0|ACCESS-AMIP 2.0]]<br />
| [[CMIP#CMIP_6|CMIP 6]]<br />
| [[Unified_Model|UM]] 10.6 + [[JULES|JULES]]<br />
|-<br />
| [[ACCESS-AMIP_1.4|ACCESS-AMIP 1.4]]<br />
| &nbsp;<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]] 2.0<br />
|-<br />
| [[ACCESS1.3_AMIP|ACCESS-AMIP 1.3]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[CABLE|CABLE]]<br />
|-<br />
| [[ACCESS1.0_AMIP|ACCESS-AMIP 1.0]]<br />
| [[CMIP#CMIP_5|CMIP 5]]<br />
| [[Unified_Model|UM]] 7.3 + [[MOSES|MOSES]]<br />
|}<br />
<br />
=== Ocean-Only ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version<br />
! CMIP<br />
! Description<br />
|-<br />
| [[ACCESS-OM_2.0|ACCESS-OM2]]<br />
| &nbsp;<br />
| [[MOM|MOM]] 5 @ 1° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-025|ACCESS-OM2-025]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.25° + [[CICE|CICE]]<br />
|-<br />
| [[ACCESS-OM2-01|ACCESS-OM2-01]]<br />
|<br />
| [[MOM|MOM]] 5 @ 0.1° + [[CICE|CICE]]<br />
|-<br />
| [[AusCOM|AusCOM]]<br />
| &nbsp;<br />
| [[MOM|MOM]] + [[CICE|CICE]]<br />
|}<br />
<br />
=Guides= <br />
<br />
For information on accessing the code and setup of the environment, see the [[Unified Model | UM page]]<br />
<br />
[https://accessdev.nci.org.au/trac/wiki/access/ACCESS1.0_raijin How to setup Pre-Industrial Control and Historical runs for CMIP5 Simulations] (ACCESS 1.0)<br />
<br />
* [[How to setup Historical Runs for CMIP5 Simulations in ACCESS 1.3]]<br />
* [[Building ACCESS]]<br />
* [[How to build ACCESS v1.4]]<br />
* [[ACCESS CMIP5 Forcings]]<br />
* [[DataMover]]<br />
<br />
= Unsupported Configurations =<br />
<br />
These configurations have been used previously but are not presently being supported<br />
<br />
* [[ACCESS0.X Slab Reosc]]<br />
<br />
[[Category:ACCESS]][[Category:Coupled Models]][[Category:Atmosphere Models]][[Category:Ocean Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3027Conda2021-06-29T04:55:49Z<p>S.wales: /* Update History */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data3/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.04 (Current Unstable) ===<br />
<br />
Conda has dropped support for the old operating system version used on the Accessdev VM. It is likely that future versions of analysis3 after 21.04 will not work on Accessdev, consider setting up tasks to run on Gadi if they require the Conda environment.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics/ML'''<br />
** [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
** [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
** [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
* '''Geospatial'''<br />
** [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
<br />
* '''Visualisation'''<br />
** [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
<br />
* '''Developer Tools'''<br />
** [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
** [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
** [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
** [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
<br />
=== 21.01 (Current Stable) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== 20.10 (Unsupported) ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Conda&diff=3026Conda2021-06-29T04:48:27Z<p>S.wales: /* 21.04 (Current Unstable) */</p>
<hr />
<div><br />
== Conda Python Environments ==<br />
<br />
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.<br />
<br />
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded.<br />
<br />
To use any of the conda environments:<br />
<br />
#Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) <br />
#You must first run (to do at each session) <br />
<br />
<code>module use /g/data3/hh5/public/modules</code><br />
<br />
You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file<br />
<br />
If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.<br />
<syntaxhighlight lang="none">#PBS -l storage=gdata/hh5<br />
</syntaxhighlight><br />
<br />
=== Stable Environment ===<br />
<br />
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Gadi. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.<br />
<br />
<code>module load conda/analysis3</code><br />
<br />
=== Unstable Environment ===<br />
<br />
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.<br />
<br />
<code>module load conda/analysis3-unstable</code><br />
<br />
When we do our quarterly update the unstable environment becomes the new stable environment.<br />
<br />
=== Removed Environments ===<br />
<br />
Normally after three quarters have passed old environments are removed, to reduce disk space and support burden. Conda environment.yml descriptions of past environments are available at https://github.com/coecms/conda-history.<br />
<br />
== Creating personal environments ==<br />
<br />
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like:<br />
<br />
&nbsp;<br />
<syntaxhighlight lang="none"><br />
auto_activate_base: false<br />
envs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/envs<br />
- /g/data/hh5/public/apps/miniconda3/envs<br />
pkgs_dirs:<br />
- /scratch/$PROJECT/$USER/conda/pkgs<br />
conda-build:<br />
root-dir: /scratch/$PROJECT/$USER/conda/bld<br />
</syntaxhighlight><br />
<br />
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.<br />
<br />
== Interactive Analysis / Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
Note that on Windows the Jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Requesting new packages ==<br />
<br />
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. Please include a link to the package documentation to your request.<br />
<br />
It would be appreciated if you can check the package isn't already installed before putting in a request. To do so, please load the unstable environment and use <code>conda list</code> to list the packages included in that environment<br />
<br />
As a general rule we will only install packages from the [https://conda-forge.org/feedstocks/ 'conda-forge'] channel. Newly installed packages will be available in the conda/analysis3-unstable environment.<br />
<br />
== Update History ==<br />
<br />
=== 21.04 (Current Unstable) ===<br />
<br />
- [https://github.com/spyder-ide/spyder-kernels spyder-kernels] allow conda environment to work with [https://www.spyder-ide.org/ Spyder IDE]<br />
- [https://xgboost.readthedocs.io/en/latest/index.html dask-xgboost] optimized distributed gradient boosting library<br />
- [https://www.atlassian.com/git/tutorials/git-subtree git-subtree] subtree extension for Git<br />
- [https://xarray-spatial.org/ xarray-spatial] common raster analysis functions<br />
- [https://github.com/rasbt/watermark watermark] IPython magic extension for printing date and time stamps, version numbers, and hardware information<br />
- [https://www.cvxpy.org/ cvxpy] modeling language for convex optimization problems<br />
- [https://rechunker.readthedocs.io/en/latest/ rechunker] efficient and scalable manipulation of the chunk structure of chunked array formats such as Zarr and TileDB<br />
- [https://github.com/jbusecke/xmovie xmovie] simple way of creating beautiful movies from xarray objects<br />
- [https://github.com/mmhs013/pymannkendall pymannkendall] analyze time series data for consistently increasing or decreasing trends<br />
<br />
=== 21.01 (Current Stable) ===<br />
<br />
The conda environment now uses a conda provided OpenMPI, rather than Gadi's OpenMPI module. This impacts users of mpi4py, esmf, esmpy and xesmf<br />
<br />
era5grib, the tool for converting NCI's ERA5 archive to GRIB format for use in UM/WRF limited area runs, now uses the NCI managed ERA5 archive in projects [https://opus.nci.org.au/display/ERA5/ERA5+Community+Home rt52 and zz93]. The new archive has global coverage, allowing limited area models to be run anywhere on the globe. The previously used CLEX archive will be removed on the 28th April to free up disk space, until this happens previous behavior can be accessed using the flag '--source CLEX'.<br />
<br />
'''Notable new packages'''<br />
<br />
* '''Statistics'''<br />
** [https://ml.dask.org/index.html dask-ml] Scalable machine learning<br />
** [https://geostat-framework.readthedocs.io/projects/pykrige/en/stable/ pykrige] Gaussian process regression toolkit<br />
** [https://nctoolkit.readthedocs.io/en/latest/ nctoolkit] Toolkit for analysing NetCDF data<br />
<br />
* '''Geospatial'''<br />
** [http://xarray-spatial.org/ xarray-spatial] Raster-based spatial analysis<br />
** [https://pypi.org/project/Fiona/ fiona] GDAL vector API for Python<br />
<br />
* '''Visualisation'''<br />
** [https://residentmario.github.io/geoplot/index.html geoplot] Geospatial data visualisation<br />
** [https://python-visualization.github.io/folium/ folium] Leaflet webpage map manipulation<br />
** [https://github.com/jwass/mplleaflet mplleaflet] Use matplotlib on Leaflet webpage maps<br />
<br />
* '''Developer Tools'''<br />
** [https://mamba.readthedocs.io/en/latest/ mamba] Faster conda environment setup<br />
** [https://github.com/psf/black black] Python code formatter<br />
** [https://intake-esm.readthedocs.io/en/latest/ intake-esm] Data cataloguing tool<br />
<br />
=== 20.10 (Unsupported) ===<br />
<br />
Python has been updated to 3.8 ([https://docs.python.org/3.8/whatsnew/3.8.html changes])<br />
<br />
There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working<br />
<br />
*pynio (https://github.com/conda-forge/pynio-feedstock/issues/90)<br />
*<s>cf_units</s> (was renamed to 'cf-units')<br />
*<s>pymunge</s> (fixed)<br />
*<s>ants</s> (fixed)<br />
<br />
'''Notable new packages'''<br />
<br />
*[https://github.com/corteva/rioxarray rioxarray] geospatial xarray extension powered by rasterio <br />
*[https://github.com/cupy/cupy cupy] CUDA accellerated numpy<br />
*[https://github.com/jwkvam/celluloid celluloid] simplified animations with matplotlib<br />
* [https://pythonhosted.org/rasterstats/ rasterstats] summarizing geospatial raster datasets based on vector geometries<br />
* [https://github.com/python-windrose/windrose windrose] manage wind data, draw windrose (also known as a polar rose plot), draw probability density function and fit Weibull distribution<br />
* [https://pyam-iamc.readthedocs.io/en/stable/ pyam] analysis and visualization of integrated-assessment scenarios<br />
* [https://github.com/apache/arrow pyarrow] a cross-language development platform for in-memory data <br />
* [https://github.com/jdowner/gist python-gist] a command line interface for working with github gists<br />
* [https://sourceforge.net/projects/mcmc-jags/ jags] statistical analysis of Bayesian hierarchical models by Markov Chain Monte Carlo<br />
<br />
=== <s>20.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) <br />
*[http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis <br />
*[https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) <br />
*[https://pypi.org/project/descartes/ descartes] - plot shapefiles <br />
*[https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) <br />
*[https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output <br />
<br />
=== <s>20.04 (Unsupported)</s> ===<br />
<br />
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes<br />
<br />
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI<br />
<br />
'''Notable New Packages'''<br />
<br />
*geopy: Locate lat/lon coordinates of places <br />
*ninja: mom6 build system <br />
<br />
=== <s>20.01 (Unsupported)</s> ===<br />
<br />
Python has been updated to 3.7<br />
<br />
'''Notable New Packages'''<br />
<br />
*xlrd: Read excel files <br />
*ants: Unified Model Ancillary tools <br />
*climtas: Dask-aware Xarray timeseries processing <br />
<br />
=== <s>19.10 (Unsupported)</s> ===<br />
<br />
basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages<br />
<br />
'''Notable New Packages'''<br />
<br />
*addmeta <br />
*[http://holoviews.org/ holoviews]: Fancy plots <br />
*[https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes <br />
<br />
=== <s>19.07 (Unsupported)</s> ===<br />
<br />
'''Notable New Packages'''<br />
<br />
*pyferret <br />
*pyngl <br />
*pynio <br />
*xgcm <br />
*xrft <br />
<br />
=== <s>19.04 (unsupported)</s> ===<br />
<br />
'''Notable Changes'''<br />
<br />
*arccssive has been renamed to clef <br />
*A bug preventing Iris from opening UM files without a date has been fixed <br />
<br />
'''Notable New Packages'''<br />
<br />
*[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations <br />
*[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files <br />
*[https://github.com/SciTools/cf-units cfunits] Convert between CF units <br />
*[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py <br />
*[https://github.com/intake/intake intake] Lightweight data catalogues <br />
*[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files <br />
*[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays <br />
<br />
[[Category:NCI]] [[Category:Python]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Nesting_Suite&diff=2884Nesting Suite2021-06-01T04:19:49Z<p>S.wales: </p>
<hr />
<div>[[Category: Unified Model]]<br />
<br />
Up-to-date information and news about the nested suite can be found on the RMED wiki and Rose issue tracker:<br />
* https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting<br />
* https://code.metoffice.gov.uk/trac/roses-u/ticket/52<br />
<br />
Job Ids on Gadi:<br />
* u-bm651: Copy of u-bi769 (UM vn11.4 suite), used for 2.2km nest of [[Aus400]]<br />
* u-cb265: Copy of u-bi769, set up for era5 boundary conditions</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Nesting_Suite&diff=2883Nesting Suite2021-06-01T04:19:21Z<p>S.wales: </p>
<hr />
<div>[[Category: Unified Model]]<br />
<br />
Up-to-date information and news about the nested suite can be found on the RMED wiki and Rose issue tracker:<br />
* https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting<br />
* https://code.metoffice.gov.uk/trac/roses-u/ticket/52<br />
<br />
Job Ids on Gadi:<br />
- u-bm651: Copy of u-bi769 (UM vn11.4 suite), used for 2.2km nest of [[Aus400]]<br />
- u-cb265: Copy of u-bi769, set up for era5 boundary conditions</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Nesting_Suite&diff=2881Nesting Suite2021-06-01T04:12:21Z<p>S.wales: S.wales moved page U-aa753 Nested Suite to Nesting Suite</p>
<hr />
<div>{{Needs Update}}<br />
[[Category: Unified Model]]<br />
<br />
Up-to-date information and news about the nested suite can be found on the RMED wiki and Rose issue tracker:<br />
* https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting<br />
* https://code.metoffice.gov.uk/trac/roses-u/ticket/52<br />
<br />
==Nesting inside ERA-Interim== <br />
<br />
The suite can be driven from ERA-Interim analyses with a small amount of setup.<br />
<br />
At NCI we have 6 hourly ERA-Interim analyses in GRIB format under the directory /g/data1/ub4/erai/grib. As ERA-Interim is under licence, make sure you comply with requirements [[ERA INTERIM | here]] before starting <br />
<br />
Currently code changes required to run from ERA-Interim are in the job '''u-at264''', which is based on the 2017 nesting suite.<br />
<br />
Make a copy of this job with<br />
<syntaxhighlight lang=text><br />
rosie copy u-at264<br />
</syntaxhighlight><br />
<br />
Open the job in `<span style="font-family:monospace">rose edit</span>` and make the following changes to '''suite conf''' to enable ERA-Interim driving:<br />
* In 'General run options' make sure SITE is NCI<br />
* In 'Driving model setup', set:<br />
** 'dm_name' to 'ECMWF forecast fields'<br />
** 'dm_ec_ic_lbc_dir' to '<span style="font-family:monospace">\$ROSE_DATA/erai</span>'<br />
** 'dm_ec_cb_freq' to '21600' (6 hours, the ERA-Interim analysis frequency)<br />
<br />
To run from ERA interim the suite needs to generate lateral boundary conditions from the ERA-interim GRIB files. To enable this you need to create a new 'Resolution setup' that is a bit larger than your coarsest LAM domain. This resolution won't be used to run the model, instead the ERA interim data will be interpolated to it so that the boundary conditions can be cut out for the LAM. For instance if your coarsest LAM is a 200x200 grid with resolution 0.1 x 0.1 degrees, a 220x220 grid at the same resolution would be fine - this gives a border of 10 grid points around each edge to use for the boundary conditions. You could also use less grid points at a coarser resolution, though be cautious as this adds an extra interpolation step.<br />
<br />
You will need some extra ancillary files when using ERA-Interim, set the following in your new resolution, as well as your coarsest LAM resolution:<br />
* vegsoil_anc: True<br />
* lai_anc: True<br />
* alb_anc: False<br />
* orog_strm: False<br />
* 3D_ancils: True<br />
* sstsmc_anc: True<br />
<br />
Set the `<span style="font-family:monospace">nmods</span>` to 0, since we just want the ancillaries at this resolution (this will print a warning in Rose, which can be ignored)<br />
<br />
To make the ERA interim driver use this resolution you just set up, in the 'Driving model setup' page set '<span style="font-family:monospace">dm_ec_lam_ancil_dir}}' to the new resolution's ancillary directory. For instance, if the new resolution is `resn_2` of region `Regn1`, set `{{dm_ec_lam_ancil_dir}}` to `{{\$ROSE_DATA/ancils/Regn1/resn_2</span>`<br />
<br />
Note that the resolution of the ERA Interim source dataset is ~ 0.75 x 0.75 degrees, keep this in mind when choosing the resolution of your coarsest LAM (i.e. make sure your LAM isn't inside a single ERA Interim grid point!).<br />
<br />
https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting/ECdriver<br />
<br />
[[include component="page" wikiName="climate-cms" page="Index UM" editable="1"]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=U-aa753_Nested_Suite&diff=2882U-aa753 Nested Suite2021-06-01T04:12:21Z<p>S.wales: S.wales moved page U-aa753 Nested Suite to Nesting Suite</p>
<hr />
<div>#REDIRECT [[Nesting Suite]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Training&diff=2722Training2021-05-13T04:45:13Z<p>S.wales: </p>
<hr />
<div>== Training Needs ==<br />
<br />
=== General ===<br />
<br />
* [[NCI|Accessing and using NCI]]<br />
* [[Programming]]<br />
* [[ProgrammingTools]]<br />
<br />
== CMS Presentations ==<br />
<br />
CMS runs weekly training sessions presenting computational and data related topics<br />
<br />
To get updates on our next training sessions '''[https://www.lists.unsw.edu.au/mailman/listinfo/training-coe sign up]''' to our email list<br />
<br />
All our training videos are available at our '''[https://www.youtube.com/channel/UCSmoK6oWV9O0Hmyt9UdDNsQ Youtube channel]'''<br />
<br />
* [https://www.youtube.com/watch?v=aYukDeA-8hE Python Basics] ([https://www.youtube.com/watch?v=-UktTJDh_48 Part 2]) ([https://www.youtube.com/watch?v=pCabwrUa26E Part 3])<br />
* [https://www.youtube.com/watch?v=2GmFf9x1qg0 Introduction to NCI]<br />
* [https://www.youtube.com/watch?v=5z75iHJyT8Q Large Dataset Analysis with Jupyter]<br />
* [https://www.youtube.com/watch?v=aNUe-ukitpU Jupyter Introduction]<br />
* [https://www.youtube.com/watch?v=pk-UD_4ZAcE WRF at NCI]<br />
* [https://www.youtube.com/watch?v=DzbtG3UiMUw WRF Q&A]<br />
* [[Training:NCI Storage Monitoring | NCI Storage Monitoring]] 2019-10-02<br />
* [https://youtu.be/4tzug0DMZVI Licensing for Data and Software] 2019-07-10<br />
* [[Training: Basic Python | Basic Python]] 2019-06-05 - 2019-07-03<br />
* [[Training:Big Data | Analysing Big Data]] 2019-05-15 - 2019-05-22<br />
* [https://youtu.be/gYKc1mJitEw Data at NCI] 2019-05-08<br />
* [[Git Introduction old]] 2019-04-10 - 2019-04-17<br />
* [[Git Introduction]] 2020-09-09 - 2020-09-16 <br />
* [[Bash Introduction]] 2019-03-13 - 2019-04-03<br />
* [[WRF Introduction]] 2019-10-23 - 2019-10-30<br />
* [http://climate-cms.wikis.unsw.edu.au/Category:Setting_up Setting up]<br />
<br />
== Software Carpentry ==<br />
<br />
For general programming training we recommend the material from [https://software-carpentry.org/ Software Carpentry], a group whos focus is teaching researchers programming skills. Check with your University if they run Software Carpentry Workshops.<br />
<br />
[[Category:Training]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Getting_started_on_Windows&diff=2715Getting started on Windows2021-05-12T06:38:11Z<p>S.wales: /* X11 */</p>
<hr />
<div>[[Category:Setting up]]<br />
Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way.<br />
<br />
You should be able to:<br />
* Connect to NCI to access models and data<br />
* Develop, test and run Python, Fortran etc. programs locally<br />
* Use a common interface on your local computer and NCI<br />
<br />
For more information on how to use the terminal and Bash, check out the [http://swcarpentry.github.io/shell-novice/01-intro/index.html Software Carpentry Bash course]<br />
<br />
'''[https://youtu.be/gnldlGztqpU Video Introduction]'''<br />
<br />
= NCI Virtual Desktop (VDI) =<br />
<br />
NCI's virtual desktops (VDI) are an easy way to connect to NCI facilities from desktop computers. VDI provides a Linux desktop in a window, running on NCI's cloud<br />
<br />
Check NCI's VDI documentation for the current installation and setup instructions https://vdi.nci.org.au/help, and the [[Getting started on Linux]] page for details on using the Linux environment<br />
<br />
'''Advantages:'''<br />
* Access to NCI's /g/data filesystem<br />
* [[Conda]] and Matlab (see NCI documentation) available<br />
* No compute time/SU charge when running analyses on VDI<br />
* Doesn't require a powerful home computer<br />
<br />
'''Disadvantages:'''<br />
* VDI servers are shared between multiple users, can run out of memory<br />
* Requires network connection - can't use VDI on a plane etc.<br />
<br />
= Windows Subsystem for Linux (WSL) =<br />
<br />
Windows subsystem for Linux (WSL) is a way to have a Linux environment on your Windows computer, without having to install and switch between different operating systems. <br />
<br />
For up-to-date WSL installation instructions, see Microsoft's documentation at https://docs.microsoft.com/en-us/windows/wsl/install-win10. There are a few different Linux variants you can install, a good starting point is Ubuntu if you can't decide.<br />
<br />
Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu<br />
<syntaxhighlight><br />
sudo apt install vim<br />
</syntaxhighlight><br />
will install the vim text editor.<br />
<br />
With WSL installed you can then follow the [[Getting started on Linux]] instructions to set up SSH etc.<br />
<br />
'''Advantages:'''<br />
* Runs on your own computer<br />
* Install whatever programs you need<br />
<br />
'''Disadvantages:'''<br />
* Large size (~2GB for base install, another GB or so for Anaconda)<br />
* Remote X11 windows can be slow if you have a bad connection<br />
<br />
== Windows Terminal ==<br />
<br />
Windows Terminal is a nicer command line interface for Windows than the default Power Shell program, with more customisation options. It integrates with WSL, letting you use it to start Bash terminal sessions<br />
<br />
Windows terminal can be installed through the Windows app store, further details at<br />
https://docs.microsoft.com/en-us/windows/terminal/<br />
<br />
== X11 ==<br />
<br />
In order to show remote windows from Gadi on your own screen you'll need an X11 server program running on your computer.<br />
<br />
If X11 isn't running, you will get errors when starting programs that bring up a new window, e.g:<br />
<syntaxhighlight lang="bash"><br />
saw562@gadi-login-04 ~ $ xclock<br />
Error: Can't open display: <br />
</syntaxhighlight><br />
<br />
There are a few different X11 server programs available for Windows, including:<br />
* https://sourceforge.net/projects/vcxsrv/<br />
* https://github.com/microsoft/wslg<br />
<br />
If you encounter problems with WSL 2 and VcXsrv, check that:<br />
* On your machine the environment variable DISPLAY is set using <syntaxhighlight>export DISPLAY=$(ip route list default | awk '{print $3}'):0.0</syntaxhighlight><br />
* Incomming connections to VcXsrv are allowed through your firewall. In the start menu, search for 'Windows Defender Firewall with Advanced Security'. In the list 'Inbound Rules' find all entries for 'VcXsrv windows xserver' and make sure they have a green check. If they don't double click the entry and change the 'Action' to 'Allow the connection'.<br />
<br />
== VSCode & Power Shell SSH ==<br />
<br />
[[VS Code]] is a nice code editor that can run on Windows, Mac and Linux. You can edit code locally, or use plugins to remotely connect to servers over SSH. It also integrates with WSL allowing you to edit code in the Linux environment, and with Anaconda, letting you run Python programs in different environments.<br />
<br />
Windows includes its own version of SSH that you can use directly from Power Shell. Visual Studio Code can use this to connect to Gadi from your Windows computer. You will need some special options when creating a SSH key with Power Shell SSH:<br />
<syntaxhighlight lang="powershell"><br />
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa<br />
</syntaxhighlight><br />
Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of the new public key file .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually.<br />
<br />
The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.:<br />
<syntaxhighlight><br />
Host gadi<br />
HostName gadi.nci.org.au<br />
User saw562 # Replace with your own username<br />
ForwardAgent yes<br />
</syntaxhighlight><br />
<br />
You can also use the Power Shell SSH to connect to Gadi, bypassing WSL, but that may have [https://github.com/PowerShell/Win32-OpenSSH/issues/1515 problems running X11 windows]<br />
<br />
https://code.visualstudio.com/<br />
<br />
= Software =<br />
== NCI Jupyter ==<br />
<br />
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.<br />
<br />
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.<br />
<br />
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.<br />
<br />
On Windows, the jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).<br />
<br />
== Anaconda Python / Rstudio ==<br />
<br />
We recommend using [https://www.anaconda.com/products/individual Anaconda] if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the [[Conda]] environments at NCI. Anaconda also supports Rstudio if you work with R.<br />
<br />
You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported.<br />
<br />
== Matlab ==<br />
<br />
Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server from your own computer.<br />
<br />
You can also use Matlab on NCI's systems via VDI, however you need to use your university license. See the [https://opus.nci.org.au/display/Help/3.+VDI+software VDI docs] or contact help@nci.org.au for more information.<br />
<br />
= Other Alternatives =<br />
<br />
== Putty ==<br />
<br />
Putty is an alternative SSH implementation for Windows<br />
<br />
https://www.putty.org/<br />
<br />
== Cygwin ==<br />
<br />
Cygwin works similarly to WSL, it lets you run Linux programs on a Windows computer<br />
<br />
https://www.cygwin.com/<br />
<br />
Git for windows includes a Bash terminal based on Cygwin as well<br />
<br />
https://gitforwindows.org/<br />
<br />
== File transfers ==<br />
It is sometimes useful to transfer files between NCI servers and your local computer. You can do so via the command line with programs like rsync which is readily available. If you prefer a graphical interface, NCI advises you can install a program like [https://cyberduck.io/?l=en Cyberduck]. You can then use a SFTP connection. To access files on Gadi, it is preferable to use the data mover nodes: gadi-dm.nci.org.au for the server's name.</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Global_Atmosphere&diff=2700Global Atmosphere2021-05-10T00:37:04Z<p>S.wales: </p>
<hr />
<div>Global Atmosphere (GA) is the name of the UK Met Office's resolution independent global configuration of the [[Unified Model]]<br />
<br />
It is normally paired with a Global Land (GL) configuration of the [[JULES]] land-surface model<br />
<br />
It may also be coupled with the NEMO ocean and CICE sea ice models to form the Global Coupled (GC) coupled model (the basis of [[ACCESS-S]])<br />
<br />
Met Office documentation of the configuration can be found on the [https://code.metoffice.gov.uk/trac/gmed GMED (Global Model Development)] website (Requires a [[MOSRS|MOSRS account]])<br />
<br />
The configuration is versioned separately from the model code, so you can run a GA 7 job using UM versions from 10.3 to 11.4<br />
<br />
GA jobs are run using [[Rose-Cylc]]<br />
<br />
==== Experiments on Gadi ====<br />
<br />
These jobs are based off official Met Office releases, ported to NCI<br />
<br />
{| class="wikitable"<br />
|-<br />
! NCI Experiment Id !! Model Version !! Science Version!! Met Office Id !! Notes !! Performance<br />
|-<br />
| u-ce355 || 11.6 || [https://code.metoffice.gov.uk/trac/GA/wiki/GADocumentation/GAChanges/GA7.1 GA 7.1] || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bs606 u-bs606] || ||<br />
|-<br />
| u-bo915 || 11.4 || [https://code.metoffice.gov.uk/trac/GA/wiki/GADocumentation/GA7.0 GA 7.0] || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bk228 u-bk228] || || [[UM On Gadi#UM11.4]]<br />
|}<br />
<br />
<br />
[[Category:Unified Model]][[Category:Atmosphere Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Global_Atmosphere&diff=2699Global Atmosphere2021-05-10T00:35:42Z<p>S.wales: </p>
<hr />
<div>Global Atmosphere (GA) is the name of the UK Met Office's resolution independent global configuration of the [[Unified Model]]<br />
<br />
It is normally paired with a Global Land (GL) configuration of the [[JULES]] land-surface model<br />
<br />
It may also be coupled with the NEMO ocean and CICE sea ice models to form the Global Coupled (GC) coupled model (the basis of [[ACCESS-S]])<br />
<br />
Met Office documentation of the configuration can be found on the [https://code.metoffice.gov.uk/trac/gmed GMED (Global Model Development)] website (Requires a [[MOSRS|MOSRS account]])<br />
<br />
The configuration is versioned separately from the model code, so you can run a GA 7 job using UM versions from 10.3 to 11.4<br />
<br />
GA jobs are run using [[Rose-Cylc]]<br />
<br />
==== Experiments on Gadi ====<br />
<br />
These jobs are based off official Met Office releases, ported to NCI<br />
<br />
{| class="wikitable"<br />
|-<br />
! NCI Experiment Id !! Model Version !! Science Version!! Met Office Id !! Notes !! Performance<br />
|-<br />
| u-ce355 || 11.6 || GA 7.1 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bs606 u-bs606] || ||<br />
|-<br />
| u-bo915 || 11.4 || GA 7.0 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bk228 u-bk228] || || [[UM On Gadi#UM11.4]]<br />
|}<br />
<br />
<br />
[[Category:Unified Model]][[Category:Atmosphere Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Global_Atmosphere&diff=2698Global Atmosphere2021-05-10T00:34:58Z<p>S.wales: </p>
<hr />
<div>Global Atmosphere (GA) is the name of the UK Met Office's resolution independent global configuration of the [[Unified Model]]<br />
<br />
It is normally paired with a Global Land (GL) configuration of the [[JULES]] land-surface model<br />
<br />
It may also be coupled with the NEMO ocean and CICE sea ice models to form the Global Coupled (GC) coupled model (the basis of [[ACCESS-S]])<br />
<br />
Met Office documentation of the configuration can be found on the [https://code.metoffice.gov.uk/trac/gmed GMED (Global Model Development)] website (Requires a [[MOSRS|MOSRS account]])<br />
<br />
The configuration is versioned separately from the model code, so you can run a GA 7 job using UM versions from 10.3 to 11.4<br />
<br />
GA jobs are run using [[Rose-Cylc]]<br />
<br />
==== GA 7.0 ====<br />
<br />
These jobs are based off official Met Office releases, ported to NCI<br />
<br />
{| class="wikitable"<br />
|-<br />
! NCI Experiment Id !! Model Version !! Science Version!! Met Office Id !! Notes !! Performance<br />
|-<br />
| u-ce355 || 11.6 || GA 7.1 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bs606 u-bs606] || ||<br />
|-<br />
| u-bo915 || 11.4 || GA 7.0 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bk228 u-bk228] || || [[UM On Gadi#UM11.4]]<br />
|}<br />
<br />
<br />
[[Category:Unified Model]][[Category:Atmosphere Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Global_Atmosphere&diff=2697Global Atmosphere2021-05-10T00:34:27Z<p>S.wales: </p>
<hr />
<div>Global Atmosphere (GA) is the name of the UK Met Office's resolution independent global configuration of the [[Unified Model]]<br />
<br />
It is normally paired with a Global Land (GL) configuration of the [[JULES]] land-surface model<br />
<br />
It may also be coupled with the NEMO ocean and CICE sea ice models to form the Global Coupled (GC) coupled model (the basis of [[ACCESS-S]])<br />
<br />
Met Office documentation of the configuration can be found on the [https://code.metoffice.gov.uk/trac/gmed GMED (Global Model Development)] website (Requires a [[MOSRS|MOSRS account]])<br />
<br />
The configuration is versioned separately from the model code, so you can run a GA 7 job using UM versions from 10.3 to 11.4<br />
<br />
GA jobs are run using [[Rose-Cylc]]<br />
<br />
==== GA 7.0 ====<br />
<br />
These jobs are based off official Met Office releases, ported to NCI<br />
<br />
{| class="wikitable"<br />
|-<br />
! NCI Experiment Id !! Model Version !! Met Office Id !! Notes !! Performance<br />
|-<br />
| u-ce355 || 11.6 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bs606 u-bs606] || ||<br />
|-<br />
| u-bo915 || 11.4 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bk228 u-bk228] || || [[UM On Gadi#UM11.4]]<br />
|}<br />
<br />
<br />
[[Category:Unified Model]][[Category:Atmosphere Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Global_Atmosphere&diff=2696Global Atmosphere2021-05-10T00:33:33Z<p>S.wales: </p>
<hr />
<div>Global Atmosphere (GA) is the name of the UK Met Office's resolution independent global configuration of the [[Unified Model]]<br />
<br />
It is normally paired with a Global Land (GL) configuration of the [[JULES]] land-surface model<br />
<br />
It may also be coupled with the NEMO ocean and CICE sea ice models to form the Global Coupled (GC) coupled model (the basis of [[ACCESS-S]])<br />
<br />
Met Office documentation of the configuration can be found on the [https://code.metoffice.gov.uk/trac/gmed GMED (Global Model Development)] website (Requires a [[MOSRS|MOSRS account]])<br />
<br />
The configuration is versioned separately from the model code, so you can run a GA 7 job using UM versions from 10.3 to 11.4<br />
<br />
GA jobs are run using [[Rose-Cylc]]<br />
<br />
==== GA 7.0 ====<br />
<br />
These jobs are based off official Met Office releases, ported to NCI<br />
<br />
{| class="wikitable"<br />
|-<br />
! Experiment Id !! Model Version !! Met Office Id !! Notes !! Performance<br />
|-<br />
| u-ce355 || 11.6 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bs606 u-bs606] || ||<br />
| u-bo915 || 11.4 || [https://code.metoffice.gov.uk/trac/GA/wiki/GAJobs/u-bk228 u-bk228] || || [[UM On Gadi#UM11.4]]<br />
|}<br />
<br />
<br />
[[Category:Unified Model]][[Category:Atmosphere Models]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Unified_Model&diff=2684Unified Model2021-04-30T03:34:53Z<p>S.wales: /* Getting Started */</p>
<hr />
<div><br />
The Unified Model is the atmospheric component of [[ACCESS|ACCESS]], developed by the UK's Met Office. The Centre of Excellence sublicences the UM from CAWCR, our licence allows us to run the UM at NCI only. To run the UM you will first need an NCI account, then ask for connection to the ACCESS group through [https://my.nci.org.au/mancini/project/access/join NCI's account website].<br />
<br />
The ACCESS climate models are based on version 7.3 of the UM, most Australian university research uses this version. The Bureau makes use of the more recent version 7.8 for forecasting, and is evaluating moving to version 8.2. These are the primary versions supported at NCI, other versions are available but may not be quite as well tested. Different versions include changed science configurations, for details see the [https://code.metoffice.gov.uk/trac/um/wiki/ReleaseNotes release notes].<br />
<br />
= Getting Started =<br />
<br />
The UM is configured through a graphical interface, which is run from the server [[Accessdev]]. Follow the [[Accessdev#Getting_Set_Up|Set Up Instructions]] for the version of the model you want to run.<br />
<br />
There are two ways to run the model. Newer versions (post UM 10) use Rose & Cylc, older versions (pre UM 10) use a program called UMUI.<br />
<br />
The Met Office have a tutorial for the model at https://code.metoffice.gov.uk/doc/um/latest/<br />
<br />
A variety of model configurations are within the Centre. Feel free to contact CMS to discuss the different model types and what might be most appropriate for your work. If you plan on contributing work to the Met Office make sure you use the most recent version of the model.<br />
<br />
= UM 10 / Global Atmosphere / ACCESS 2.x =<br />
<br />
The current version of the Unified Model is available from the Met Office's repository site [https://code.metoffice.gov.uk https://code.metoffice.gov.uk].<br />
<br />
ARCCSS researchers need to email [mailto:climate_help@nci.org.au climate_help@nci.org.au] to request an account on the Met Office server.<br />
<br />
=== Links & More Information ===<br />
<br />
*[https://accessdev.nci.org.au/trac/wiki/GettingConnected Connecting to Accessdev and running Rose] <br />
*[[Running_the_UM_with_Rose|Tips on using Rose with the UM]] <br />
* [http://nespclimate.com.au/wp-content/uploads/2020/10/Instruction-document-Getting_started_with_ACCESS.pdf Getting Started with ACCESS]<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM 10.X Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2017 UM User Tutorial 2017] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2016 UM User Tutorial 2016] <br />
<br />
*[[GA7_Performance|GA7 Performance]]<br />
<br />
= Pre-UM 10 =<br />
<br />
See [[Unified Model pre10]] for documentation on older versions of the model<br />
<br />
= Experiments =<br />
<br />
A full list of experiments we support or have helped set up can be found at [[UM_Experiments|UM Experiments]]<br />
<br />
These configurations are currently our primary support targets:<br />
<br />
*[[ACCESS_2.0|ACCESS 2.0]] <br />
*[[ACCESS-ESM|ACCESS-ESM]] <br />
*[[Global Atmosphere|GA 7]] <br />
*[[U-aa753_Nested_Suite|Nesting Suite]]<br />
<br />
= Guides =<br />
<br />
Assorted information on running the UM and available job configurations is listed in the sections below. Feel free to email [mailto:cws_help@nci.org.au climate_help@nci.org.au] if you have any difficulties using the UM or suggestions for how we can make using the UM simpler.<br />
<br />
*[[Getting_Help_with_the_UM|Getting Help with the UM]] <br />
*[[Um-errors|Common Errors]] <br />
<br />
== Starting Up ==<br />
<br />
*[[Running_the_UM_with_Rose|Running the UM with Rose]]<br />
*[[UM_Helper_Scripts|UM Helper Scripts]]<br />
<br />
== Run Types ==<br />
<br />
*[[AMIP|AMIP]] <br />
*[[ACCESS|CMIP]] <br />
*[https://trac.nci.org.au/trac/access/wiki/SCM_testcases Single Column] <br />
*[[KPP_Coupling|KPP Coupling]] <br />
*[https://trac.nci.org.au/trac/access/wiki/APS1_ACCESS_C ACCESS-C High-resolution NWP] <br />
*[[UM_Nested_Models|Nested / Limited Area Models]] <br />
*[[UKCA|UKCA Chemistry]] <br />
<br />
*[http://www.ukca.ac.uk/wiki/index.php/UKCA_&_UMUI_Tutorials UKCA Tutorials] <br />
*[https://trac.nci.org.au/trac/access/wiki/AccessModelExperimentLibrary CAWCR's ACCESS experiment library] (log in with your NCI account)<br />
<br />
= Common Errors =<br />
<br />
If a run fails, search for 'UM ERROR' in the .leave file to see what went wrong. Common issues are listed here<br />
<br />
*[[Um-errors#pp-headers|REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH]] <br />
*[https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints Known UM failures at UKMO] <br />
<br />
== Job Input ==<br />
<br />
*[[UM_Ancillary_files|UM Ancillary files]] <br />
*[[Modifying_UM_Files|Modifying UM files with Mule]] <br />
*[[SCM_with_ERA-Interim|SCM with ERA-Interim]] <br />
<br />
== Job Output ==<br />
<br />
*[[Using_STASH|Using STASH]] <br />
*[[UM_Filenames|UM output filename convention]] <br />
*[[Analysing_UM_outputs|Analysing UM outputs & Converting to NetCDF]] <br />
*[[Archiving_Output|Archiving Output]] <br />
*[[Comparing_UM_Files|Comparing UM Files]] <br />
*[https://code.metoffice.gov.uk/trac/UKESM/wiki/CMIP6/DiagnosticMappings Mapping UM outputs to CMIP fields] <br />
<br />
== Modifying the UM ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/working_practices Met Office UM Development Working Practices] <br />
*[[Adding New Fields to STASH]] <br />
*[[Profiling_with_IPM_and_Vampir|Profiling MPI]] <br />
<br />
== Informative ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/PastReleases UM Version Summaries] <br />
*[[Downloading_from_MASS|Downloading from MASS]] <br />
*[[Vertical_Coordinates|Vertical Coordinates]]<br />
<br />
== [[UM_and_ACCESS_FAQ|UM and ACCESS FAQs]] ==<br />
<br />
*[[UM_and_ACCESS_FAQ#Who_can_gain_access_to_the_UM_and_the_ACCESS_models.3F|Who can gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_gain_access_to_the_UM_and_the_ACCESS_models.3F|What to do to gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_use_the_software_in_.2Fprojects.2Faccess.2Fbin.3F|What to do to use the software in /projects/access/bin?]]<br />
<br />
= Resources =<br />
<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://cms.ncas.ac.uk/ NCAS] <br />
<br />
*[[UM_Admin|UM Admin]] <br />
*[[UM_Partial_guides|UM Partial guides]] <br />
<br />
[[Category:Unified Model]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Unified_Model&diff=2683Unified Model2021-04-30T03:28:50Z<p>S.wales: /* Getting Started */</p>
<hr />
<div><br />
The Unified Model is the atmospheric component of [[ACCESS|ACCESS]], developed by the UK's Met Office. The Centre of Excellence sublicences the UM from CAWCR, our licence allows us to run the UM at NCI only. To run the UM you will first need an NCI account, then ask for connection to the ACCESS group through [https://my.nci.org.au/mancini/project/access/join NCI's account website].<br />
<br />
The ACCESS climate models are based on version 7.3 of the UM, most Australian university research uses this version. The Bureau makes use of the more recent version 7.8 for forecasting, and is evaluating moving to version 8.2. These are the primary versions supported at NCI, other versions are available but may not be quite as well tested. Different versions include changed science configurations, for details see the [https://code.metoffice.gov.uk/trac/um/wiki/ReleaseNotes release notes].<br />
<br />
= Getting Started =<br />
<br />
The UM is configured through a graphical interface, which is run from the server [[Accessdev]]. Follow the [[Accessdev#Getting_Set_Up|Set Up Instructions]] for the version of the model you want to run.<br />
<br />
There are two ways to run the model. Newer versions (post UM 10) use Rose & Cylc, older versions (pre UM 10) use a program called UMUI.<br />
<br />
The Met Office have a tutorial for the model at https://code.metoffice.gov.uk/doc/um/latest/<br />
<br />
= UM 10 / Global Atmosphere / ACCESS 2.x =<br />
<br />
The current version of the Unified Model is available from the Met Office's repository site [https://code.metoffice.gov.uk https://code.metoffice.gov.uk].<br />
<br />
ARCCSS researchers need to email [mailto:climate_help@nci.org.au climate_help@nci.org.au] to request an account on the Met Office server.<br />
<br />
=== Links & More Information ===<br />
<br />
*[https://accessdev.nci.org.au/trac/wiki/GettingConnected Connecting to Accessdev and running Rose] <br />
*[[Running_the_UM_with_Rose|Tips on using Rose with the UM]] <br />
* [http://nespclimate.com.au/wp-content/uploads/2020/10/Instruction-document-Getting_started_with_ACCESS.pdf Getting Started with ACCESS]<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM 10.X Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2017 UM User Tutorial 2017] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2016 UM User Tutorial 2016] <br />
<br />
*[[GA7_Performance|GA7 Performance]]<br />
<br />
= Pre-UM 10 =<br />
<br />
See [[Unified Model pre10]] for documentation on older versions of the model<br />
<br />
= Experiments =<br />
<br />
A full list of experiments we support or have helped set up can be found at [[UM_Experiments|UM Experiments]]<br />
<br />
These configurations are currently our primary support targets:<br />
<br />
*[[ACCESS_2.0|ACCESS 2.0]] <br />
*[[ACCESS-ESM|ACCESS-ESM]] <br />
*[[Global Atmosphere|GA 7]] <br />
*[[U-aa753_Nested_Suite|Nesting Suite]]<br />
<br />
= Guides =<br />
<br />
Assorted information on running the UM and available job configurations is listed in the sections below. Feel free to email [mailto:cws_help@nci.org.au climate_help@nci.org.au] if you have any difficulties using the UM or suggestions for how we can make using the UM simpler.<br />
<br />
*[[Getting_Help_with_the_UM|Getting Help with the UM]] <br />
*[[Um-errors|Common Errors]] <br />
<br />
== Starting Up ==<br />
<br />
*[[Running_the_UM_with_Rose|Running the UM with Rose]]<br />
*[[UM_Helper_Scripts|UM Helper Scripts]]<br />
<br />
== Run Types ==<br />
<br />
*[[AMIP|AMIP]] <br />
*[[ACCESS|CMIP]] <br />
*[https://trac.nci.org.au/trac/access/wiki/SCM_testcases Single Column] <br />
*[[KPP_Coupling|KPP Coupling]] <br />
*[https://trac.nci.org.au/trac/access/wiki/APS1_ACCESS_C ACCESS-C High-resolution NWP] <br />
*[[UM_Nested_Models|Nested / Limited Area Models]] <br />
*[[UKCA|UKCA Chemistry]] <br />
<br />
*[http://www.ukca.ac.uk/wiki/index.php/UKCA_&_UMUI_Tutorials UKCA Tutorials] <br />
*[https://trac.nci.org.au/trac/access/wiki/AccessModelExperimentLibrary CAWCR's ACCESS experiment library] (log in with your NCI account)<br />
<br />
= Common Errors =<br />
<br />
If a run fails, search for 'UM ERROR' in the .leave file to see what went wrong. Common issues are listed here<br />
<br />
*[[Um-errors#pp-headers|REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH]] <br />
*[https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints Known UM failures at UKMO] <br />
<br />
== Job Input ==<br />
<br />
*[[UM_Ancillary_files|UM Ancillary files]] <br />
*[[Modifying_UM_Files|Modifying UM files with Mule]] <br />
*[[SCM_with_ERA-Interim|SCM with ERA-Interim]] <br />
<br />
== Job Output ==<br />
<br />
*[[Using_STASH|Using STASH]] <br />
*[[UM_Filenames|UM output filename convention]] <br />
*[[Analysing_UM_outputs|Analysing UM outputs & Converting to NetCDF]] <br />
*[[Archiving_Output|Archiving Output]] <br />
*[[Comparing_UM_Files|Comparing UM Files]] <br />
*[https://code.metoffice.gov.uk/trac/UKESM/wiki/CMIP6/DiagnosticMappings Mapping UM outputs to CMIP fields] <br />
<br />
== Modifying the UM ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/working_practices Met Office UM Development Working Practices] <br />
*[[Adding New Fields to STASH]] <br />
*[[Profiling_with_IPM_and_Vampir|Profiling MPI]] <br />
<br />
== Informative ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/PastReleases UM Version Summaries] <br />
*[[Downloading_from_MASS|Downloading from MASS]] <br />
*[[Vertical_Coordinates|Vertical Coordinates]]<br />
<br />
== [[UM_and_ACCESS_FAQ|UM and ACCESS FAQs]] ==<br />
<br />
*[[UM_and_ACCESS_FAQ#Who_can_gain_access_to_the_UM_and_the_ACCESS_models.3F|Who can gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_gain_access_to_the_UM_and_the_ACCESS_models.3F|What to do to gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_use_the_software_in_.2Fprojects.2Faccess.2Fbin.3F|What to do to use the software in /projects/access/bin?]]<br />
<br />
= Resources =<br />
<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://cms.ncas.ac.uk/ NCAS] <br />
<br />
*[[UM_Admin|UM Admin]] <br />
*[[UM_Partial_guides|UM Partial guides]] <br />
<br />
[[Category:Unified Model]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Unified_Model&diff=2682Unified Model2021-04-30T03:28:02Z<p>S.wales: /* UM 10 / Global Atmosphere / ACCESS 2.x */</p>
<hr />
<div><br />
The Unified Model is the atmospheric component of [[ACCESS|ACCESS]], developed by the UK's Met Office. The Centre of Excellence sublicences the UM from CAWCR, our licence allows us to run the UM at NCI only. To run the UM you will first need an NCI account, then ask for connection to the ACCESS group through [https://my.nci.org.au/mancini/project/access/join NCI's account website].<br />
<br />
The ACCESS climate models are based on version 7.3 of the UM, most Australian university research uses this version. The Bureau makes use of the more recent version 7.8 for forecasting, and is evaluating moving to version 8.2. These are the primary versions supported at NCI, other versions are available but may not be quite as well tested. Different versions include changed science configurations, for details see the [https://code.metoffice.gov.uk/trac/um/wiki/ReleaseNotes release notes].<br />
<br />
= Getting Started =<br />
<br />
The UM is configured through a graphical interface, which is run from the server [[Accessdev]]. Follow the [[Accessdev#Getting_Set_Up|Set Up Instructions]] for the version of the model you want to run.<br />
<br />
There are two ways to run the model. Newer versions (post UM 10) use [[Rose_&_Cylc|Rose & Cylc]], older versions (pre UM 10) use a program called [[UMUI|UMUI]].<br />
<br />
The Met Office have a tutorial for the model at https://code.metoffice.gov.uk/doc/um/latest/<br />
<br />
= UM 10 / Global Atmosphere / ACCESS 2.x =<br />
<br />
The current version of the Unified Model is available from the Met Office's repository site [https://code.metoffice.gov.uk https://code.metoffice.gov.uk].<br />
<br />
ARCCSS researchers need to email [mailto:climate_help@nci.org.au climate_help@nci.org.au] to request an account on the Met Office server.<br />
<br />
=== Links & More Information ===<br />
<br />
*[https://accessdev.nci.org.au/trac/wiki/GettingConnected Connecting to Accessdev and running Rose] <br />
*[[Running_the_UM_with_Rose|Tips on using Rose with the UM]] <br />
* [http://nespclimate.com.au/wp-content/uploads/2020/10/Instruction-document-Getting_started_with_ACCESS.pdf Getting Started with ACCESS]<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM 10.X Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2017 UM User Tutorial 2017] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2016 UM User Tutorial 2016] <br />
<br />
*[[GA7_Performance|GA7 Performance]]<br />
<br />
= Pre-UM 10 =<br />
<br />
See [[Unified Model pre10]] for documentation on older versions of the model<br />
<br />
= Experiments =<br />
<br />
A full list of experiments we support or have helped set up can be found at [[UM_Experiments|UM Experiments]]<br />
<br />
These configurations are currently our primary support targets:<br />
<br />
*[[ACCESS_2.0|ACCESS 2.0]] <br />
*[[ACCESS-ESM|ACCESS-ESM]] <br />
*[[Global Atmosphere|GA 7]] <br />
*[[U-aa753_Nested_Suite|Nesting Suite]]<br />
<br />
= Guides =<br />
<br />
Assorted information on running the UM and available job configurations is listed in the sections below. Feel free to email [mailto:cws_help@nci.org.au climate_help@nci.org.au] if you have any difficulties using the UM or suggestions for how we can make using the UM simpler.<br />
<br />
*[[Getting_Help_with_the_UM|Getting Help with the UM]] <br />
*[[Um-errors|Common Errors]] <br />
<br />
== Starting Up ==<br />
<br />
*[[Running_the_UM_with_Rose|Running the UM with Rose]]<br />
*[[UM_Helper_Scripts|UM Helper Scripts]]<br />
<br />
== Run Types ==<br />
<br />
*[[AMIP|AMIP]] <br />
*[[ACCESS|CMIP]] <br />
*[https://trac.nci.org.au/trac/access/wiki/SCM_testcases Single Column] <br />
*[[KPP_Coupling|KPP Coupling]] <br />
*[https://trac.nci.org.au/trac/access/wiki/APS1_ACCESS_C ACCESS-C High-resolution NWP] <br />
*[[UM_Nested_Models|Nested / Limited Area Models]] <br />
*[[UKCA|UKCA Chemistry]] <br />
<br />
*[http://www.ukca.ac.uk/wiki/index.php/UKCA_&_UMUI_Tutorials UKCA Tutorials] <br />
*[https://trac.nci.org.au/trac/access/wiki/AccessModelExperimentLibrary CAWCR's ACCESS experiment library] (log in with your NCI account)<br />
<br />
= Common Errors =<br />
<br />
If a run fails, search for 'UM ERROR' in the .leave file to see what went wrong. Common issues are listed here<br />
<br />
*[[Um-errors#pp-headers|REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH]] <br />
*[https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints Known UM failures at UKMO] <br />
<br />
== Job Input ==<br />
<br />
*[[UM_Ancillary_files|UM Ancillary files]] <br />
*[[Modifying_UM_Files|Modifying UM files with Mule]] <br />
*[[SCM_with_ERA-Interim|SCM with ERA-Interim]] <br />
<br />
== Job Output ==<br />
<br />
*[[Using_STASH|Using STASH]] <br />
*[[UM_Filenames|UM output filename convention]] <br />
*[[Analysing_UM_outputs|Analysing UM outputs & Converting to NetCDF]] <br />
*[[Archiving_Output|Archiving Output]] <br />
*[[Comparing_UM_Files|Comparing UM Files]] <br />
*[https://code.metoffice.gov.uk/trac/UKESM/wiki/CMIP6/DiagnosticMappings Mapping UM outputs to CMIP fields] <br />
<br />
== Modifying the UM ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/working_practices Met Office UM Development Working Practices] <br />
*[[Adding New Fields to STASH]] <br />
*[[Profiling_with_IPM_and_Vampir|Profiling MPI]] <br />
<br />
== Informative ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/PastReleases UM Version Summaries] <br />
*[[Downloading_from_MASS|Downloading from MASS]] <br />
*[[Vertical_Coordinates|Vertical Coordinates]]<br />
<br />
== [[UM_and_ACCESS_FAQ|UM and ACCESS FAQs]] ==<br />
<br />
*[[UM_and_ACCESS_FAQ#Who_can_gain_access_to_the_UM_and_the_ACCESS_models.3F|Who can gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_gain_access_to_the_UM_and_the_ACCESS_models.3F|What to do to gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_use_the_software_in_.2Fprojects.2Faccess.2Fbin.3F|What to do to use the software in /projects/access/bin?]]<br />
<br />
= Resources =<br />
<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://cms.ncas.ac.uk/ NCAS] <br />
<br />
*[[UM_Admin|UM Admin]] <br />
*[[UM_Partial_guides|UM Partial guides]] <br />
<br />
[[Category:Unified Model]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Unified_Model_pre10&diff=2681Unified Model pre102021-04-30T03:25:46Z<p>S.wales: Created page with "Guides for running the Unified Model before version 10, when Rose and Cylc were introduced. See Unified Model for guides post version 10, many topics not related to the U..."</p>
<hr />
<div>Guides for running the Unified Model before version 10, when Rose and Cylc were introduced.<br />
<br />
See [[Unified Model]] for guides post version 10, many topics not related to the UI will be similar<br />
<br />
= UM 7.3 / ACCESS 1.x =<br />
<br />
The ACCESS climate model as used in CMIP5 uses model version 7.3. This version of the model uses the older `UMUI` user interface<br />
<br />
===Links & More Information ===<br />
<br />
*[[UM_Environment|Setting up the UMUI environment]] <br />
*[[Introduction_to_UMUI|Introduction to UMUI]] <br />
*[https://accessdev.nci.org.au/umdocs/7.3 UM 7.3 Documentation] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2013 UM User Tutorial 2013] <br />
<br />
== Starting Up ==<br />
<br />
*[[Building_the_UM|Building the UM]] <br />
*[[Standard_UM_Runs|Standard UM Runs]] <br />
*[[Restarting_UM_Runs|Running the UM using automatic resubmission]] <br />
*[[Recovering_from_an_interrupted_job|Recovering from an interrupted job]] <br />
*[[UM_Helper_Scripts|UM Helper Scripts]]<br />
<br />
== Job Input ==<br />
<br />
*[[UM_Ancillary_files|UM Ancillary files]] <br />
*[[Concatenate_AMIP_Ancillaries|Concatenate AMIP Ancillaries]] <br />
*[[Copying_CO2_values|Copying CO2 values]] <br />
*[[Modifying_UM_Files|Modifying UM files with Mule]] <br />
*[[SCM_with_ERA-Interim|SCM with ERA-Interim]] <br />
<br />
<br />
== Job Output ==<br />
<br />
*[[Using_STASH|Using STASH]] <br />
*[[Copying_STASH_settings_between_jobs|Copying STASH settings between jobs]] <br />
*[[UM_Filenames|UM output filename convention]] <br />
*[[Analysing_UM_outputs|Analysing UM outputs & Converting to NetCDF]] <br />
*[[Archiving_Output|Archiving Output]] <br />
*[[Comparing_UM_Files|Comparing UM Files]] <br />
*[https://code.metoffice.gov.uk/trac/UKESM/wiki/CMIP6/DiagnosticMappings Mapping UM outputs to CMIP fields] <br />
<br />
<br />
== Informative ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/PastReleases UM Version Summaries] <br />
*[[UM_Run_Scripts|UM Run Scripts]] <br />
*[[UMUI_over_VNC|UMUI over VNC]] <br />
*[[Downloading_from_MASS|Downloading from MASS]] <br />
*[[Vertical_Coordinates|Vertical Coordinates]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Unified_Model&diff=2680Unified Model2021-04-30T03:25:38Z<p>S.wales: </p>
<hr />
<div><br />
The Unified Model is the atmospheric component of [[ACCESS|ACCESS]], developed by the UK's Met Office. The Centre of Excellence sublicences the UM from CAWCR, our licence allows us to run the UM at NCI only. To run the UM you will first need an NCI account, then ask for connection to the ACCESS group through [https://my.nci.org.au/mancini/project/access/join NCI's account website].<br />
<br />
The ACCESS climate models are based on version 7.3 of the UM, most Australian university research uses this version. The Bureau makes use of the more recent version 7.8 for forecasting, and is evaluating moving to version 8.2. These are the primary versions supported at NCI, other versions are available but may not be quite as well tested. Different versions include changed science configurations, for details see the [https://code.metoffice.gov.uk/trac/um/wiki/ReleaseNotes release notes].<br />
<br />
= Getting Started =<br />
<br />
The UM is configured through a graphical interface, which is run from the server [[Accessdev]]. Follow the [[Accessdev#Getting_Set_Up|Set Up Instructions]] for the version of the model you want to run.<br />
<br />
There are two ways to run the model. Newer versions (post UM 10) use [[Rose_&_Cylc|Rose & Cylc]], older versions (pre UM 10) use a program called [[UMUI|UMUI]].<br />
<br />
The Met Office have a tutorial for the model at https://code.metoffice.gov.uk/doc/um/latest/<br />
<br />
= UM 10 / Global Atmosphere / ACCESS 2.x =<br />
<br />
The current version of the Unified Model is available from the Met Office's repository site [https://code.metoffice.gov.uk https://code.metoffice.gov.uk].<br />
<br />
ARCCSS researchers need to email [mailto:climate_help@nci.org.au climate_help@nci.org.au] to request an account on the Met Office server.<br />
<br />
=== Links & More Information ===<br />
<br />
*[https://accessdev.nci.org.au/trac/wiki/GettingConnected Connecting to Accessdev and running Rose] <br />
*[[Running_the_UM_with_Rose|Tips on using Rose with the UM]] <br />
* [http://nespclimate.com.au/wp-content/uploads/2020/10/Instruction-document-Getting_started_with_ACCESS.pdf Getting Started with ACCESS]<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM 10.X Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2017 UM User Tutorial 2017] <br />
*[http://collab.metoffice.gov.uk/twiki/bin/view/Support/UMUserTutorial2016 UM User Tutorial 2016] <br />
<br />
*[[GA7_Performance|GA7 Performance]]<br />
<br />
= Experiments =<br />
<br />
A full list of experiments we support or have helped set up can be found at [[UM_Experiments|UM Experiments]]<br />
<br />
These configurations are currently our primary support targets:<br />
<br />
*[[ACCESS_2.0|ACCESS 2.0]] <br />
*[[ACCESS-ESM|ACCESS-ESM]] <br />
*[[Global Atmosphere|GA 7]] <br />
*[[U-aa753_Nested_Suite|Nesting Suite]]<br />
<br />
= Guides =<br />
<br />
Assorted information on running the UM and available job configurations is listed in the sections below. Feel free to email [mailto:cws_help@nci.org.au climate_help@nci.org.au] if you have any difficulties using the UM or suggestions for how we can make using the UM simpler.<br />
<br />
*[[Getting_Help_with_the_UM|Getting Help with the UM]] <br />
*[[Um-errors|Common Errors]] <br />
<br />
== Starting Up ==<br />
<br />
*[[Running_the_UM_with_Rose|Running the UM with Rose]]<br />
*[[UM_Helper_Scripts|UM Helper Scripts]]<br />
<br />
== Run Types ==<br />
<br />
*[[AMIP|AMIP]] <br />
*[[ACCESS|CMIP]] <br />
*[https://trac.nci.org.au/trac/access/wiki/SCM_testcases Single Column] <br />
*[[KPP_Coupling|KPP Coupling]] <br />
*[https://trac.nci.org.au/trac/access/wiki/APS1_ACCESS_C ACCESS-C High-resolution NWP] <br />
*[[UM_Nested_Models|Nested / Limited Area Models]] <br />
*[[UKCA|UKCA Chemistry]] <br />
<br />
*[http://www.ukca.ac.uk/wiki/index.php/UKCA_&_UMUI_Tutorials UKCA Tutorials] <br />
*[https://trac.nci.org.au/trac/access/wiki/AccessModelExperimentLibrary CAWCR's ACCESS experiment library] (log in with your NCI account)<br />
<br />
= Common Errors =<br />
<br />
If a run fails, search for 'UM ERROR' in the .leave file to see what went wrong. Common issues are listed here<br />
<br />
*[[Um-errors#pp-headers|REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH]] <br />
*[https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints Known UM failures at UKMO] <br />
<br />
== Job Input ==<br />
<br />
*[[UM_Ancillary_files|UM Ancillary files]] <br />
*[[Modifying_UM_Files|Modifying UM files with Mule]] <br />
*[[SCM_with_ERA-Interim|SCM with ERA-Interim]] <br />
<br />
== Job Output ==<br />
<br />
*[[Using_STASH|Using STASH]] <br />
*[[UM_Filenames|UM output filename convention]] <br />
*[[Analysing_UM_outputs|Analysing UM outputs & Converting to NetCDF]] <br />
*[[Archiving_Output|Archiving Output]] <br />
*[[Comparing_UM_Files|Comparing UM Files]] <br />
*[https://code.metoffice.gov.uk/trac/UKESM/wiki/CMIP6/DiagnosticMappings Mapping UM outputs to CMIP fields] <br />
<br />
== Modifying the UM ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/working_practices Met Office UM Development Working Practices] <br />
*[[Adding New Fields to STASH]] <br />
*[[Profiling_with_IPM_and_Vampir|Profiling MPI]] <br />
<br />
== Informative ==<br />
<br />
*[https://code.metoffice.gov.uk/trac/um/wiki/PastReleases UM Version Summaries] <br />
*[[Downloading_from_MASS|Downloading from MASS]] <br />
*[[Vertical_Coordinates|Vertical Coordinates]]<br />
<br />
== [[UM_and_ACCESS_FAQ|UM and ACCESS FAQs]] ==<br />
<br />
*[[UM_and_ACCESS_FAQ#Who_can_gain_access_to_the_UM_and_the_ACCESS_models.3F|Who can gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_gain_access_to_the_UM_and_the_ACCESS_models.3F|What to do to gain access to the UM and the ACCESS models?]]<br />
*[[UM_and_ACCESS_FAQ#What_to_do_to_use_the_software_in_.2Fprojects.2Faccess.2Fbin.3F|What to do to use the software in /projects/access/bin?]]<br />
<br />
= Resources =<br />
<br />
*[https://code.metoffice.gov.uk/trac/um UM code & issue tracker] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/umdp.html UM Documentation] <br />
*[https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Rose Interface Tutorial] <br />
*[http://cms.ncas.ac.uk/ NCAS] <br />
<br />
*[[UM_Admin|UM Admin]] <br />
*[[UM_Partial_guides|UM Partial guides]] <br />
<br />
[[Category:Unified Model]]</div>S.waleshttp://climate-cms.wikis.unsw.edu.au/index.php?title=Running_the_UM_with_Rose&diff=2679Running the UM with Rose2021-04-30T03:14:44Z<p>S.wales: /* Resources */</p>
<hr />
<div>For a general overview of how to use the UM with Rose and Cylc see the [https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Met Office's UM tutorial]<br />
<br />
== Restarting and Extending Rose Suites ==<br />
<br />
To restart a stopped job run<br />
<syntaxhighlight lang=text><br />
rose suite-run --restart<br />
</syntaxhighlight><br />
This will put the suite back into the exact same state it was when Cylc stopped - failed tasks will still be failed, and if it reached the end of the run Cylc will promptly stop running again.<br />
<br />
Resubmit a failed task by right clicking on it and selecting 'Trigger Task'<br />
<br />
To extend the run dates you'll need to change the end time in the Rose editor, and then reload the configuration.<br />
<br />
If the configuration has changed (say you have edited the suite end date to make it run for another year) you need to reload the configuration when you restart it, which you can do with<br />
<syntaxhighlight lang=text><br />
rose suite-run --reload --restart<br />
</syntaxhighlight><br />
<br />
=== Resubmitting Tasks ===<br />
<br />
If a UM task has failed (i.e. it has a red box in the Cylc GUI) you can re-submit it by right clicking the task and selecting 'Trigger (run now)'<br />
<br />
The task will continue from the most recent restart file, provided that it is not the very first task. Resubmitting the first UM task will restart the run from the beginning.<br />
<br />
==Copying STASH fields== <br />
<br />
A pair of macros are provided with UM suites for copying STASH settings between jobs, named STASHExport and STASHImport. You can find the macros in the editor under <span style="font-family:monospace">Metadata -> um</span><br />
<br />
[[File:stashexport.png|800x558px]]<br />
<br />
The exported STASH configuration will be saved into 'app/um/STASHexport.ini'. To import the settings into a new job copy this file to the new suite's 'app/um/STASHImport.ini' and run the 'STASHImport' macro in the Rose editor on that suite.<br />
<br />
==Porting suites to NCI== <br />
<br />
See [https://accessdev.nci.org.au/trac/wiki/gadi#ConfiguringACCESSJobsforGadi the ACCESS wiki] for Rose/Cylc configuration settings on Gadi<br />
<br />
==Resources== <br />
<br />
* [https://accessdev.nci.org.au/trac/wiki/GettingConnected Getting Connected to Accessdev]<br />
* [https://code.metoffice.gov.uk/doc/um/latest/um-training/index.html Unified Model Rose Tutorial]<br />
* [https://metomi.github.io/rose/doc/rose.html Rose Documentation]<br />
* [https://cylc.github.io/cylc/html/single/cug-html.html Cylc Documentation]<br />
<br />
* [https://github.com/metomi/rose Rose on Github]<br />
* [https://github.com/cylc/cylc Cylc on Github]<br />
[[Category:Unified Model]][[Category:Rose]]</div>S.wales