Conda

Revision as of 20:53, 4 November 2020 by A.heerdegen (talk | contribs) (Add storage flag reminder)

Conda Python Environments

CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.

You can find the most recent list of libraries at our github repository, or run conda list with an environment loaded.

To use any of the conda environments:

  1. Request access to hh5 (to do once)
  2. You must first run (to do at each session)

module use /g/data3/hh5/public/modules

You can safely put this in your ~/.profile or ~/.login file

If you need to use the conda environment in a PBS job you will need to add the hh5 project to your storage flags, e.g.

#PBS -l storage=gdata/hh5

Stable Environment

We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Raijin. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.

module load conda/analysis3

Unstable Environment

The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.

module load conda/analysis3-unstable

When we do our quarterly update the unstable environment becomes the new stable environment.

Creating personal environments

You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file ~/.condarc like:

 

auto_activate_base: false
envs_dirs:
  - /scratch/$PROJECT/$USER/conda/envs
  - /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
  - /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
  root-dir: /scratch/$PROJECT/$USER/conda/bld

This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.

Interactive Analysis / Jupyter

For interactive analysis we encourage making use of NCI's VDI system, which allows you to run Jupyter notebooks without waiting in the queue

Requesting new packages

You can ask for a new package to be installed or for an existing package to be updated by emailing cws_help@nci.org.au.

As a general rule we will only install packages from the 'conda-forge' channel. Newly installed packages will be available in the conda/analysis3-unstable environment.

Update History

20.10 (Current Unstable)

Python has been updated to 3.8 (changes)

There are a small number of packages in analysis3-20.07 that are not compatible with 3.8, these have been disabled until we can get them working

  • pynio
  • cf_units
  • pymunge
  • ants

Notable new packages

  • rioxarray geospatial xarray extension powered by rasterio
  • cupy CUDA accellerated numpy

20.07 (Current Stable)

Notable New Packages

  • xesmf - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
  • sharppy - sounding and holograph analysis
  • earthpy - spatial raster and vector tools (e.g. rasterise shapefiles)
  • descartes - plot shapefiles
  • era5grib - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
  • xmitgcm - read mitgcm binary output

20.04 (Unsupported)

Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes

Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI

Notable New Packages

  • geopy: Locate lat/lon coordinates of places
  • ninja: mom6 build system

20.01 (Unsupported)

Python has been updated to 3.7

Notable New Packages

  • xlrd: Read excel files
  • ants: Unified Model Ancillary tools
  • climtas: Dask-aware Xarray timeseries processing

19.10 (Unsupported)

basemap has been removed as it is no longer supported and caused conflicts with other packages

Notable New Packages

19.07 (Unsupported)

Notable New Packages

  • pyferret
  • pyngl
  • pynio
  • xgcm
  • xrft

19.04 (unsupported)

Notable Changes

  • arccssive has been renamed to clef
  • A bug preventing Iris from opening UM files without a date has been fixed

Notable New Packages

  • bottleneck Fast rolling operations
  • cfgrib CF metadata for GRIB files
  • cfunits Convert between CF units
  • h5netcdf Pythonic interface to netCDF4 via h5py
  • intake Lightweight data catalogues
  • nccmp Compare netcdf files
  • sparse Sparse multi-dimensional arrays