Difference between revisions of "Conda"
(Created page with "== Conda Python Environments == CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries. You can find the most recen...") |
(→20.07 (Current Unstable)) |
||
(21 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
+ | |||
== Conda Python Environments == | == Conda Python Environments == | ||
Line 5: | Line 6: | ||
You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded. | You can find the most recent list of libraries at our [https://github.com/coecms/conda-envs/blob/analysis3/environment.yml github repository], or run <code>conda list</code> with an environment loaded. | ||
− | To use any of the conda environments | + | To use any of the conda environments: |
+ | # Request access to [https://my.nci.org.au/mancini/project/hh5 hh5] (to do once) | ||
+ | # You must first run (to do at each session) | ||
− | <code> | + | <code>module use /g/data3/hh5/public/modules</code> |
− | |||
− | </code> | ||
− | You can safely put this in your | + | You can safely put this in your <code>~/.profile</code> or <code>~/.login</code> file |
=== Stable Environment === | === Stable Environment === | ||
Line 17: | Line 18: | ||
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Raijin. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken. | We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Raijin. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken. | ||
− | <code> | + | <code>module load conda/analysis3</code> |
− | module load conda/analysis3 | ||
− | </code> | ||
=== Unstable Environment === | === Unstable Environment === | ||
Line 25: | Line 24: | ||
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here. | The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here. | ||
− | <code> | + | <code>module load conda/analysis3-unstable</code> |
− | module load conda/analysis3-unstable | ||
− | </code> | ||
When we do our quarterly update the unstable environment becomes the new stable environment. | When we do our quarterly update the unstable environment becomes the new stable environment. | ||
+ | |||
+ | == Creating personal environments == | ||
+ | |||
+ | You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file <code>~/.condarc</code> like: | ||
+ | |||
+ | <syntaxhighlight lang="none"> | ||
+ | auto_activate_base: false | ||
+ | envs_dirs: | ||
+ | - /scratch/$PROJECT/$USER/conda/envs | ||
+ | - /g/data/hh5/public/apps/miniconda3/envs | ||
+ | pkgs_dirs: | ||
+ | - /scratch/$PROJECT/$USER/conda/pkgs | ||
+ | conda-build: | ||
+ | root-dir: /scratch/$PROJECT/$USER/conda/bld | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota. | ||
+ | |||
+ | == Interactive Analysis / Jupyter == | ||
+ | |||
+ | For interactive analysis we encourage making use of NCI's [[VDI|VDI]] system, which allows you to run Jupyter notebooks without waiting in the queue | ||
== Requesting new packages == | == Requesting new packages == | ||
You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. | You can ask for a new package to be installed or for an existing package to be updated by emailing [mailto:cws_help@nci.org.au cws_help@nci.org.au]. | ||
+ | |||
+ | == Update History == | ||
+ | |||
+ | === 20.10 (Future Unstable) === | ||
+ | |||
+ | === 20.07 (Current Unstable) === | ||
+ | |||
+ | '''Notable New Packages''' | ||
+ | * [https://xesmf.readthedocs.io/en/latest/ xesmf] - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI) | ||
+ | * [http://sharp.weather.ou.edu/dev/ sharppy] - sounding and holograph analysis | ||
+ | * [https://earthpy.readthedocs.io/en/latest/ earthpy] - spatial raster and vector tools (e.g. rasterise shapefiles) | ||
+ | * [https://pypi.org/project/descartes/ descartes] - plot shapefiles | ||
+ | * [https://github.com/coecms/era5grib era5grib] - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool) | ||
+ | * [https://xmitgcm.readthedocs.io xmitgcm] - read mitgcm binary output | ||
+ | |||
+ | === 20.04 (Current Stable) === | ||
+ | |||
+ | Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes | ||
+ | |||
+ | Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI | ||
+ | |||
+ | '''Notable New Packages''' | ||
+ | * geopy: Locate lat/lon coordinates of places | ||
+ | * ninja: mom6 build system | ||
+ | |||
+ | === 20.01 (Unsupported) === | ||
+ | |||
+ | Python has been updated to 3.7 | ||
+ | |||
+ | '''Notable New Packages''' | ||
+ | * xlrd: Read excel files | ||
+ | * ants: Unified Model Ancillary tools | ||
+ | * climtas: Dask-aware Xarray timeseries processing | ||
+ | |||
+ | === 19.10 (Unsupported) === | ||
+ | |||
+ | basemap has been removed as it is [https://matplotlib.org/basemap/users/intro.html#deprecation-notice no longer supported] and caused conflicts with other packages | ||
+ | |||
+ | '''Notable New Packages''' | ||
+ | * addmeta | ||
+ | * [http://holoviews.org/ holoviews]: Fancy plots | ||
+ | * [https://jiffyclub.github.io/palettable/ palettable]: Matplotlib colour palettes | ||
+ | |||
+ | === 19.07 (Unsupported) === | ||
+ | |||
+ | '''Notable New Packages''' | ||
+ | * pyferret | ||
+ | * pyngl | ||
+ | * pynio | ||
+ | * xgcm | ||
+ | * xrft | ||
+ | |||
+ | === 19.04 (unsupported) === | ||
+ | |||
+ | '''Notable Changes''' | ||
+ | |||
+ | *arccssive has been renamed to clef | ||
+ | *A bug preventing Iris from opening UM files without a date has been fixed | ||
+ | |||
+ | '''Notable New Packages''' | ||
+ | |||
+ | *[https://github.com/kwgoodman/bottleneck bottleneck] Fast rolling operations | ||
+ | *[https://github.com/ecmwf/cfgrib cfgrib] CF metadata for GRIB files | ||
+ | *[https://github.com/SciTools/cf-units cfunits] Convert between CF units | ||
+ | *[https://github.com/shoyer/h5netcdf h5netcdf] Pythonic interface to netCDF4 via h5py | ||
+ | *[https://github.com/intake/intake intake] Lightweight data catalogues | ||
+ | *[https://gitlab.com/remikz/nccmp/ nccmp] Compare netcdf files | ||
+ | *[https://github.com/pydata/sparse sparse] Sparse multi-dimensional arrays | ||
+ | |||
+ | [[Category:NCI]] [[Category:Python]] |
Revision as of 21:48, 29 September 2020
Contents
Conda Python Environments
CMS maintain an Anaconda Python environment at NCI, with a wide variety of climate and weather related libraries.
You can find the most recent list of libraries at our github repository, or run conda list
with an environment loaded.
To use any of the conda environments:
- Request access to hh5 (to do once)
- You must first run (to do at each session)
module use /g/data3/hh5/public/modules
You can safely put this in your ~/.profile
or ~/.login
file
Stable Environment
We update the stable environment once a quarter, around when NCI do their quarterly maintenance of Raijin. Otherwise everything in the environment stays fixed, we don't update packages or install new packages unless something is very broken.
module load conda/analysis3
Unstable Environment
The unstable environment gets updated more often, as we install new packages or apply updates to existing ones. If you ask for a new package it will be installed here.
module load conda/analysis3-unstable
When we do our quarterly update the unstable environment becomes the new stable environment.
Creating personal environments
You can create your own environment if needed, but please be cautious of both the size on disk and number of files that Conda environments can create. Make a file ~/.condarc
like:
auto_activate_base: false
envs_dirs:
- /scratch/$PROJECT/$USER/conda/envs
- /g/data/hh5/public/apps/miniconda3/envs
pkgs_dirs:
- /scratch/$PROJECT/$USER/conda/pkgs
conda-build:
root-dir: /scratch/$PROJECT/$USER/conda/bld
This will set up Conda to create environments in /scratch, by default it puts them in your home directory which will rapidly use up your disk quota.
Interactive Analysis / Jupyter
For interactive analysis we encourage making use of NCI's VDI system, which allows you to run Jupyter notebooks without waiting in the queue
Requesting new packages
You can ask for a new package to be installed or for an existing package to be updated by emailing cws_help@nci.org.au.
Update History
20.10 (Future Unstable)
20.07 (Current Unstable)
Notable New Packages
- xesmf - regrids xarray data (NOTE: uses Gadi's ESMF install, won't work on VDI)
- sharppy - sounding and holograph analysis
- earthpy - spatial raster and vector tools (e.g. rasterise shapefiles)
- descartes - plot shapefiles
- era5grib - convert data from NCI ERA5 archive to GRIB for use in WRF/UM (beta tool)
- xmitgcm - read mitgcm binary output
20.04 (Current Stable)
Cartopy NaturalEarth source data has been centrally installed, so coastlines etc. can be drawn on compute nodes
Clef has been updated to 1.0, can now find ACCESS model CMIP6 data published by NCI
Notable New Packages
- geopy: Locate lat/lon coordinates of places
- ninja: mom6 build system
20.01 (Unsupported)
Python has been updated to 3.7
Notable New Packages
- xlrd: Read excel files
- ants: Unified Model Ancillary tools
- climtas: Dask-aware Xarray timeseries processing
19.10 (Unsupported)
basemap has been removed as it is no longer supported and caused conflicts with other packages
Notable New Packages
- addmeta
- holoviews: Fancy plots
- palettable: Matplotlib colour palettes
19.07 (Unsupported)
Notable New Packages
- pyferret
- pyngl
- pynio
- xgcm
- xrft
19.04 (unsupported)
Notable Changes
- arccssive has been renamed to clef
- A bug preventing Iris from opening UM files without a date has been fixed
Notable New Packages