Difference between revisions of "Transition to Gadi"

(ACCESS 1)
 
(53 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
 
This page presents information specific to CLEx users and the status of the models supported by CMS.
 
This page presents information specific to CLEx users and the status of the models supported by CMS.
  
 
The main source of information for Gadi is hosted on NCI help pages: [https://opus.nci.org.au/display/Help/Preparing+for+Gadi Preparing for Gadi]. This page will only list additional, specific information. Please continue to refer to NCI's pages for main information.
 
The main source of information for Gadi is hosted on NCI help pages: [https://opus.nci.org.au/display/Help/Preparing+for+Gadi Preparing for Gadi]. This page will only list additional, specific information. Please continue to refer to NCI's pages for main information.
 +
 +
 
 +
 +
== Progress ==
 +
 +
This is the list of model versions that will be ported by us on Gadi. Please contact [mailto:cws_help@nci.org.au cws_help@nci.org.au] if you need us to port other versions. Note: we do not support CESM or any of its derivated models. It might not be possible to port all versions requested, any problematic request will be discussed with the Infrastructure Committee for a decision. }
 +
 +
=== Updates - Week of 25th Nov ===
 +
 +
==== Messages from NCI ====
 +
 +
Nov 27 2019: Decomissioning of Portions of Normal/Express Queues Approximately half of the nodes servicing the normal/express queues have been decomissioned to allow for power reticulation works to comission Phase 2 of Gadi. Increased wait times for these queues are now likely.
 +
 +
We recommend that users migrate to Gadi as soon as possible.
 +
 +
 
 +
 +
==== ACCESS 1 ====
 +
 +
ACCESS 1 AMIP jobs are working on Gadi, but they're not currently bit-reproducible under different processor decompositions. This may lead to crashes in some decompositions that go away if you change the processor layout. Performance is roughly twice as fast in walltime as Raijin's default nodes, however due to the accounting changes the SU cost of a run will be roughly the same as it was on Raijin. We recommend starting with a decomposition of 16 x 12 processors for ACCESS 1 runs at the default 'n96' resolution.
 +
 +
Instructions for moving an existing ACCESS 1 AMIP job from Raijin are available at [https://accessdev.nci.org.au/trac/wiki/gadi#UMUIUMvn7ACCESS1 https://accessdev.nci.org.au/trac/wiki/gadi#UMUIUMvn7ACCESS1] Please let us know if this doesn't work for your run - it may require some changes for specific configurations.
 +
 +
New users of the UMUI / ACCESS 1 should follow the [[UM Environment|UM environment setup guide]]
 +
 +
The ACCESS support team is cleaning up input files - '''please check you run has all the input files it needs before Jan 6th'''
 +
 +
Working runs and performance information are available listed under [http://climate-cms.wikis.unsw.edu.au/UM_On_Gadi#UM7.3 http://climate-cms.wikis.unsw.edu.au/UM_On_Gadi#UM7.3]
 +
 +
{| class="wikitable"
 +
|-
 +
! Model
 +
! Status
 +
! More Info & Performance Metrics
 +
|-
 +
| UM 7.3 (ACCESS AMIP 1.X)
 +
| WORKING ([[Transition_to_Gadi#ACCESS_1|Not Bit-Repro]])
 +
| [[UM_On_Gadi#UM7.3|UM On Gadi#UM7.3]]
 +
|-
 +
| UM 8.5
 +
| WORKING
 +
| [[UM On Gadi#UM8.5]]
 +
|  
 +
|-
 +
| UM 10.4 (ACCESS AMIP 2)
 +
| TODO
 +
| [[UM_On_Gadi#UM10.4|UM On Gadi#UM10.4]]
 +
|-
 +
| UM 11.4 (GA 7 / Nested)
 +
| WORKING
 +
| [[UM_On_Gadi#UM11.4|UM On Gadi#UM11.4]]
 +
|-
 +
| ACCESS-ESM 1.5
 +
| IN PROGRESS
 +
|-
 +
| ACCESS-CM 2
 +
| TODO
 +
|-
 +
| ACCESS-CM 2 N48
 +
| TODO
 +
|-
 +
| MOM
 +
| TODO
 +
|-
 +
| CABLE
 +
| TODO
 +
|-
 +
| NU-WRF
 +
| TODO
 +
|-
 +
| WRF 4.1.3
 +
| WORKING
 +
| [[WRF_v4.1.3_installation|WRF v4.1.3 installation]]
 +
|-
 +
| WRF 4.1.2
 +
| WORKING
 +
| [[WRF_v4.1.2_installation|WRF v4.1.2 installation]]
 +
|-
 +
| WRF 4.1.1
 +
| WORKING
 +
| [[WRF_v4.1.1_installation|WRF v4.1.1 installation]]
 +
|-
 +
| WRF 4.0.2
 +
| WORKING
 +
| [[WRF v4.0.2 installation]]
 +
|-
 +
| WRF 3.9.1.1 Chem
 +
| WORKING
 +
|-
 +
| WRF 3.9
 +
| IN PROGRESS
 +
|-
 +
| WRF 3.7.1
 +
| WORKING
 +
| [[WRF v3.7.1 installation]]
 +
|-
 +
| WRF 3.6.1
 +
| IN PROGRESS
 +
|-
 +
| WRF 3.6
 +
| IN PROGRESS
 +
|-
 +
| WRF 3.5.1
 +
| TODO
 +
|-
 +
| WPS data
 +
| Updated. See [[How_to_run_WRF|How_to_run_WRF]] for information.
 +
Also valid for runs on Raijin
 +
 +
|-
 +
| MITgcm
 +
| WORKING
 +
| [[MITgcm_on_gadi|MITgcm on gadi]]
 +
|}
 +
 +
== Default project setting ==
 +
 +
When you first access Gadi, you should check your default project is set appropriately.
 +
 +
Open the file: ''~/.config/gadi-login.conf''. Check the value of the ''PROJECT'' variable, set it to the project you want to use as default.
  
 
== Access to projects ==
 
== Access to projects ==
On Gadi, there will no public access to any project. Additionally, only members of a project can access this project's filesystem space in jobs submitted to the HPC nodes.  
+
 
 +
On Gadi, there will no public access to any project. Additionally, only members of a project can access this project's filesystem space in jobs submitted to the HPC nodes.
 +
 
 +
Some environments you may wish to join are (follow the link to join the project):
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Project Code
 +
! Contact
 +
! Description
 +
|-
 +
| [https://my.nci.org.au/mancini/project/hh5 hh5]
 +
| cws_help@nci.org.au
 +
| Conda environment
 +
|-
 +
| [https://my.nci.org.au/mancini/project/sx70 sx70]
 +
| cws_help@nci.org.au
 +
| WRF Input Data
 +
|-
 +
| [https://my.nci.org.au/mancini/project/access access]
 +
| cws_help@nci.org.au
 +
| ACCESS/UM Input Data
 +
|-
 +
| [https://my.nci.org.au/mancini/project/xc57 xc57]
 +
| cws_help@nci.org.au
 +
| Permission to use VDI (only if you're not in a [[CLEX_projects_at_NCI|compute project]])
 +
|}
  
 
=== conda environments ===
 
=== conda environments ===
 +
 
If you are using the conda environments under /g/data/hh5/public/modules, you need to request membership of the hh5 project if not already done. [https://my.nci.org.au/mancini/project/hh5 Here].
 
If you are using the conda environments under /g/data/hh5/public/modules, you need to request membership of the hh5 project if not already done. [https://my.nci.org.au/mancini/project/hh5 Here].
  
 
== Models not supported by CMS ==
 
== Models not supported by CMS ==
You should try and update the library your model is using to the versions that will be installed on Gadi as much as possible.
 
Then test if your model is compiling and working with the updated libraries
 
  
 +
You should try and update the library your model is using to the versions that will be installed on Gadi as much as possible. Then test if your model is compiling and working with the updated libraries
 +
 +
 
 +
 +
== Climate Models supported by CMS ==
  
== Climate Models supported by CMS==
 
 
=== WRF ===
 
=== WRF ===
Some issues appeared when compiling WRF with OpenMPI v4.0.1. These are being worked out.
+
/projects/WRF will disappear. It has now been replaced by a new project: sx70. All users of WRF will need to be a [https://my.nci.org.au/mancini/project/sx70 member of sx70] to have access to the GEOG dataset. The sx70 project should now be used from Raijin or Gadi.
WRF is currently running on raijin with OpenMPI v2.1.3 so it is likely to run smoothly with other OpenMPI versions that will be installed on Gadi. We cannot test before access to Gadi.
 
  
 
=== NUWRF ===
 
=== NUWRF ===
 +
 
NUWRF is being ported to Gadi with NCI's help. We expect a working solution will be ready shortly after Gadi is available.
 
NUWRF is being ported to Gadi with NCI's help. We expect a working solution will be ready shortly after Gadi is available.
 +
 +
=== UM with UMUI (e.g. ACCESS 1.X AMIP ) ===
 +
 +
Please see the [https://accessdev.nci.org.au/trac/wiki/gadi#UMUIUMvn7ACCESS1 ACCESS wiki] for information
 +
 +
=== UM with Rose/Cylc (e.g. ACCESS 2, GA) ===
 +
 +
Please see the [https://accessdev.nci.org.au/trac/wiki/gadi#RoseCylcUMvn10ACCESS2orlater ACCESS wiki] for information
 +
 +
[[Category:Gadi]]

Latest revision as of 19:36, 16 February 2020

This page presents information specific to CLEx users and the status of the models supported by CMS.

The main source of information for Gadi is hosted on NCI help pages: Preparing for Gadi. This page will only list additional, specific information. Please continue to refer to NCI's pages for main information.

 

Progress

This is the list of model versions that will be ported by us on Gadi. Please contact cws_help@nci.org.au if you need us to port other versions. Note: we do not support CESM or any of its derivated models. It might not be possible to port all versions requested, any problematic request will be discussed with the Infrastructure Committee for a decision. }

Updates - Week of 25th Nov

Messages from NCI

Nov 27 2019: Decomissioning of Portions of Normal/Express Queues Approximately half of the nodes servicing the normal/express queues have been decomissioned to allow for power reticulation works to comission Phase 2 of Gadi. Increased wait times for these queues are now likely.

We recommend that users migrate to Gadi as soon as possible.

 

ACCESS 1

ACCESS 1 AMIP jobs are working on Gadi, but they're not currently bit-reproducible under different processor decompositions. This may lead to crashes in some decompositions that go away if you change the processor layout. Performance is roughly twice as fast in walltime as Raijin's default nodes, however due to the accounting changes the SU cost of a run will be roughly the same as it was on Raijin. We recommend starting with a decomposition of 16 x 12 processors for ACCESS 1 runs at the default 'n96' resolution.

Instructions for moving an existing ACCESS 1 AMIP job from Raijin are available at https://accessdev.nci.org.au/trac/wiki/gadi#UMUIUMvn7ACCESS1 Please let us know if this doesn't work for your run - it may require some changes for specific configurations.

New users of the UMUI / ACCESS 1 should follow the UM environment setup guide

The ACCESS support team is cleaning up input files - please check you run has all the input files it needs before Jan 6th

Working runs and performance information are available listed under http://climate-cms.wikis.unsw.edu.au/UM_On_Gadi#UM7.3

Model Status More Info & Performance Metrics
UM 7.3 (ACCESS AMIP 1.X) WORKING (Not Bit-Repro) UM On Gadi#UM7.3
UM 8.5 WORKING UM On Gadi#UM8.5  
UM 10.4 (ACCESS AMIP 2) TODO UM On Gadi#UM10.4
UM 11.4 (GA 7 / Nested) WORKING UM On Gadi#UM11.4
ACCESS-ESM 1.5 IN PROGRESS
ACCESS-CM 2 TODO
ACCESS-CM 2 N48 TODO
MOM TODO
CABLE TODO
NU-WRF TODO
WRF 4.1.3 WORKING WRF v4.1.3 installation
WRF 4.1.2 WORKING WRF v4.1.2 installation
WRF 4.1.1 WORKING WRF v4.1.1 installation
WRF 4.0.2 WORKING WRF v4.0.2 installation
WRF 3.9.1.1 Chem WORKING
WRF 3.9 IN PROGRESS
WRF 3.7.1 WORKING WRF v3.7.1 installation
WRF 3.6.1 IN PROGRESS
WRF 3.6 IN PROGRESS
WRF 3.5.1 TODO
WPS data Updated. See How_to_run_WRF for information.

Also valid for runs on Raijin

MITgcm WORKING MITgcm on gadi

Default project setting

When you first access Gadi, you should check your default project is set appropriately.

Open the file: ~/.config/gadi-login.conf. Check the value of the PROJECT variable, set it to the project you want to use as default.

Access to projects

On Gadi, there will no public access to any project. Additionally, only members of a project can access this project's filesystem space in jobs submitted to the HPC nodes.

Some environments you may wish to join are (follow the link to join the project):

Project Code Contact Description
hh5 cws_help@nci.org.au Conda environment
sx70 cws_help@nci.org.au WRF Input Data
access cws_help@nci.org.au ACCESS/UM Input Data
xc57 cws_help@nci.org.au Permission to use VDI (only if you're not in a compute project)

conda environments

If you are using the conda environments under /g/data/hh5/public/modules, you need to request membership of the hh5 project if not already done. Here.

Models not supported by CMS

You should try and update the library your model is using to the versions that will be installed on Gadi as much as possible. Then test if your model is compiling and working with the updated libraries

 

Climate Models supported by CMS

WRF

/projects/WRF will disappear. It has now been replaced by a new project: sx70. All users of WRF will need to be a member of sx70 to have access to the GEOG dataset. The sx70 project should now be used from Raijin or Gadi.

NUWRF

NUWRF is being ported to Gadi with NCI's help. We expect a working solution will be ready shortly after Gadi is available.

UM with UMUI (e.g. ACCESS 1.X AMIP )

Please see the ACCESS wiki for information

UM with Rose/Cylc (e.g. ACCESS 2, GA)

Please see the ACCESS wiki for information