Difference between revisions of "WRF v3.9 installation"

(Fix broken link)
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category: WRF]]
 +
 
This version is from 2017. Look for the latest version available at NCI at [[WRF  |  this page]].
 
This version is from 2017. Look for the latest version available at NCI at [[WRF  |  this page]].
  
=<span style="background-color: #ffffff;">'''Foreword'''</span>=  
+
=Foreword=  
----
+
 
<span style="background-color: #ffffff;">This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modifications to WRF and WPS.</span>
+
This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modifications to WRF and WPS.
<span style="background-color: #ffffff;">This document is only valid for version 3.9 of WRF and later. Please refer to the older version pages for other instructions.</span>
+
This document is only valid for version 3.9 of WRF and later. Please refer to the older version pages for other instructions.
 +
 
 +
=<span id="Source Code"></span>Source Code=
  
=<span style="background-color: #ffffff;"><span id="Source Code"></span>Source Code</span>=
+
NCAR has moved the WRF source code to Git version control. The original code for WRF v3.9 can be found on [https://github.com/NCAR/WRFV3 NCAR's release Github repository].
----
+
The source code ported to Raijin can be found both under /projects/WRF/WRFV_3.9 and on Github: [https://github.com/coecms/WRFV3 https://github.com/coecms/WRF]
<span style="background-color: #ffffff;">NCAR has moved the WRF source code to Git version control. The original code for WRF v3.9 can be found on [https://github.com/NCAR/WRFV3 | NCAR's release Github repository].</span>
 
<span style="background-color: #ffffff;">The source code ported to Raijin can be found both under /projects/WRF/WRFV_3.9 and on Github: </span>[https://github.com/coecms/WRFV3 | https:''github.com/coecms/WRF]
 
  
<span style="background-color: #ffffff;">Clone the WRF Github repository (using HTTPS or SSH protocol depending on your setup) to your own space under /short:</span>
+
Clone the WRF Github repository (using HTTPS or SSH protocol depending on your setup) to your own space under /short:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
cd /short/$PROJECT/$USER/
 
cd /short/$PROJECT/$USER/
git clone -b V3.9 https://github.com/coecms/WRF.git
+
git clone -b V3.9_raijin https://github.com/coecms/WRF.git
 
</syntaxhighlight>
 
</syntaxhighlight>
 
or
 
or
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
cd /short/$PROJECT/$USER/
 
cd /short/$PROJECT/$USER/
git clone -b V3.9 git@github.com:coecms/WRF.git
+
git clone -b V3.9_raijin git@github.com:coecms/WRF.git
  
 
</syntaxhighlight>
 
</syntaxhighlight>
=<span style="background-color: #ffffff;"><span id="Building WRF (ARW)"></span>'''Building WRF (ARW)'''</span>=  
+
=<span id="Building WRF (ARW)"></span>Building WRF (ARW)=  
----
 
<span style="background-color: #ffffff;">By default, WRF will output files in netcdf 3 format. If you would like to directly output files in netcdf 4 format with compression, you need to edit the file WRF/build.env before compilation. Note if you are planning to generate very frequent outputs, this will significantly slow down your simulation. You could be better off compressing the outputs afterwards but please don't forget to [[NetCDF Compression Tools | compress your data]]. </span>
 
  
<span style="background-color: #ffffff;">Go to the WRFV3/ subdirectory, run:</span>
+
By default, WRF will output files in netcdf 3 format. If you would like to directly output files in netcdf 4 format with compression, you need to edit the file WRF/build.env before compilation. Note if you are planning to generate very frequent outputs, this will significantly slow down your simulation. You could be better off compressing the outputs afterwards but please don't forget to [[NetCDF Compression Tools | compress your data]].
<syntaxhighlight>
+
 
 +
Go to the WRFV3/ subdirectory, run:
 +
<syntaxhighlight lang=text>
 
cd /short/$PROJECT/$USER/WRF/WRFV3/
 
cd /short/$PROJECT/$USER/WRF/WRFV3/
 
</syntaxhighlight>
 
</syntaxhighlight>
<span style="background-color: #ffffff;">There is a run_compile script to help you configure and compile WRF. This script by default will compile the WRFV3 software for a real experiment, with speed optimisations for the normal and express queues and with basic nesting. There is a range of command-line arguments accepted by the script. These are the command-line arguments accepted by the ''configure'' script, an option to clean everything prior to compilation, the choice of compilation flags and nesting type for the ''configure'' script and finally the compilation option for the ''compile'' script. You can access a full list and description by running:</span>
+
There is a run_compile script to help you configure and compile WRF. This script by default will compile the WRFV3 software for a real experiment, with speed optimisations for the normal and express queues and with basic nesting. There is a range of command-line arguments accepted by the script. These are the command-line arguments accepted by the ''configure'' script, an option to clean everything prior to compilation, the choice of compilation flags and nesting type for the ''configure'' script and finally the compilation option for the ''compile'' script. You can access a full list and description by running:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
./run_compile -h
 
./run_compile -h
 
</syntaxhighlight>
 
</syntaxhighlight>
 
To configure and compile WRFV3 with the default options, you simply run:
 
To configure and compile WRFV3 with the default options, you simply run:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
./run_compile
 
./run_compile
 
</syntaxhighlight>
 
</syntaxhighlight>
<span style="background-color: #ffffff;">The configure step will be run interactively with the output on your screen, then the compilation step will be submitted to the express queue automatically. The compilation takes about 30-45 minutes on the express queue. To check the status, you can use the </span>nqstat<span style="background-color: #ffffff;"> command.</span>
+
The configure step will be run interactively with the output on your screen, then the compilation step will be submitted to the express queue automatically. The compilation takes about 30-45 minutes on the express queue. To check the status, you can use the nqstat command.
  
<span style="background-color: #ffffff;">After a successful build, one should see the following executable files for WRF (the ARW core):</span>
+
After a successful build, one should see the following executable files for WRF (the ARW core):
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
[abc123@raijin3 WRFV3]$ ls -l main/*.exe
 
[abc123@raijin3 WRFV3]$ ls -l main/*.exe
  
Line 50: Line 52:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
**Tip''': if you are only running an idealised case and you are getting tired of specifying the option on the command-line, you can:
+
'''Tip''': if you are only running an idealised case and you are getting tired of specifying the option on the command-line, you can:
 
# change the default value of compile_case at the start of the run_compile script OR
 
# change the default value of compile_case at the start of the run_compile script OR
 
# create an alias for the command in your .bashrc or .cshrc file (it is a good idea to change the name of the command for the alias so it is clear it is an alias if you need help later on):
 
# create an alias for the command in your .bashrc or .cshrc file (it is a good idea to change the name of the command for the alias so it is clear it is an alias if you need help later on):
 
bash syntax:
 
bash syntax:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
alias my_compile='run_compile --compile_case <replace with the compile option here>'
 
alias my_compile='run_compile --compile_case <replace with the compile option here>'
 
</syntaxhighlight>
 
</syntaxhighlight>
 
csh/tcsh syntax:
 
csh/tcsh syntax:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
alias my_compile 'run_compile --compile_case <replace with the compile option here>'
 
alias my_compile 'run_compile --compile_case <replace with the compile option here>'
 
</syntaxhighlight>
 
</syntaxhighlight>
  
**Tip:''' run_compile is sourcing the environment as defined in WRF/build.env. Feel free to change the versions of software you want to use in here. Note the ''module purge'' at the start. This means your currently loaded environment will be completely ignored.
+
'''Tip:''' run_compile is sourcing the environment as defined in WRF/build.env. Feel free to change the versions of software you want to use in here. Note the ''module purge'' at the start. This means your currently loaded environment will be completely ignored.
 
=Debugging with WRF=  
 
=Debugging with WRF=  
----
+
 
 
If you want to compile WRF for debugging, the default option for compilation is not valid because it includes some optimisation. You'll need to use one of the first four architecture options. For example, to debug with the distributed memory only option (dmpar):
 
If you want to compile WRF for debugging, the default option for compilation is not valid because it includes some optimisation. You'll need to use one of the first four architecture options. For example, to debug with the distributed memory only option (dmpar):
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
./run_compile -c -a 3 -D
 
./run_compile -c -a 3 -D
 
</syntaxhighlight>
 
</syntaxhighlight>
=<span style="background-color: #ffffff;"><span id="Building WPS"></span>'''Building WPS'''</span>=  
+
=<span id="Building WPS"></span>Building WPS=  
----
+
Go to the WPS/ subdirectory, run:
<span style="background-color: #ffffff;">Go to the WPS/ subdirectory, run:</span>
+
<syntaxhighlight lang=text>
<syntaxhighlight>
 
 
cd /short/$PROJECT/$USER/WRF/WPS/
 
cd /short/$PROJECT/$USER/WRF/WPS/
 
</syntaxhighlight>
 
</syntaxhighlight>
<span style="background-color: #ffffff;">There is another ''run_compile'' script to help with configuring and compiling WPS. You can again choose the input to the configure script from the command-line option as well as cleaning a previous compilation of WPS. The default is to compile with GRIB2 support and a distributed memory code. See the inline help for ''run_compile'' for a full explanation:</span>
+
There is another ''run_compile'' script to help with configuring and compiling WPS. You can again choose the input to the configure script from the command-line option as well as cleaning a previous compilation of WPS. The default is to compile with GRIB2 support and a distributed memory code. See the inline help for ''run_compile'' for a full explanation:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
./run_compile -h
 
./run_compile -h
 
</syntaxhighlight>
 
</syntaxhighlight>
  
<span style="background-color: #ffffff;">To configure and compile WPS with the default options, you simply run:</span>
+
To configure and compile WPS with the default options, you simply run:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
./run_compile
 
./run_compile
 
</syntaxhighlight>
 
</syntaxhighlight>
<span style="background-color: #ffffff;">A file configure.wps should be created after configuring has completed.</span>
+
A file configure.wps should be created after configuring has completed.
  
<span style="background-color: #ffffff;">After a successful build, one should see the following executable files:</span>
+
After a successful build, one should see the following executable files:
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
[abc123@raijin3 WPS]$ ls -l *.exe
 
[abc123@raijin3 WPS]$ ls -l *.exe
  
Line 94: Line 95:
 
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
 
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
 
</syntaxhighlight>
 
</syntaxhighlight>
<syntaxhighlight>
+
<syntaxhighlight lang=text>
 
[abc123@raijin3 WPS]$ ls -l */src/*.exe
 
[abc123@raijin3 WPS]$ ls -l */src/*.exe
  
Line 111: Line 112:
 
-rwx------ 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe
 
-rwx------ 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe
 
</syntaxhighlight>
 
</syntaxhighlight>
<span style="background-color: #ffffff;">Note that there are some warnings in the build output like "warning: overriding commands for target `.c.o'." These are harmless and can be ignored.</span>
+
Note that there are some warnings in the build output like "warning: overriding commands for target `.c.o'." These are harmless and can be ignored.
 
 
 
 
== <span style="background-color: #ffffff;">Running WRF</span> ==
 
  
----
 
  
Please follow the [http://www2.mmm.ucar.edu/wrf/OnLineTutorial/index.php | WRF tutorial] to learn how to run WRF. For convenience, there are 2 run scripts under WRFV3/run:
+
== Running WRF ==
 +
Please follow the [http://www2.mmm.ucar.edu/wrf/OnLineTutorial/index.php WRF tutorial] to learn how to run WRF. For convenience, there are 2 run scripts under WRFV3/run:
  
 
*run_real  
 
*run_real  
Line 125: Line 123:
 
These can be used to submit both real.exe and wrf.exe to the queues.
 
These can be used to submit both real.exe and wrf.exe to the queues.
  
=<span style="background-color: #ffffff;"><span id="Tests run"></span>Tests run</span>=  
+
=<span id="Tests run"></span>Tests run=  
----
+
==<span id="Tests run-Compilation"></span>Compilation==  
==<span style="background-color: #ffffff;"><span id="Tests run-Compilation"></span><span id="Tests run-Compilation"></span>Compilation</span>==  
+
* WRF compilation with dmpar
* <span style="background-color: #ffffff;">WRF compilation with dmpar</span>
+
* WRF compilation with dm+sm
* <span style="background-color: #ffffff;">WRF compilation with dm+sm</span>
+
* WPS compilation with distributed memory with dm+sm compilation of WRF
* <span style="background-color: #ffffff;">WPS compilation with distributed memory with dm+sm compilation of WRF</span>
+
==<span id="Tests run-Tests simulations"></span>Tests simulations==  
==<span style="background-color: #ffffff;"><span id="Tests run-Tests simulations"></span><span id="Tests run-Tests simulations"></span>Tests simulations</span>==  
+
Tests simulations have been done on [https://accessdev.nci.org.au/jenkins/job/WRF/job/WRF-Core/ Jenkins]. The simulations were:
----
 
Tests simulations have been done on [https://accessdev.nci.org.au/jenkins/job/WRF/job/WRF-Core/ | Jenkins]. The simulations were:
 
 
* First tutorial case (Jan 00) with 1 nest
 
* First tutorial case (Jan 00) with 1 nest
 
* First tutorial case (Jan 00) with 2 nests
 
* First tutorial case (Jan 00) with 2 nests
 
* First tutorial case (Jan 00) with additional diagnostics turned on.
 
* First tutorial case (Jan 00) with additional diagnostics turned on.
 
WRF outputs for these tests can be found under /projects/WRF/data/KGO/. The inputs for the tests can be found in https://github.com/coecms/wrf-testing
 
WRF outputs for these tests can be found under /projects/WRF/data/KGO/. The inputs for the tests can be found in https://github.com/coecms/wrf-testing

Latest revision as of 00:29, 12 December 2019


This version is from 2017. Look for the latest version available at NCI at this page.

Foreword

This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modifications to WRF and WPS. This document is only valid for version 3.9 of WRF and later. Please refer to the older version pages for other instructions.

Source Code

NCAR has moved the WRF source code to Git version control. The original code for WRF v3.9 can be found on NCAR's release Github repository. The source code ported to Raijin can be found both under /projects/WRF/WRFV_3.9 and on Github: https://github.com/coecms/WRF

Clone the WRF Github repository (using HTTPS or SSH protocol depending on your setup) to your own space under /short:

cd /short/$PROJECT/$USER/
git clone -b V3.9_raijin https://github.com/coecms/WRF.git

or

cd /short/$PROJECT/$USER/
git clone -b V3.9_raijin git@github.com:coecms/WRF.git

Building WRF (ARW)

By default, WRF will output files in netcdf 3 format. If you would like to directly output files in netcdf 4 format with compression, you need to edit the file WRF/build.env before compilation. Note if you are planning to generate very frequent outputs, this will significantly slow down your simulation. You could be better off compressing the outputs afterwards but please don't forget to compress your data.

Go to the WRFV3/ subdirectory, run:

cd /short/$PROJECT/$USER/WRF/WRFV3/

There is a run_compile script to help you configure and compile WRF. This script by default will compile the WRFV3 software for a real experiment, with speed optimisations for the normal and express queues and with basic nesting. There is a range of command-line arguments accepted by the script. These are the command-line arguments accepted by the configure script, an option to clean everything prior to compilation, the choice of compilation flags and nesting type for the configure script and finally the compilation option for the compile script. You can access a full list and description by running:

./run_compile -h

To configure and compile WRFV3 with the default options, you simply run:

./run_compile

The configure step will be run interactively with the output on your screen, then the compilation step will be submitted to the express queue automatically. The compilation takes about 30-45 minutes on the express queue. To check the status, you can use the nqstat command.

After a successful build, one should see the following executable files for WRF (the ARW core):

[abc123@raijin3 WRFV3]$ ls -l main/*.exe

-rwx------ 1 abc123 wrf 35559897 Aug 22 17:14 main/ndown.exe
-rwx------ 1 abc123 wrf 35471615 Aug 22 17:14 main/real.exe
-rwx------ 1 abc123 wrf 35081246 Aug 22 17:14 main/tc.exe
-rwx------ 1 abc123 wrf 38763058 Aug 22 17:13 main/wrf.exe

Tip: if you are only running an idealised case and you are getting tired of specifying the option on the command-line, you can:

  1. change the default value of compile_case at the start of the run_compile script OR
  2. create an alias for the command in your .bashrc or .cshrc file (it is a good idea to change the name of the command for the alias so it is clear it is an alias if you need help later on):

bash syntax:

alias my_compile='run_compile --compile_case <replace with the compile option here>'

csh/tcsh syntax:

alias my_compile 'run_compile --compile_case <replace with the compile option here>'

Tip: run_compile is sourcing the environment as defined in WRF/build.env. Feel free to change the versions of software you want to use in here. Note the module purge at the start. This means your currently loaded environment will be completely ignored.

Debugging with WRF

If you want to compile WRF for debugging, the default option for compilation is not valid because it includes some optimisation. You'll need to use one of the first four architecture options. For example, to debug with the distributed memory only option (dmpar):

./run_compile -c -a 3 -D

Building WPS

Go to the WPS/ subdirectory, run:

cd /short/$PROJECT/$USER/WRF/WPS/

There is another run_compile script to help with configuring and compiling WPS. You can again choose the input to the configure script from the command-line option as well as cleaning a previous compilation of WPS. The default is to compile with GRIB2 support and a distributed memory code. See the inline help for run_compile for a full explanation:

./run_compile -h

To configure and compile WPS with the default options, you simply run:

./run_compile

A file configure.wps should be created after configuring has completed.

After a successful build, one should see the following executable files:

[abc123@raijin3 WPS]$ ls -l *.exe

lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:19 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:20 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
[abc123@raijin3 WPS]$ ls -l */src/*.exe

-rwx------ 1 abc123 wrf 3412782 Aug 23 10:19 geogrid/src/geogrid.exe
-rwx------ 1 abc123 wrf 3206974 Aug 23 10:20 metgrid/src/metgrid.exe
-rwx------ 1 abc123 wrf 1209622 Aug 23 10:20 ungrib/src/g1print.exe
-rwx------ 1 abc123 wrf 1407507 Aug 23 10:20 ungrib/src/g2print.exe
-rwx------ 1 abc123 wrf 2169010 Aug 23 10:20 ungrib/src/ungrib.exe
-rwx------ 1 abc123 wrf 1184660 Aug 23 10:20 util/src/avg_tsfc.exe
-rwx------ 1 abc123 wrf 1250303 Aug 23 10:20 util/src/calc_ecmwf_p.exe
-rwx------ 1 abc123 wrf 1217842 Aug 23 10:20 util/src/height_ukmo.exe
-rwx------ 1 abc123 wrf 953821 Aug 23 10:20 util/src/int2nc.exe
-rwx------ 1 abc123 wrf 1118699 Aug 23 10:20 util/src/mod_levs.exe
-rwx------ 1 abc123 wrf 3704627 Aug 23 10:20 util/src/plotfmt.exe
-rwx------ 1 abc123 wrf 3409302 Aug 23 10:20 util/src/plotgrids.exe
-rwx------ 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe

Note that there are some warnings in the build output like "warning: overriding commands for target `.c.o'." These are harmless and can be ignored.


Running WRF

Please follow the WRF tutorial to learn how to run WRF. For convenience, there are 2 run scripts under WRFV3/run:

  • run_real
  • run_mpi

These can be used to submit both real.exe and wrf.exe to the queues.

Tests run

Compilation

  • WRF compilation with dmpar
  • WRF compilation with dm+sm
  • WPS compilation with distributed memory with dm+sm compilation of WRF

Tests simulations

Tests simulations have been done on Jenkins. The simulations were:

  • First tutorial case (Jan 00) with 1 nest
  • First tutorial case (Jan 00) with 2 nests
  • First tutorial case (Jan 00) with additional diagnostics turned on.

WRF outputs for these tests can be found under /projects/WRF/data/KGO/. The inputs for the tests can be found in https://github.com/coecms/wrf-testing