Difference between revisions of "How to run WRF"

 
Line 32: Line 32:
 
For some configurations, it might be advantageous to run with OpenMP in addition to OpenMPI or OpenMP alone for some idealised cases. It is impossible to know beforehand whether OpenMP will improve or deteriorate performances. You will need to try your simulation on a short time to see the effect.
 
For some configurations, it might be advantageous to run with OpenMP in addition to OpenMPI or OpenMP alone for some idealised cases. It is impossible to know beforehand whether OpenMP will improve or deteriorate performances. You will need to try your simulation on a short time to see the effect.
  
By default, the compilation is done for OpenMPI and OpenMP so you don't need to recompile. Below is an example of an mpirun command with OpenMP as well:
+
By default, the compilation is done for OpenMPI and OpenMP so you don't need to recompile. Below is an example of a mpirun command with OpenMP in bash:
<code>
+
 
 +
<pre>
 
export OMP_NUM_THREADS=2
 
export OMP_NUM_THREADS=2
 
NCORE=$PBS_NCPUS/$OMP_NUM_THREADS
 
NCORE=$PBS_NCPUS/$OMP_NUM_THREADS
 
mpirun -np $NCORE --map-by node:PE=$OMP_NUM_THREADS --rank-by core ./wrf.exe
 
mpirun -np $NCORE --map-by node:PE=$OMP_NUM_THREADS --rank-by core ./wrf.exe
 +
</pre>
 +
 +
<code>OMP_NUM_THREADS</code>:
 +
 +
indicates the number of OpenMP threads you want to use:
 +
* If using OpenMPI + OpenMP, then 2 OpenMP threads are usually enough
 +
* If using OpenMP only, you'll probably want more threads but the number depends on your setup. You can use up to a node (48 threads)
 +
 +
<code>NCORE</code>:
 +
 +
For this, you need to understand each OpenMP thread needs its own CPU. This means you need to "make room" for your threads in your CPU request in PBS. So if you are using 2 threads, you only use half the CPU requested for MPI and the other half for the OpenMP parts of the code.
 +
 +
Calculating <code>NCORE</code> as above allows you to simply change the number of threads and automatically request the correct number of  CPUs for OpenMPI.

Latest revision as of 01:00, 31 March 2020

NCAR provides a detailed tutorial on how to run WRF, but note:

  • Skip the configure and compile steps of the NCAR tutorial and go straight to the Basics tab. You should have followed the NCI-specific installation instructions for building the model at NCI before attempting to run the model
  • Skip the geogrid, ungrib, and metgrid steps as these are included and compiled during the installation step
  • To run the January 2000 tutorial case do not download the metgrid data; it is available at NCI, see below
  • When the tutorial says to run the model see below for local run scripts

Data for WRF

You need to ask for membership to the sx70 project here

Geographical data

The geographical data has been redownloaded to include all the data available from NCAR except the NMM data, this includes data only available over the US.

The new complete datasets are under:

  • For V3: /g/data/sx70/data/WPS_GEOG_v3
  • For V4: /g/data/sx70/data/WPS_GEOG_v4

The previous incomplete datasets are also present to ensure you can reproduce current inputs:

  • For V3: /g/data/sx70/data/WPS_GEOG_20180313
  • For V4: /g/data/sx70/data/WPS_GEOG_20190418

Meteorological data for the tutorial

The data needed to run the January 2000 tutorial case is stored under:

  • For V3 of WRF: /g/data/sx70/data/JAN00_v3
  • For V4 of WRF: /g/data/sx70/data/JAN00_v4

Run scripts for WPS, real.exe and wrf.exe

Example scripts to submit WPS, real.exe and wrf.exe to the queues (do not try to run these on the login nodes) are provided under WPS and WRFV3/run. The files are named run_WPS.sh, run_real and run_mpi. Those scripts allow you to run WRF from any filesystem independently from the location of the source code. In particular, you can copy WRFV3/run on /scratch as well as the inputs needed for WPS and then run from /scratch (which is recommended). Those files are good to use as-is for the tutorial. For other configurations, feel free to configure those to your needs.

Running with OpenMP

For some configurations, it might be advantageous to run with OpenMP in addition to OpenMPI or OpenMP alone for some idealised cases. It is impossible to know beforehand whether OpenMP will improve or deteriorate performances. You will need to try your simulation on a short time to see the effect.

By default, the compilation is done for OpenMPI and OpenMP so you don't need to recompile. Below is an example of a mpirun command with OpenMP in bash:

export OMP_NUM_THREADS=2
NCORE=$PBS_NCPUS/$OMP_NUM_THREADS
mpirun -np $NCORE --map-by node:PE=$OMP_NUM_THREADS --rank-by core ./wrf.exe

OMP_NUM_THREADS:

indicates the number of OpenMP threads you want to use:

  • If using OpenMPI + OpenMP, then 2 OpenMP threads are usually enough
  • If using OpenMP only, you'll probably want more threads but the number depends on your setup. You can use up to a node (48 threads)

NCORE:

For this, you need to understand each OpenMP thread needs its own CPU. This means you need to "make room" for your threads in your CPU request in PBS. So if you are using 2 threads, you only use half the CPU requested for MPI and the other half for the OpenMP parts of the code.

Calculating NCORE as above allows you to simply change the number of threads and automatically request the correct number of CPUs for OpenMPI.