How to run WRF
NCAR provides a detailed tutorial on how to run WRF, but note:
- Skip the configure and compile steps of the NCAR tutorial and go straight to the Basics tab. You should have followed the NCI-specific installation instructions for building the model at NCI before attempting to run the model
- Skip the geogrid, ungrib, and metgrid steps as these are included and compiled during the installation step
- To run the January 2000 tutorial case do not download the metgrid data; it is available at NCI, see below
- When the tutorial says to run the model see below for local run scripts
Data for WRF
You need to ask for membership to the sx70 project here
The geographical data has been redownloaded to include all the data available from NCAR except the NMM data, this includes data only available over the US.
The new complete datasets are under:
- For V3: /g/data/sx70/data/WPS_GEOG_v3
- For V4: /g/data/sx70/data/WPS_GEOG_v4
The previous incomplete datasets are also present to ensure you can reproduce current inputs:
- For V3: /g/data/sx70/data/WPS_GEOG_20180313
- For V4: /g/data/sx70/data/WPS_GEOG_20190418
Meteorological data for the tutorial
The data needed to run the January 2000 tutorial case is stored under:
- For V3 of WRF: /g/data/sx70/data/JAN00_v3
- For V4 of WRF: /g/data/sx70/data/JAN00_v4
Run scripts for WPS, real.exe and wrf.exe
Example scripts to submit WPS, real.exe and wrf.exe to the queues (do not try to run these on the login nodes) are provided under WPS and WRFV3/run. The files are named run_WPS.sh, run_real and run_mpi. Those scripts allow you to run WRF from any filesystem independently from the location of the source code. In particular, you can copy WRFV3/run on /scratch as well as the inputs needed for WPS and then run from /scratch (which is recommended). Those files are good to use as-is for the tutorial. For other configurations, feel free to configure those to your needs.
Running with OpenMP
For some configurations, it might be advantageous to run with OpenMP in addition to OpenMPI or OpenMP alone for some idealised cases. It is impossible to know beforehand whether OpenMP will improve or deteriorate performances. You will need to try your simulation on a short time to see the effect.
By default, the compilation is done for OpenMPI and OpenMP so you don't need to recompile. Below is an example of a mpirun command with OpenMP in bash:
export OMP_NUM_THREADS=2 NCORE=$PBS_NCPUS/$OMP_NUM_THREADS mpirun -np $NCORE --map-by node:PE=$OMP_NUM_THREADS --rank-by core ./wrf.exe
indicates the number of OpenMP threads you want to use:
- If using OpenMPI + OpenMP, then 2 OpenMP threads are usually enough
- If using OpenMP only, you'll probably want more threads but the number depends on your setup. You can use up to a node (48 threads)
For this, you need to understand each OpenMP thread needs its own CPU. This means you need to "make room" for your threads in your CPU request in PBS. So if you are using 2 threads, you only use half the CPU requested for MPI and the other half for the OpenMP parts of the code.
NCORE as above allows you to simply change the number of threads and automatically request the correct number of CPUs for OpenMPI.