UM Nested Models
The UM Nested Model has been developed by Stu Webster at the Met Office in order to do high-resolution limited area models. You can find more information on it in the | Met Office Wiki, this is mainly a description of how it's been set up at NCI.
Contents
Running the standard test case
Before you start modifying the model you should first check that the standard test case vadr works. This should also give you a decent overview of how the nesting system works.
Scripts and data used by the nesting system are stored on Raijin under the directory `~access/umdir/nesting`. There are subdirectories under here for different model versions.
Set up
To start setting up the model run the command
~access/umdir/nesting/vn8.5-2014-07/setup_nesting
This script creates hand edits needed for the nesting, as well as creates ancillary data files for the nested area. The ancillary files include things like orography, vegetation and emission levels for the area of interest, and are generated from high-resolution source datasets using the Met Office's Central Ancillary Program (CAP). Jobs for generating the ancillaries will be submitted to Raijin's queue. Once an ancillary job has created its file it will then submit the next job, until all the ancillary files have been created.
Wait until all the ancillary files have been created and check that all jobs have finished by running
$ qstat -u $USER
You should then copy the hand edits to Accessdev with the command
$ /short/$PROJECT/$USER/ancil.vn8.3/MEL3/um_hand_edits/latest/scp_general_hand_edits
This copies a number of hand edits needed to run the nesting.
You'll also need to set up some hand edits especially for the nested grid you're going to run. To set up the grids for the standard job vadr you should run on Accessdev
$ ~/um_nesting/vn8.5/scp_expt_hedits_lin vadr
Submitting the job
Now you'll want to go into the UMUI and to the experiment vadr. There are multiple jobs here:
- a: Global reconfiguration
- b: Global forecast
- e: 1.5 km reconfiguration
- f: 1.5 km forecast
The reconfiguration jobs get run only at the start to set up the model's reconfiguration. The forecast jobs on the other hand will run the model for 1 hour then resubmit themselves until the model reaches the end time (which is specified by the setup_nesting script). The different resolution runs are shuffled, so the global run goes first for 1 hour, then the first nest, then the second, and so on until the last nest submits the global model again for the next time interval.
Open up each of the jobs in the vadr experiment and process them. Once all of the jobs have been processed you only need to submit the a job, the nesting scripts handle the rest of the submissions automatically.
Output
The output files can be found under
/short/$PROJECT/$USER/MEL3/vadr
There are different directories here for each time interval the model ran for
There are a number of log files to check if anything goes wrong. Firstly you can find an overview of what jobs got submitted in the file (replace $TIME with the last timestamp)
/short/$PROJECT/$USER/MEL3/vadr/$TIME/vadr.out
You can also find the UM log files under
~/output
Logs from creating the boundary conditions for each nest can be found under
/short/$PROJECT/$USER/MEL3/vadr/serial_jobs
Creating your own Job
To create your own nested model you need to copy two things - the `setup_nesting` script on Raijin at `~access/umdir/nesting/vn8.5-2014-07/setup_nesting` and the UMUI job vadr.
Take a look inside the setup_nesting script. There are a number of things you can modify here, including the target area, time domain and number of nests and their resolutions.
The first thing you should change is the UMUI experiment ID. Change the line
export exptid=vadr
to point to your new copy of the UMUI experiment.
You can also change the nesting name to whatever you like (it's used for the name of the output directory). It's a good idea to use a new name for each different nest location you set up.
export regn=MEL3
Once you've edited the setup script to your liking you need to run it as well as the commands to copy over the hand-edits. If you've changed the nesting make sure all of the ancillary files have been created before you copy the hand-edits, as some information affects the hand edit values (e.g. the number of land points in a nest).
Changing the time domain
To change the start and end dates edit the values
export TIME_START=2010020112 # 1st cycle timestamp (YYYYMMDDHH)
export TIME_END=2010020115 # Last cycle timestamp
You may also need to change the initial condition file to match this date
export START_DIR=~access/data/um_nesting/vn8.5/DUMPS/ND
export START_DUMP=20100201.qwqg12.T+0
Moving the nest target area
The nesting script centres all of the nests on a single point. To change the centre point edit the values
export latitude="-37.821455"
export longitude="144.964973"
If you're working close to the poles or date line you may wish to rotate the model grid with respect to the globe.
export rotated_grid="false"
Depending on the location you're looking at the ancillary data may have a small offset to the true position. You can correct this in the ancillary files by modifying the variables
export offsetx="0"
export offsety="0"
Adding more nests
To add more nests to the model, first add a new value to the list of resolutions to make
export resolutions_to_make="1 2"
Note that this needs to be a list, not a number (e.g. "5" will create nest 5, not 5 different nests)
Create a new block of nest values and edit them to the needed values
export models[2]="km0p5"
export npts_x[2]="300"
export npts_y[2]="300"
export del_x[2]="0.0045"
export del_y[2]="0.0045"
export levs[2]="L70_40km"
export ITE[2]=no
export time_offset[2]=0
export SRTM[2]=no
The model levels value `levs` should match the name of a file in `~access/umdir/nesting/vn8.5-2014-07/hpc/set_up/vert_grid`. The "models" variable can be whatever you like as long as they're unique, it's used for file names for the different nests.
Create two new jobs in the UMUI by making a copy of an existing reconfiguration and forecast nest job. It makes things easier if the resolutions are sequential in the UMUI, e.g.
- a: Global reconfiguration
- b: Global forecast
- e: 1.5 km reconfiguration
- f: 1.5 km forecast
- g: 500 m reconfiguration (copy of 'e')
- h: 500 m forecast (copy of 'f')
In these two new jobs edit the names of the handedit files to match the new nest's name that you specified in the setup_nesting script. Under "Input/Output Control" -> "User hand edits" change the names 'km1p5_grid_h_eg', 'km1p5_grid_v_eg' and 'km1p5_time', replacing the 'km1p5' with your new grid's name.
Next you'll need to chain your new job into the list of jobs to be submitted. In the panel "Input/Output Control" -> "Script Inserts" set the following environment variables:
- JDMP: What job to use for the initial state. For a reconfiguration job this should point to the reconfiguration from the previous resolution. For a forecast job JDMP_1 should point to the reconfiguration at the same resolution, JDMP_N should point to itself (as it will initialise from the last output of the previous run).
- JLBC: What job to use for the lateral boundary conditions. This should point to the forecast from the previous resolution
- JP1: The next job to submit. For a reconfiguration job this will be the forecast at the same resolution. For a forecast job JP1_1 should point to the reconfiguration at the next resolution, JP1_N should point to the forecast at the next resolution. For the highest resolution job JP1 should point to the global forecast job to start the next cycle.
For the standard job the variables are set as:
- a
- JP1 = b
- b
- JDMP_1 = a
- JDMP_N = b
- JP1_1 = e
- JP1_N = f
- e
- JDMP = a
- JLBC = b
- JP1 = f
- f
- JDMP_1 = e
- JDMP_N = f
- JLBC = b
- JP1 = b
If you added a new 500 m nest with jobs g and h the variables would change to
- f
- JDMP_1 = e
- JDMP_N = f
- JLBC = b
- JP1_1 = g
- JP1_N = h
- g
- JDMP = e
- JLBC = f
- JP1 = h
- h
- JDMP_1 = g
- JDMP_N = h
- JLBC = f
- JP1 = b
You'll also need to make sure that boundary conditions files are being generated for all intermediate nests. In the UMUI Panel "Atmosphere -> STASH -> STASH Macros -> Makebc Macro" choose the mode "Standard Macro". This will generate the files needed by MakeBC to create the lateral boundary conditions. You may also need to alter the Time Specification section depending on the temporal resolution of the next nest.