CodeBreak 08/12/2021 - Revision history

C.carouge: Created page with " =
2021-12-15T01:00:38Z

Created page with " = <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal;..."

New page

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Trajectory Filtering</span> =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">With a source dataset of trajectories identifying the numbered region a particle is in at each point of time, identify which particles are moving consistently from unique region to region and which are circling back and forth between a small number of regions.</span>

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The most efficient way will be to use array operations, e.g. find transitions between regions with <tt>diff = region[1:] - region[:-1]</tt>. Count of total regions entered will then be <tt>diff.where(diff != 0).count()</tt>. Explored in more detail </span>[https://climate-cms.org/posts/2021-12-08-trajectories.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">on the CMS blog</span>]

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Fortran code strange behaviour</span> =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Some inherited Fortran Code seemed to modify the content of an array depending on what index one starts to output of the array. We were not able to resolve this. The code is of questionable coding practices with a lot of implicit typing, questionable scoping, and complete lack of indentation. More in-depth analysis is needed that cannot be completed within the CodeBreak time.</span>

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Old Python code crash because of package not updated</span> =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">An old Python code was unable to locate a required module, `global_land_mask` when run in the environment `conda/analysis3` and `conda/analysis3-unstable`. However it worked in `conda/analysis3-21.04`. It turns out that the library had been installed by the user as it is not a conda package and was located in `.local/lib/python3.8`. Since the Python version has been upgraded since analysis3-21.04, the user installed package needs to be reinstalled by the user. </span>

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Using ERA5 data as input for HySplit</span> =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">HySplit, a particle trajectory code, needs a very specific format for its input meteorological data. On Github, we found </span>[https://github.com/sunt05/hysplit_metdata <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">this repository</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">with code to prepare ERA5 data in GRIB to the HySplit input format. The requestor wanted to use their own Windows computer but this didn’t work well. We’ll follow up with instructions to install and use on Gadi.</span>

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Cosima time slicing</span> =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">When opening a dataset with Cosima cookbook, the dates specified aren't exact - the cookbook will return to the nearest file boundary, so if a file is containing a decade of data and you ask for 1994 to 1996, the cookbook will return a dataset covering 1990 to 2000 since that's what's in the file. You can then get the exact period you need with <tt>.sel(time=slice('1994','1996'))</tt></span>

= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Understanding xarray, Dask and chunking </span> =

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Starting from an example script provided by Ram, we reviewed the basics of accessing big datasets. In particular:</span>
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">You can use <tt>ncdump -sh <filename></tt> to find out how the original files are chunked  </span></li>
</ul>

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The -h option shows the file header, adding -s shows the “hidden” attributes, like chunks, compression and more details on the file format</span>
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;">
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Trying to use chunks which are aligned to the way the files are chunked is important</span></li>
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Consider how you will be accessing the data in the analysis, if you are averaging along the entire time axis and you have only 1 timestep per chunk (as it is often true for climate data) then you will need to load the entire variable for each grid point.</span></li>
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Keeping chunks size around 200MB </span></li>
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Use of a cluster to run in parallel with dask</span></li>
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Use vectorise operations as much as you can. I.e. don’t use a loop to execute an operation on the entire variable.</span></li>
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Make use of lazy computation by dask. When running in parallel, Dask won’t compute what you write, it only computes the way to do it. So it’s ok to write operations that might be too big for the memory knowing it will only be computed one subset at a time.</span></li>
</ul>

<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">To see more on this </span>[https://coecms-training.github.io/parallel/dask-intro.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">https://coecms-training.github.io/parallel/dask-intro.html</span>]  

<span style="font-size:11pt; font-family:Arial; color:#3c4043; background-color:#ffffff; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">We also discussed useful commands for jobs on the PBS scheduler: <tt>uqstat</tt> and <tt>qcost</tt>. Those are included in the nci_scripts module on hh5. See the information on </span>[https://github.com/coecms/nci_scripts <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:#ffffff; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">the nci_scripts repository</span>]<span style="font-size:11pt; font-family:Arial; color:#3c4043; background-color:#ffffff; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">.</span>