http://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_13/10/2021&feed=atom&action=history
CodeBreak 13/10/2021 - Revision history
2024-03-28T17:04:49Z
Revision history for this page on the wiki
MediaWiki 1.31.0
http://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_13/10/2021&diff=3372&oldid=prev
C.carouge at 04:43, 20 October 2021
2021-10-20T04:43:38Z
<p></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:43, 20 October 2021</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l5" >Line 5:</td>
<td colspan="2" class="diff-lineno">Line 6:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none">Applying land masking to CMIP data</span></li></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none">Applying land masking to CMIP data</span></li></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">netCDF compression</span></li></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">netCDF compression</span></li></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del style="font-weight: bold; text-decoration: none;"><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Understanding Dask better</span></li></del></div></td><td colspan="2"> </td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Bathymetry data</span></li></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Bathymetry data</span></li></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div></ul></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div></ul></div></td></tr>
</table>
C.carouge
http://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_13/10/2021&diff=3370&oldid=prev
C.carouge: C.carouge moved page CodeBreak 13/102021 to CodeBreak 13/10/2021 without leaving a redirect
2021-10-20T04:40:07Z
<p>C.carouge moved page <a href="/index.php?title=CodeBreak_13/102021&action=edit&redlink=1" class="new" title="CodeBreak 13/102021 (page does not exist)">CodeBreak 13/102021</a> to <a href="/CodeBreak_13/10/2021" title="CodeBreak 13/10/2021">CodeBreak 13/10/2021</a> without leaving a redirect</p>
<table class="diff diff-contentalign-left" data-mw="interface">
<tr class="diff-title" lang="en">
<td colspan="1" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="1" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:40, 20 October 2021</td>
</tr><tr><td colspan="2" class="diff-notice" lang="en"><div class="mw-diff-empty">(No difference)</div>
</td></tr></table>
C.carouge
http://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_13/10/2021&diff=3368&oldid=prev
C.carouge at 04:38, 20 October 2021
2021-10-20T04:38:50Z
<p></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:38, 20 October 2021</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Summary of topics =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Summary of topics =</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none<del class="diffchange diffchange-inline">;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap</del>">Applying land masking to CMIP data</span></li></div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none">Applying land masking to CMIP data</span></li></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">netCDF compression</span></li></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">netCDF compression</span></li></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Understanding Dask better</span></li></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><li><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Understanding Dask better</span></li></div></td></tr>
</table>
C.carouge
http://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_13/10/2021&diff=3367&oldid=prev
C.carouge: Created page with " = Summary of topics = <ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"> <li><span style="font-size:11pt; font-family:Arial; color:#000000; background-co..."
2021-10-20T04:37:50Z
<p>Created page with " = Summary of topics = <ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"> <li><span style="font-size:11pt; font-family:Arial; color:#000000; background-co..."</p>
<p><b>New page</b></p><div><br />
= Summary of topics =<br />
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"><br />
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Applying land masking to CMIP data</span></li><br />
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">netCDF compression</span></li><br />
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Understanding Dask better</span></li><br />
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Bathymetry data</span></li><br />
</ul><br />
<br />
= Land masking =<br />
<br />
<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Requirement: compare CMIP6 outputs for several models and sometimes mask the data over the ocean. CMIP6 data does not come with a land-mask data, but it comes with a land fraction field, sftlf. To use it as a mask over the ocean (resp. land), multiply the field by sftlf (resp. 1.-sftlf). Then mask the points where sftlf is 0. (resp. 1.) using a </span><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">.where()</span><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">to get NaNs over the masked area.</span> &nbsp; <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">When comparing several models on different grids, it can be a complicated problem as the masks are different between several models. There are several ways to go around this:</span><br />
<ul style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"><br />
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Mask the original data then regrid. Then, only keep the data on points where all the models are non-NaNs.</span></li><br />
<li><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Regrid the land fraction from one model and apply it to all models.</span></li><br />
</ul><br />
<br />
<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Both solutions will create some approximation. Considering both masked and unmasked data needed to be on the common grid, the second solution seems the most straightforward.</span><br />
<br />
= NetCDF compression =<br />
<br />
<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Some NetCDF files created using a custom python program produced very large (16GB) files. These files were not compressed.</span> &nbsp; <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Closer inspection with </span><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">ncdump -hs</span><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">shows that some dimension fields are compressed, but the actual data field is not. Attempts to compress after the fact with </span><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">nccompress</span><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">fail: nc2nc does not work as the files are in NETCDF4 format, and </span><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">nccopy</span><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">fails with a chunking error, presumably it gets confused because some fields are already compressed.</span><br/> <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">We’ve shown the blog post at </span>[https://climate-cms.org/2018/10/12/create-netcdf.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">https://climate-cms.org/2018/10/12/create-netcdf.html</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">which explains how to create compressed NetCDF files with python, and settled on a solution to use python scripts in a dedicated queued job with sufficient memory to read the old file, and write a </span> <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">new one with the </span><span style="font-size:11pt; font-family:'Courier New'; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">encoding={field:{‘shuffle’: True, ‘zlib’: True, ‘complevel’: 5}}</span><span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">argument. This will take a few minutes per file.</span> &nbsp;<br />
<br />
= Bathymetry Data =<br />
<br />
<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">A follow up from last session: projecting a South-polar stereo bathymetry with coordinates in metres to latitude/longitude. That was solved satisfactorily, but the next required step is to create a new bathymetry in rectilinear lat/lon coordinates on the same grid as the existing ACCESS-OM2 0.1 degree model. This requires some work and follow-up offline.</span><br/> &nbsp;</div>
C.carouge