http://climate-cms.wikis.unsw.edu.au/index.php?action=history&feed=atom&title=CodeBreak_24%2F11%2F2021CodeBreak 24/11/2021 - Revision history2026-10-15T12:30:25ZRevision history for this page on the wikiMediaWiki 1.31.0http://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_24/11/2021&diff=3395&oldid=prevC.carouge at 04:53, 30 November 20212021-11-30T04:53:10Z<p></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:53, 30 November 2021</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l3" >Line 3:</td>
<td colspan="2" class="diff-lineno">Line 3:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span></div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Antecedent precipitation index</span> =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Antecedent precipitation index</span> =</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <tt>dask.array.map_blocks</tt> but it can be more tricky to get it right. The following blog post <del class="diffchange diffchange-inline">show </del>how to proceed for those 2 approaches </span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;</span></div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <tt>dask.array.map_blocks</tt> but it can be more tricky to get it right. The following blog post <ins class="diffchange diffchange-inline">shows </ins>how to proceed for those 2 approaches </span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;</span></div></td></tr>
</table>C.carougehttp://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_24/11/2021&diff=3394&oldid=prevC.carouge at 04:52, 30 November 20212021-11-30T04:52:56Z<p></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:52, 30 November 2021</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l3" >Line 3:</td>
<td colspan="2" class="diff-lineno">Line 3:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span></div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span></div></td></tr>
<tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Antecedent precipitation index</span> =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Antecedent precipitation index</span> =</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <tt>dask.array.map_blocks</tt> but it can be more tricky to get it right. The following blog post show how to proceed for those 2 approaches</span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;</span></div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <tt>dask.array.map_blocks</tt> but it can be more tricky to get it right. The following blog post show how to proceed for those 2 approaches </span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;</span></div></td></tr>
</table>C.carougehttp://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_24/11/2021&diff=3393&oldid=prevC.carouge at 04:52, 30 November 20212021-11-30T04:52:27Z<p></p>
<table class="diff diff-contentalign-left" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 04:52, 30 November 2021</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l6" >Line 6:</td>
<td colspan="2" class="diff-lineno">Line 6:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Antecedent precipitation index</span> =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Antecedent precipitation index</span> =</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <<del class="diffchange diffchange-inline">code</del>>dask.array.map_blocks</<del class="diffchange diffchange-inline">code</del>> but it can be more tricky to get it right. The following blog post show how to proceed for those 2 approaches<del class="diffchange diffchange-inline"></span><span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap"></del></span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;</span></div></td><td class='diff-marker'>+</td><td style="color: #222; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt; <ins class="diffchange diffchange-inline"> </ins>font-family:Arial; <ins class="diffchange diffchange-inline"> </ins>color:#000000; <ins class="diffchange diffchange-inline"> </ins>background-color:transparent; <ins class="diffchange diffchange-inline"> </ins>font-weight:400; <ins class="diffchange diffchange-inline"> </ins>font-style:normal; <ins class="diffchange diffchange-inline"> </ins>font-variant:normal; <ins class="diffchange diffchange-inline"> </ins>text-decoration:none; <ins class="diffchange diffchange-inline"> </ins>vertical-align:baseline; <ins class="diffchange diffchange-inline"> </ins>white-space:pre; <ins class="diffchange diffchange-inline"> </ins>white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <<ins class="diffchange diffchange-inline">tt</ins>>dask.array.map_blocks</<ins class="diffchange diffchange-inline">tt</ins>> but it can be more tricky to get it right. The following blog post show how to proceed for those 2 approaches</span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;</span></div></td></tr>
</table>C.carougehttp://climate-cms.wikis.unsw.edu.au/index.php?title=CodeBreak_24/11/2021&diff=3392&oldid=prevC.carouge: Created page with " = <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal;..."2021-11-30T04:51:50Z<p>Created page with " = <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal;..."</p>
<p><b>New page</b></p><div><br />
= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Eddy kinetic energy calculation</span> =<br />
<br />
<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span><br />
<br />
= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Antecedent precipitation index</span> =<br />
<br />
<span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <code>dask.array.map_blocks</code> but it can be more tricky to get it right. The following blog post show how to proceed for those 2 approaches</span><span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap"></span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">&nbsp;</span></div>C.carouge