Difference between revisions of "CodeBreak 24/11/2021"
| Line 3: | Line 3: | ||
<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span> | <span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the </span>[https://cosima-recipes.readthedocs.io/en/latest/documented_examples/Decomposing_kinetic_energy_into_mean_and_transient.html#gallery-documented-examples-decomposing-kinetic-energy-into-mean-and-transient-ipynb <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">cosima-recipe</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">on which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).</span> | ||
| + | |||
= <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Antecedent precipitation index</span> = | = <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">Antecedent precipitation index</span> = | ||
| − | <span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <tt>dask.array.map_blocks</tt> but it can be more tricky to get it right. The following blog post | + | <span id="docs-internal-guid-fc14831d-7fff-4114-765f-ff5bb7b6b81c" style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with <tt>dask.array.map_blocks</tt> but it can be more tricky to get it right. The following blog post shows how to proceed for those 2 approaches </span>[https://climate-cms.org/posts/2021-11-24-api.html <span style="font-size:11pt; font-family:Arial; color:#1155cc; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:underline; -webkit-text-decoration-skip:none; text-decoration-skip-ink:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap">https://climate-cms.org/posts/2021-11-24-api.html</span>]<span style="font-size:11pt; font-family:Arial; color:#000000; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal; text-decoration:none; vertical-align:baseline; white-space:pre; white-space:pre-wrap"> </span> |
Latest revision as of 00:53, 30 November 2021
Eddy kinetic energy calculation
Calculation would not complete due to dask error. It was suggested to precalculate the mean of the velocity fields to reduce the size and complexity of the dask task graph. This did not solve the immediate issue, but checking the cosima-recipeon which it was based suggested this calculation requires more resources (200GB of RAM, 48 cpus) than the largest OOD instance (46GB, 16 cpus).
Antecedent precipitation index
The question was to calculate antecedent precipitation index efficiently for the entire AGCD dataset. There are several ways to proceed, one can split the calculation on different time periods and write the output for each time period to different netcdf files. It is also possible to use dask with dask.array.map_blocks but it can be more tricky to get it right. The following blog post shows how to proceed for those 2 approaches https://climate-cms.org/posts/2021-11-24-api.html
