Difference between revisions of "CodeBreak 02/03/2022"

(Created page with " = <span style="font-size:13.999999999999998pt; font-family:Arial; color:#434343; background-color:transparent; font-weight:400; font-style:normal; font-variant:normal;...")
 
 
Line 1: Line 1:
  
= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Copying global attributes from NetCDF</span> =
+
= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Installing Python Package -- </span>[https://bitbucket.org/lbl-cascade/climextremes-dev/src/master/ <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">climextRemes</span>] =
  
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">There was a question where data was converted using a python notebook and xarray, but the global attributes of the input file should be copied over to the output file.</span>
+
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">When importing climextRemes for the first time, it tries to install R libraries. This fails at some point with the error message:</span> &nbsp;
  
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The new file was saved using the line</span>&nbsp;
+
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">KeyError: 'title'</span> &nbsp;
<pre>pr.to_dataset(name='pr').to_netcdf(<filename>)</pre>
 
  
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">We recommended to do it in two steps, and add the copying of the attributes in between:</span> &nbsp;
+
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Digging deeper, it turns out that the bug is known to the developer:</span>
<pre>ds = pr.to_dataset(name='pr')
+
<blockquote>
ds.attrs = f.attrs
+
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Finally note that we recently discovered that climextremes is incompatible with recent versions of rpy2 (>= 3.4.1). While we are fixing this, if you get the error KeyError: 'title' when importing climextremes in Python, please force installation of an older rpy2.</span>
ds.to_netcddf(<filename>)</pre>
+
</blockquote>  
 +
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Pinning the version of rpy2 using the conda environment yaml file below works:</span>
  
= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Calculating differential in xarray</span> =
+
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">name: cliextremes</span>
  
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The question revolved around calculating the differential of the density (⍴) with respect to the depth (</span><span style="font-size:11pt;  font-family:'Times New Roman';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">z</span><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">). Attempting this calculation using np.gradient led to broadcasting errors. It is much easier to use an xarray aware function like </span>[https://xarray.pydata.org/en/stable/generated/xarray.DataArray.differentiate.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">xarray.differentiate</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">, which knows how to broadcast because of the named coordinates and uses the spacing in the </span><span style="font-size:11pt;  font-family:'Times New Roman';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">z</span><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">coordinate to properly calculate the differential δ⍴/δ</span><span style="font-size:11pt;  font-family:'Times New Roman';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">z</span><span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">.</span>
+
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">channels:</span>
  
= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Regression analysis on spatial data</span> =
+
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- cascade</span>
  
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The question was about the suitability of using </span>[https://xarray.pydata.org/en/stable/generated/xarray.DataArray.polyfit.html?highlight=polyfit <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">xarray.Dataset.polyfit</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">to do regression analysis on spatial data. This might not be the correct tool, as it has limited options for the fitting function. If more than a simple fit is required, it might be better to use something like </span>[https://sklearn-xarray.readthedocs.io/en/latest/auto_examples/plot_linear_regression.html <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">sklearn-xarray</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">or wrap </span>[https://stackoverflow.com/questions/62105673/applying-stats-linregress-to-every-grid-cell-in-a-netcdf-file <span style="font-size:11pt;  font-family:Arial;  color:#1155cc;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:underline;  -webkit-text-decoration-skip:none;  text-decoration-skip-ink:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">scipy.stats.lingress</span>]<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">in a function and apply this to all points.</span>
+
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- conda-forge</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:nonevertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- defaults</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">dependencies:</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- python>=3.9</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baselinewhite-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- jupyterlab</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- rpy2==3.3.6</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- pytest</span>
 +
 
 +
<span style="font-size:11pt;  font-family:'Courier New';  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baselinewhite-space:pre;  white-space:pre-wrap">&nbsp;&nbsp;- climextremes</span>
 +
 
 +
= <span style="font-size:13.999999999999998pt;  font-family:Arial;  color:#434343;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">Understand and use a Fortran program</span> =
 +
 
 +
<span style="font-size:11pt;  font-family:Arial;  color:#000000;  background-color:transparent;  font-weight:400;  font-style:normal;  font-variant:normal;  text-decoration:none;  vertical-align:baseline;  white-space:pre;  white-space:pre-wrap">The person had a Fortran program given to him to reproduce an analysis from a paper. They needed help to understand how to use the program. After having a look at it, we came to the conclusion they would need to get more information from the authors of the software. Mostly, the program was lacking information on the many arguments needed as inputs.</span>

Latest revision as of 19:03, 22 March 2022

Installing Python Package -- climextRemes

When importing climextRemes for the first time, it tries to install R libraries. This fails at some point with the error message:  

KeyError: 'title'  

Digging deeper, it turns out that the bug is known to the developer:

Finally note that we recently discovered that climextremes is incompatible with recent versions of rpy2 (>= 3.4.1). While we are fixing this, if you get the error KeyError: 'title' when importing climextremes in Python, please force installation of an older rpy2.

Pinning the version of rpy2 using the conda environment yaml file below works:

name: cliextremes

channels:

  - cascade

  - conda-forge

  - defaults

dependencies:

  - python>=3.9

  - jupyterlab

  - rpy2==3.3.6

  - pytest

  - climextremes

Understand and use a Fortran program

The person had a Fortran program given to him to reproduce an analysis from a paper. They needed help to understand how to use the program. After having a look at it, we came to the conclusion they would need to get more information from the authors of the software. Mostly, the program was lacking information on the many arguments needed as inputs.