http://climate-cms.wikis.unsw.edu.au/api.php?action=feedcontributions&user=Paolautas&feedformat=atomclimate-cms wikis.unsw.edu.au - User contributions [en]2024-03-28T13:51:09ZUser contributionsMediaWiki 1.31.0http://climate-cms.wikis.unsw.edu.au/index.php?title=File:Application_JRA_Data_Access.pdf&diff=191File:Application JRA Data Access.pdf2019-01-27T15:09:16Z<p>Paolautas: </p>
<hr />
<div></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=File:Porva2.html&diff=158File:Porva2.html2019-01-27T15:08:08Z<p>Paolautas: </p>
<hr />
<div></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=File:Slide1.jpg&diff=134File:Slide1.jpg2019-01-27T15:07:35Z<p>Paolautas: </p>
<hr />
<div></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=File:Extract_from-Giovanni_news&diff=124File:Extract from-Giovanni news2019-01-27T15:06:52Z<p>Paolautas: </p>
<hr />
<div></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=File:ARCCSS_data_perspectives-21st_Jan_2014.docx&diff=118File:ARCCSS data perspectives-21st Jan 2014.docx2019-01-27T15:06:36Z<p>Paolautas: </p>
<hr />
<div></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ERA5&diff=137ERA52019-01-21T20:28:59Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>ERA5 is a climate reanalysis dataset, covering the period 1950 to present. ERA5 is being developed through the [http://climate.copernicus.eu/ | Copernicus Climate Change Service (C3S)].<br />
Data processing for ERA5 is carried out by [http://www.ecmwf.int/ | ECMWF], using ECMWFS' [https://www.ecmwf.int/en/research/modelling-and-prediction | Earth System model IFS], cycle 41r2. The name ERA refers to 'ECMWF ReAnalysis', with ERA5 being the fifth major global reanalysis produced by ECMWF (after FGGE, ERA-15, ERA-40, ERA-Interim). The next reanalysis, ERA6, is planned for around 2020.<br />
<br />
We are providing a small subset of ERA5 while waiting for a longer term solution to store and manage this big dataset.<br />
Currently data is available from 2000 to present.<br />
ERA5 comprises different data stream, we have only a small subset of the surface operative analysis. For the moment we are not considering downloading model or pressure level data as the size of these are quite big and we wouldn't be able to provide enough data to use them to run a model.<br />
For more information on ERA5 data refer to the [https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation | ECMWF website].<br />
<br />
If you need other variables for your current research then you can contact us at climate_help@nci.org.au<br />
===NB this is a temporary fix so we will download only variables which are useful for a variety of projects and that are needed now, if you can use ERA-Interim instead or you will be using ERA5 in the future but not in the next few months, then we won't consider your request.=== <br />
<br />
===Requests constraints=== <br />
<br />
<span class="s1">We will only accept requests for:</span><br />
<span class="s1"> - data on regular 0.25 by 0.25 lat/lon grid</span><br />
<span class="s1"> - global region for surface variables </span><br />
<span class="s1">- Austral-Asian region for pressure level. The region includes Australia and the Maritime continent and the CORDEX domain and is defined as:</span><br />
<span class="s1">Latitude: 20 to -57</span><br />
<span class="s1">Longitude: 78 to 220</span><br />
<span class="s1"> NB longitude for the global region is -180 to 180</span><br />
<span class="s1"> - analysis data and no ensemble data (analysis includes surface forecast fields such as precipitation). It is currently impossible to provide input data for a climate model</span><br />
<span class="s1">You can submit a request for downloading following these criteria. We will only consider requests that specify the exact code/stream etc. You can refer for more information to the documentation here and to the <span class="s3">[https://protect-au.mimecast.com/s/-dp6CZYM5DFXmX6BIzaRdX?domain=apps.ecmwf.int | ERA5 catalogue]</span>. Be as specific as possible in your request: grib or netcdf format? which forecast steps do you need if you request forecast data? (see ERA5 website) </span><br />
<br />
===Known Issues=== <br />
[https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation#ERA5datadocumentation-Knownissues | Known issues from ECMWF website]<br />
<br />
Dec 7 2018: an error has been discovered affecting the values of the following variables<br />
{| <br />
| '''CDS Dataset''' || [https://protect-au.mimecast.com/s/hEqUClxwAVHDmEV9F9ILGT?domain=cds.climate.copernicus.eu | ERA5 hourly data on pressure levels from 2000 to present] <br />
|-<br />
| '''Parameters affected''' || U-component of wind, V-component of wind <br />
|-<br />
| '''Level''' || All pressure levels <br />
|-<br />
| '''Time periods affected''' || 01 Jan. 2000 - 31 Dec. 2008<br>01 Jun. 2018 - present <br />
|-<br />
|}<br />
This problem was caused by the computation of U/V from Vorticity (Vo) and Divergence (D) already interpolated onto a regular grid in CDS. U/V should have been computed from Vo and D in the spherical harmonics representation (the native model representation) and then interpolated to a regular grid before saving to the climate data store.<br />
<br />
This error has yet to be fixed, which means that any data corresponding to the description above is wrong, and will continue to be wrong until a fix is in place. You should not use this data and wait until the problem is fixed.<br />
<br />
The problem is now fixed and we are in the process of re-downloading the data.<br />
From the CDS communication:<br />
"<span class="s1">During our investigations, we found that wind values are far too low on pressure levels at the poles in CDS, which has been documented as a <span class="s2">[https://protect-au.mimecast.com/s/vf55CQnzVqt4vDL7hxh02-?domain=confluence.ecmwf.int | known issue]</span>. Please use winds at neighbouring locations at 89.75 (N/S) and 89.5 (N/S) for Reanalysis and Ensemble members winds instead of 90 (N/S)."</span><br />
<br />
===License=== <br />
<span style="background-color: #ffffff; color: #172b4d; font-family: -apple-system,system-ui,&#39;Segoe UI&#39;,Roboto,Oxygen,Ubuntu,&#39;Fira Sans&#39;,&#39;Droid Sans&#39;,&#39;Helvetica Neue&#39;,sans-serif; font-size: 14px;">ERA5 data is open access and free to download for all uses, including commercial use.</span><br />
As for ERAI and other ECMWF datasets, though, you need to have an [https://apps.ecmwf.int/registration/ | account with ECMWF] and accept their terms and conditions, in this particular case ERA5 is covered by the [http://apps.ecmwf.int/datasets/licences/copernicus/ | Copernicus license]<br />
<br />
===ERA5 on raijin=== <br />
<br />
===='''Surface'''==== <br />
data is downloaded as netcdf on a regular 0.25x0.25 longitude/latitude global grid at 1 hr resolution, as available from the Copernicus server. These are the maximum spatial and temporal resolutions possible for this dataset.<br />
<br />
/g/data/ub4/era5/netcdf/surface/<variable>/<yr>/<files><br />
<br />
<span class="s1">filename:</span><br />
<variable><span class="s1">_era5_global_<from-date>_<to-date>.nc</span><br />
<br />
===='''Pressure levels'''==== <br />
data is downloaded as netcdf on a regular 0.25x0.25 longitude/latitude grid at 1 hr resolution, on an Australian region defined as in the criteria.<br />
<br />
/g/data/ub4/era5/netcdf/pressure/<variable>/<yr>/<files><br />
<br />
<span class="s1">filename:</span><br />
<variable><span class="s1">_era5_aus_<from-date>_<to-date>.nc</span><br />
<br />
<span class="s1">NB variable names are as in the ECMWF param tables, our ERAI netcdf files were following where possible the CMIP conventions</span><br />
<br />
===='''Wave surface level'''==== <br />
data is downloaded as netcdf on a regular 0.5x0.5 longitude/latitude grid at 1 hr resolution, on a global grid.<br />
<br />
/g/data/ub4/era5/netcdf/wave/<variable>/<yr>/<files><br />
<br />
<span class="s1">filename:</span><br />
<variable><span class="s1">_era5_global_<from-date>_<to-date>.nc</span><br />
<br />
===='''Requested variables'''==== <br />
<br />
[https://docs.google.com/spreadsheets/d/1CVc2xYYhgQOTN9eb5qS2HfFUa2ubuqZ0Ox6Ws3Y5KTQ/edit?usp=sharing | ERA5 variable list]<br />
<br />
===Note on forecasts=== <br />
<span class="s1">Forecasts are accumulated values defined at 20 steps (0-19) starting from times 06.00 and 18.00.</span><br />
<br />
<span class="s1">The accumulations are over the hour ending at the forecast step which is a different behaviour from era interim.</span><br />
<span class="s1">The 0 step is always zero because it would be the accumulation from timestep and timestep +0 hours (I.e. 06.00 to 06.00)</span><br />
<br />
<span class="s1">The others would be for example for precipitation:</span><br />
<span class="s1"> Time 06.00 Step 1: forecast of accumulated precipitation between 06.00-07.00</span><br />
<span class="s1"> Time 06.00 Step 19: forecast of accumulated precipitation between 06.00-01.00+1 day</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Data_FAQ&diff=120Data FAQ2019-01-17T05:44:32Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>==='''<span style="color: #1e49e0;">Why the CMIP5 data on Raijin has such a complicated directory structure?</span>'''=== <br />
<br />
The CMIP5 dataset is overall a Pb sized dataset contributed by as many as 60 different modelling groups, hence it is inherently complicated to organise. As well because the modelling groups didn’t stick to the rules and did their own thing. Unfortunately, the guidelines in regard to versioning the dataset were not sufficiently detailed and so they've been interpreted differently by different groups. When the climate community started downloading data on Raijin it was decided that the only way to keep track of the dataset "version" was to re-create their DRS as in the web server (thredds) which are unique. We also always download the originally published dataset and no replicas from other nodes.<br />
When a web server reaches its capacity then the new datasets are published from a new server and this means that you could have a different root for the same model, sometimes for the same experiment.<br />
Currently NCI is re-downloading the latest versions of CMIP5 non-Australian data into a more coherent directory structure. the new replicated data is stored in the al33 group. Refer to their [https://opus.nci.org.au/display/CMIP/CMIP+Community+Home | climate community page] for information and updates.<br />
<br />
===<span style="color: #1e49e0;">Do CMIP5 variables coming from the same simulations have the same version number?</span>=== <br />
<br />
<span class="s1">This question is difficult to answer and it is a really sore point with CMIP5, hopefully, they are implementing changes so it won't happen again in CMIP6.</span><br />
<span class="s1">There's no way to be 100% sure that two versions come out of the same simulation. The versioning instructions where quite unclear and interpreted differently by different modelling groups. In the last couple of years, it occurred to few groups (for example GFDL) to add a "simulation_id" to their attributes but this is the exception rather than the norm. I'd like to assume that same version means the same simulation, but having a different version really just means that the group of variables has been published later, or maybe that one of them was calculated wrongly in the post-processing and has been re-published under a new version.</span><br />
<span class="s1">A completely different simulation with a different configuration, initialization etc should have a different ensemble code so anything with r1i1p1, for example, should come from the same run, even if part of it or its post-processing might have been updated.</span><br />
<span class="s1">More information might be available directly from the modelling groups.</span><br />
<span class="s1">Another cautionary approach could be if you find a few of the variables you need which have a more recent version, you can send an e-mail to [[mailto:climate_help@nci.org.au | climate_help]] and we will check if everything you need it is up to date. Users request only what they need and it's very well possible that someone updated just part of an ensemble.</span><br />
<br />
===<span style="color: #1e49e0;">Using the ARCCSS DMPonline is not useful for me as I don't use NCI servers.</span>=== <br />
<br />
<span class="s1">Using DMPonline is really independent of NCI, while it contains some specific information on NCI systems since they are common to many in the Centre, it is more about data management in general, regardless which hardware you use. Even if you are using your laptop it is good to have a data workflow, a plan of what you will be doing at different stages of your research. For example, to publish an RDA record for your data with us, you would now fill one of these plans, and the advantage would be that you can easily export that as a document which you can always adapt and re-use.</span><br />
<br />
<span class="s1">DMP will be compulsory for universities and ARC grants, publishing your data is already compulsory for most journals. Plus, at CMS we really want to hear from users that are not using NCI, users we don't normally hear from. So we get a better idea of what everybody in the Centre is doing and we can create new training resources or support all our users in a better way.</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Data_resources&diff=119Data resources2019-01-17T05:32:28Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>This page contains documentation about data relevant to the ARCCSS data climate data and how to access data available at NCI and on web-based climate portals.<br />
<br />
===[[DATA NEWS]]=== <br />
Only the most recent appear here, follow the link to the news page to see all of them<br />
<br />
- The ARC has added a new [http://www.arc.gov.au/research-data-management | Research Data Management] page to their website<br />
<br />
- CMIP5 update: most of the ESGF nodes are back online, still missing are NCI and the Chinese nodes.<br />
<br />
- CMIP5: the [http://arccssive.readthedocs.org/en/latest/CMIP5.html#examples | ARCCSSive] module has now a new stable version and the database has been updated to include all the data replicated on raijin, both by users and by NCI as a bulk transfer.<br />
<br />
- There is a new [https://opus.nci.org.au/display/CMIP/CMIP+Community+Home | CMIP community page] on NCI confluence, for information and updates relating to the CMIP data and activities at NCI. This includes regular updates on CMIP6 progress. Anyone <span class="s1">with an NCI login can access and add comments to the pages.</span> This is also the right way to communicate issues with the data and its management on NCI.<br />
<br />
===Data policies and services within the Centre=== <br />
* What data should I publish - Centre position statement:[[Data Management Plan | Preparing a Data Management Plan]]<br />
** [[:File:ARCCSS data perspectives-21st Jan 2014.docx]]<br />
* [[Data induction | Data management induction]]<br />
* [[Data publishing guidelines | How to publish data when submitting a paper]]<br />
* [[mailto:paolap@utas.edu.au | Who to ask for help]]<br />
* [http://dmp.climate-cms.org:3000/ | ARCCSS DMPonline] - an online data management plan tool<br />
* [[storage-request | Requesting extra disk allocation]]<br />
* [[Researcher-ID | Getting a researcher ID]]<br />
<br />
===University data policies and services=== <br />
* [[UNSW data requirements and tools | UNSW]]<br />
* [[Melbourne University data requirements and tools | Melbourne University]]<br />
* [[UTAS data requirements and tools | UTAS]]<br />
* [[ANU data requirements and tools | ANU]]<br />
* [[Monash data requirements and tools | Monash]]<br />
<br />
===Datasets hosted on raijin and managed by the ARCCSS=== <br />
* [[CMIP | CMIP5]] - Coupled Model Intercomparison Project Phase 5 data on raijin<br />
* [[CMIP | CMIP6]] - Coupled Model Intercomparison Project Phase 6 data on raijin<br />
* [[ERA INTERIM]] - ECMWF re-analysis on raijin<br />
* [[ERA5]] - ECMWF re-analysis<br />
* [[MACC]] - ECMWF<br />
* [[YOTC]] - ECMWF re-analysis on raijin<br />
* [[CABLE Data | CABLE]] datasets collection (in collaboration with the CABLE users group)<br />
* [[OSTIA-SST]]<br />
* [[NOAA_OISST]] - Optimum Interpolation Sea Surface Temperature from NOAA<br />
* CMIP5_ocean_processing<br />
* [http://polar.ncep.noaa.gov/sst/ | NCEP Polar SST]<br />
* [[OFES]] - OGCM For the Earth Simulator<br />
* [[MERRA2]] - Modern-Era Retrospective Analysis for Research and Applications 2<br />
* [[JRA55]] - Japanese 55-year Reanalysis<br />
* [[JRA55-do]] - JRA‐55 based data set for Driving Ocean ‐ sea ice model<br />
* [[Upper Air Sounding Observations for Australia 2000-2015]]<br />
* [[Lightning Stroke Counts on ECMWF Era Interim Grid | Lightning Stroke Counts ]]on ECMWF Era Interim Grid 20080301-20151031<br />
* [[NASA-TRMM | NASA-TRMM]] - Real-Time TRMM Multi-Satellite Precipitation Analysis<br />
* [[C20C]]+ - International CLIVAR C20C+ Detection and Attribution project<br />
* [[CESM1-LME]] - CESM1 Last Millenium Ensemble<br />
* [[CESM1-CAM5-BGC-LE]] - CESM1-CAM5 BioGeoChemestry 20C + RCP8.5 Large Ensemble model output<br />
* [[sealevel_GLO_PHY_L4_REP_observations_008_047]] - Global Ocean - Multimission altimeter satellite gridded sea surface heights and derived variables (previously known as AVISO)<br />
* [[CMORPH | CMORPH ]]-<br />
* [[GSMaP | GSMaP - Global Satellite Mapping of Precipitation]]<br />
* [[GHCN]] - Global Historical Climatology Network<br />
<br />
===Other datasets hosted on raijin=== <br />
* [http://nci.org.au/data-collections/data-collections/ | NCI collections]<br />
* [http://geonetwork.nci.org.au/geonetwork/srv/eng/main.home | NCI geodata network records]<br />
* [http://dap.nci.org.au/thredds/catalog.html | NCI Thredds catalog]<br />
<br />
===ARCCSS datasets and software published on Research Data Australia (RDA)=== <br />
<br />
The Centre of Excellence for Climate System Science has started publishing its datasets on Research Data Australia (RDA), the Australian National Data Service (ANDS) metadata repository. The first datasets to be published were from the Climate Model Downscaling Data for Impacts Research (CliMDDIR), then the ACCESS CMIP5 simulations.<br />
* [[ACCESS_CoE_simulations | ACCESS]] - CMIP5 simulations<br />
* [http://climddir.org/ | CliMDDIR] - Climate Model Downscaling Data for Impacts Research<br />
* [[ARCCSS published datasets | ARCCSS collection]] - ARCCSS datasets on the NCI Data Catalogue<br />
* [https://researchdata.ands.org.au/atmospheric-access13-historical-sub-project/645160 | C20C+ ACCESS] - <span style="color: #454545;">Atmospheric ACCESS1.3 historical all forcing model output for the Climate of the 20th Century Plus (C20C+) Detection and Attribution sub-project</span><br />
* <span style="color: #454545;">[https://researchdata.ands.org.au/marine-heatwaves-detection-code/814983 | marineHeatWaves] - Marine heatwaves detection code</span><br />
<br />
===External data resources=== <br />
<br />
* [[GIOVANNI]] - GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure<br />
* [http://dl.tpac.org.au | TPAC] Climate and Ocean data portal<br />
* [http://www.ands.org.au | The Australian National Data Service (ANDS)]<br />
* [http://researchdata.ands.org.au | Research Data Australia]<br />
* [http://cfconventions.org/ | CF metadata conventions website]<br />
* [http://www.bom.gov.au/climate/data/ | BoM Climate data services]<br />
* [http://www.bom.gov.au/cyclone/history/tracks/ | BoM Tropical Cyclone data services]<br />
* [https://tropicaldatahub.org/ | Tropical Data Hub]<br />
[[Category:Main Menu]]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CMIP&diff=80CMIP2019-01-17T05:31:24Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>Coupled Model Intercomparison Phase: CMIP5 and CMIP6<br />
<br />
The information for CMIP data is available from the <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">NCI [https://opus.nci.org.au/display/CMIP/CMIP+Community+Home | CMIP community space] (see NCI communication below)</span><br />
<br />
You will find there:<br />
* how the data is organised on raijin<br />
* links to the official documentation<br />
* how to access the dataset<br />
* how to cite the data<br />
* available CMIP relative tools, services and training<br />
This is a "community" resource so you can help in making it more useful by adding comments and feedback.<br />
<br />
===Communication from NCI for planned switchover to new CMIP community website:=== <br />
<br />
<span class="s1">The CMIP Community website has been refreshed for CMIP6 and NCI is ready to reveal the changes. '''The planned switchover date is 22nd November'''.</span><br />
<span class="s1">''What will change?''</span><br />
<span class="s2">The website <span class="s3">[[https://opus.nci.org.au/display/CMIP/CMIP+Community+Home]]</span> will be updated</span><br />
* <span class="s1">If you have used bookmarks to access pages within that space, you will need to update to the new area.</span><br />
* <span class="s1">If you can’t find what you’re looking for then just email <span class="s4">[[mailto:help@nci.org.au | help@nci.org.au]]</span> and they will get back to you.</span><br />
<span class="s1">Note that you’ll still need to be logged in with your NCI credentials to access the full content available.</span><br />
<span class="s1">Features of the new site include the latest updates on CMIP6 and information on tools you can use to search variables and how to check what’s in the queue for download on our NCI ESGF service.</span><br />
<span class="s1">NCI will continue to update and add information to this website so your feedback will help them keep this up to date as a useful community resource.</span><br />
<span class="s1">Further emails about preparation for CMIP6 will follow in the coming weeks to CMIP groups members (al33, rr3, oi10 and ua6), but feel free to contact them if you have any questions.</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CLEx_induction&diff=56CLEx induction2019-01-17T05:28:33Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>The CMS team is here to help you with the technical aspects of your work. A lot of you (probably all) will have to work with computer code and software, usually in a Linux/Unix environment. You might also need to access and/or share large amounts of data, work on different servers, etc.<br />
Some of you will be somewhat familiar with these tasks whereas they will be new to others.<br />
We are here to help whatever your technical knowledge is. We are happy to answer any question, we can't necessarily do everything but feel free to ask as we might have ideas or we might know who you should contact and what to exactly ask.<br />
<br />
The members of the CMS team are:<br />
{| <br />
! Name !! Home-based Institution !! Email <br />
|-<br />
| Claire Carouge || ANU || c.carouge@unsw.edu.au <br />
|-<br />
| Aidan Heerdegen || ANU || aidan.heerdegen@anu.edu.au <br />
|-<br />
| Paola Petrelli || U Tasmania || paola.petrelli@utas.edu.au <br />
|-<br />
| Scott Wales || U Melbourne || scott.wales@unimelb.edu.au <br />
|-<br />
| Holger Wolff || Monash || holger.wolff@monash.edu <br />
|-<br />
| Danny Eisenberg || UNSW || d.eisenberg@unsw.edu.au <br />
|-<br />
|}<br />
(Note: there is no typo, Claire is sitting at ANU but has a UNSW email address).<br />
Although we won't ignore emails sent to our individual addresses, our preferred way of contact is through our help desk email address: [[mailto:climate_help@nf.nci.org.au | climate_help@nf.nci.org.au]]. Contacting the help desk will make sure your email is seen by one of us in case the person at your institution is away for example. Don't worry about the help desk email being an NCI account, you can still ask any question there.<br />
=Step 1: Get an account with NCI= <br />
----<br />
NCI operates the computers that you will be doing most of your work on. NCI provides services to the centre, they are not part of the Centre of Excellence.<br />
<br />
To get access to NCI servers, you need to do three things:<br />
# <span style="line-height: 1.5;">You need to find out which project(s) you need to be connected to. Your project decides who gets billed for what you do on the NCI Servers. Your supervisor should be able to tell you which project(s) you should get connected to. Be mindful that there are two types of projects: projects for computation and projects for datasets. The mapping of the computational projects for CLEX can be found [[CLEX projects at NCI | here]]. We recommend to join all the projects relevant to your research project but to choose 1 default project to work from. The same page with the list of computational projects explains how to set your default project. The projects for datasets do not have any computational time added to them so you need to make sure you ask for at least one computation project. Your supervisor should know. Note if you will run the ACCESS model or the UM model, you need to ask a connection to the "access" project in addition to at least one computation project.</span><br />
# <span style="line-height: 1.5;">You need to register with NCI. This can be done on [https://my.nci.org.au/mancini/signup/0 | NCI registration page] You will need to supply some information, and read and accept their policies. In order to be added to the system, you need to request connection to at least one project and be accepted on the project. Note that this can take a bit of time as the Lead chief investigator of the project has receive and confirm your request via email. Note this is an automatic email process, you don't need to send an email to the lead chief investigator. Although you might want to do so if you think you need to introduce yourself to him/her. Note that you should be able to request connection to several projects while getting signed up.</span><br />
# <span style="line-height: 1.5;">You might want to ask your supervisor whether you also should be connected to any of the following data projects. These don’t provide any computing allowances, but they give access to certain datasets or models that you might need.</span><br />
{| <br />
! Project !! Purpose !! Administrator <br />
|-<br />
|''' ua8 ||''' access to ARCCSS published data, GSWP3, NCEP Polar sst, [[OSTIA-SST | ostia sst]], CMIP5 ocean processing ||= CMS <br />
|-<br />
|''' rq7 ||''' [[YOTC | ECMWF Year of Tropical Convection re-analysis]] ||= CMS <br />
|-<br />
|''' rq5 ||''' OFES - OGCM for the Earth Simulator ocean re-analysis ||= CMS <br />
|-<br />
|''' ub4 ||''' [[ERA INTERIM | ECMWF ERA Interim re-analysis 6 hrs data]] ||= CMS <br />
|-<br />
|''' al33 ||''' CMIP3, [[CMIP | CMIP5]] replica data ||= NCI <br />
|-<br />
|''' rr3 ||''' [[CMIP | CMIP5]] published data ||= NCI <br />
|-<br />
|''' oi10 ||''' [[CMIP | CMIP6]] replica data ||= NCI <br />
|-<br />
|''' rr7 ||''' BoM re-analysis collection including ERA-Interim monthly data and JRA55, MERRA2 ||= BoM <br />
|-<br />
| <span style'''"display: block; text-align: center;">cable</span> ||''' Give access to the [https://trac.nci.org.au/trac/cable/wiki | CABLE model] and<br><span style'''"display: block; text-align: center;">[[CABLE Data | CABLE benchmarking dataset]]</span>. Please read the [https://trac.nci.org.au/trac/cable/wiki/CableRegistration | CABLE registration] information. ||''' CAWCR <br />
|-<br />
|''' access ||''' Give access to the [[Unified Model | ACCESS model]] ||= CMS - CAWCR <br />
|-<br />
|}<br />
<br />
=Step 2: Set up your Connection= <br />
----<br />
<span style="line-height: 1.5;">The standard method of connecting to the NCI systems is SSH. We want you to use a passphrase protected private key authorisation method with an ssh-agent and agent forwarding for convenience. If you understood every part of that sentence, go ahead and set it up. If not, the next part describes what to do.</span><br />
<br />
The instructions describe how to set up your connection to both raijin.nci.org.au and accessdev.nci.org.au. You '''can not''' set up a connection to accessdev if you are not going to use the ACCESS or UM models. So please only follow the instructions you need!<br />
==Linux and MacOS:== <br />
<br />
# <span style="line-height: 1.5;">'''Create the ssh directory''' on your computer</span><br />
> <syntaxhighlight lang=bash><br />
$ mkdir -p ~/.ssh<br />
</syntaxhighlight><br />
# '''Create a config file'''<br />
> <syntaxhighlight lang=bash><br />
$ nano ~/.ssh/config<br />
</syntaxhighlight><br />
> Enter the following contents, replacing <span style="font-family:monospace">xx0000</span> with your NCI user name.<br />
> <syntaxhighlight><br />
Host raijin<br />
HostName raijin.nci.org.au<br />
User xx0000<br />
ForwardX11 yes<br />
ForwardX11Trusted yes<br />
Host access<br />
HostName accessdev.nci.org.au<br />
User xx0000<br />
ForwardX11 yes<br />
ForwardX11Trusted yes<br />
ForwardAgent yes<br />
</syntaxhighlight><br />
# '''Create a key pair.''' It is imperative that you select a strong passphrase when asked for it.<br />
> <syntaxhighlight lang=bash><br />
$ ssh-keygen -t rsa<br />
</syntaxhighlight><br />
# '''Distribute the key pair.''' The two commands below will each need your NCI password once.<br />
## for raijin:<br />
>> <syntaxhighlight lang=bash><br />
$ cat ~/.ssh/id_rsa.pub | ssh raijin "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"<br />
</syntaxhighlight><br />
## for accessdev:<br />
>> <syntaxhighlight lang=bash><br />
$ cat ~/.ssh/id_rsa.pub | ssh access "mkdir -p ~/.ssh/; cat >> ~/.ssh/authorized_keys"<br />
</syntaxhighlight><br />
On Linux/Unix systems, there is sometimes a short-hand for these lines, called <span style="font-family:monospace">ssh-copy-id}}, and used just like this: {{ssh-copy-id raijin</span>. But this program is not available on all systems, particularly not on MacOS.<br />
# '''Test whether you have an agent.''' If you have an agent already running, this command will ask for the passphrase for the just-created key pair<br />
> <syntaxhighlight lang=bash><br />
$ ssh-add<br />
</syntaxhighlight><br />
# If the last command told you that it couldn’t open a connection, then you don’t have an agent. Come talk to someone of the CMS team, we will help you set it up.<br />
==Windows== <br />
Windows is not Unix based, so it doesn’t come with standard SSH programs. The most popular SSH program for Windows is PuTTY, available [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html | here].<br />
You will need at least PuTTY, Pageant (which is the agent), and PuTTYgen, the key generator. A nice video on how to set it up is on [https://youtu.be/2nkAQ9M6ZF8 | YouTube].<br />
Windows also doesn't come with an X11 Server, which is needed to display graphical user interfaces. At this point in time, we suggest something like [http://www.straightrunning.com/XmingNotes/ | Xming]. The last free version of Xming came out in 2007, this version should be sufficient. If you really want to, you can 'donate' GBP10 (ca. $20) to get an up-to-date version of Xming.<br />
<br />
Another option is to install [https://cygwin.com/ | Cygwin], which is a large collection of linux utilities that run on windows. Cygwin includes a shell, SSH, and an X11 server, as well as many other useful tools.<br />
<br />
==SSH-keys for file transfers== <br />
Additionally, some groups transfer data from NCI to outside NCI, for example for storage on University maintained servers. For these transfers, the best way to proceed is to create a restricted ssh key pair used only for file transfer. The setup is explained on this [http://nf.nci.org.au/facilities/software/SSH/ssh-without-passwds.php | NCI page].<br />
This page has not been updated since the Vayu machine. Most of the instructions are still valid, except for step 3. The restricted command prefix should now be:<br />
<syntaxhighlight><br />
from="chipmunk*.nci.org.au,r-dm*.nci.org.au,gopher*.nci.org.au,raijin*.nci.org.au",command="~/bin/rrsync /data/archive",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa<br />
</syntaxhighlight><br />
<br />
=Step 3: Basics of the NCI system= <br />
----<br />
<span style="line-height: 1.5;">We are mainly using 2 systems of the NCI: raijin and accessdev. Raijin is the supercomputer that we run our models on. Accessdev is a virtual server that we have software installed that are required to allow the running the models. Both servers run Linux as operating system.</span><br />
==Read the manual!== <br />
Considering you will be using systems that are provided by the NCI, you should take the time to read the documentation provided by them about their systems. That is everything on their [https://opus.nci.org.au/display/Help/NCI+Help | help knowledge base].<br />
==Additional notes== <br />
===Modules:=== <br />
The servers have a lot of software installed, some of which they have multiple versions of. If you need to use a software, you usually will need to load a certain module. A full help of the module command is accessible through the man pages on Raijin. Here are the most important commands regarding the modules:<br />
{| <br />
| $ module <span style="color: #8000ff;">help</span> || Show help about the module system <br />
|-<br />
| $ module <span style="color: #8000ff;">avail</span> || List all available modules (a lot!) <br />
|-<br />
| $ module <span style="color: #8000ff;">list</span> || List the currently loaded modules <br />
|-<br />
| $ module <span style="color: #8000ff;">load</span> intel-fc || Load the default version of the intel fortran compiler <br />
|-<br />
| $ module <span style="color: #8000ff;">load</span> intel-fc/14.1.106 || Load a specific version of the intel fortran compiler <br />
|-<br />
| $ module <span style="color: #8000ff;">switch</span> intel-fc/13.4.183 || Switch to a different version of the intel fortran compiler (unload and load) <br />
|-<br />
| $ module <span style="color: #8000ff;">unload</span> intel-fc/13.4.183 || Unload the specified version of the intel fortran compiler <br />
|-<br />
| $ module <span style="color: #8000ff;">use</span> ~access/modules || Add our Centre's curated modules to the list of available modules <br />
|-<br />
|}<br />
Some modules conflict with one another, for example different versions of the same software. You might have to unload one module first to load a different one.<br />
<br />
===PBS:=== <br />
The other important system you need to understand in order to work on raijin is the scheduler PBS. This is the system for running software on dedicated nodes inside raijin. Basically, a user tells PBS that they want to run this job on this many CPUs with these resources for at least so long, and PBS places it in a queue, and when the required resources are available, it will assign this job to them.<br />
<br />
Again, NCI has a very good documentation about how to run this [https://opus.nci.org.au/display/Help/Job+Submission+and+Scheduling | here]<br />
<br />
The most important commands are:<br />
{| <br />
| qsub || Submit a job to the queue <br />
|-<br />
| qstat<br>nqstat || Display the status of the queued jobs <br />
|-<br />
| qdel || Remove a job from the queue <br />
|-<br />
|}<br />
Of these, qsub is the most important, and there are so many parameters that we won’t tell them all here.<br />
<br />
In general, the fewer resources you request, the earlier your job will run. But if your program exceeds the requested resources at any time, PBS will kill the job, and you have to start over.<br />
<br />
More specifically:<br />
* <span style="line-height: 1.5;">Raijin is comprised of nodes with 16 CPUs each. If you request less than 16 CPUs, you will get the number of CPUs that you request. If you request more, you will get as many nodes as you need. So it is usually a good idea for jobs with more than 16 CPUs to use a multiple of 16 CPUs.</span><br />
* Nodes come with between 32GB and 126GB of memory to share within the 16 CPUs. So if your job requires 16 or more CPUs, you might as well request just under 32GB per node, for example 95GB for a 48-cpu job.<br />
* If you request more than 32GB per 16 CPUs, your job will possibly have to wait longer for enough high-memory nodes to become available.<br />
<br />
=Step 4: How to run models= <br />
----<br />
A lot of information to gain access to models or run the models can be found on the ARCCSS CMS wiki (see link at the end).<br />
<br />
A very important message from the CMS team:<br />
<span style="background-color: #fdf988; display: block; text-align: center;">Often you will work with source code. Please ensure that your version of the source code is under version control! We use SVN and git (usually hosted on github) for the model codes and smaller utilities.</span><span style="background-color: #fdf988; display: block; text-align: center;">'''DO NOT JUST COPY SOURCE CODE! CHECK IT OUT, OR CHECK IT IN!'''</span><br />
==UM== <br />
The Unified model is currently transitioning from the old user environment umui to the new rose/cylc user interface. Either way, you need to log into accessdev.nci.org.au.<br />
<br />
===umui(x)=== <br />
The old user interface has to be used for versions up to 7.x, and can be used for 8.x. umui was developed by the UM, umuix is very similar, but more convenient developed by CSIRO. An introduction into its use is on our wiki: http://climate-cms.unsw.wikispaces.net/Introduction+to+UMUI<br />
<br />
===rose/cylc=== <br />
The new user interface can be used from version 8.x, and has to be used for versions 9.0 and later.<br />
<br />
==WRF== <br />
The CMS team ports the WRF model on Raijin. The latest versions can be found under /projects/WRF on Raijin. From WRF v3.6.0 onwards, the WRF codes under /projects are managed under Git. Information on using the model is on the CMS wiki. Users should also be aware that we do not reproduce the information from the [http://www2.mmm.ucar.edu/wrf/users/ | WRF model users website] which stays a very important resource for all users. In particular, every new user is strongly encouraged to run through the official tutorials (found under User Support on the WRF model users site).<br />
<br />
==CABLE== <br />
CABLE is under licence so you need to request access to the code. All the information you need to get access and use CABLE is on the [https://trac.nci.org.au/trac/cable | CABLE trac site]<br />
<br />
=Data management= <br />
----<br />
Most of the information on how the data is managed is stored in this wiki in the [[Data | Data Services]] page. Here you can find information on the datasets managed by the Center as well the relevant policies in regard to data.<br />
Take some time to get familiar with it, here are some suggestions:<br />
# Read the Center data policy and how to prepare a [[Data Management Plan | data management plan]]. You can use the [http://dmp.climate-cms.org:3000/ | ARCCSS DMPonline tool] to do this. Because this tool stores your plan into a database, you can save it and access it as many times as you want, which means you can build your data management plan as you progress with your research. It is good though to start as soon as you can, DMPonline is structured as to teach about data practices and the resources available to you, it also informs the CMS team on which data, software and/or infrastructure you are planning to use, so we can better tailor our efforts to your needs.<br />
# Get familiar with [[Data publishing guidelines | publishing requirements]], while this might seem faraway, it is a good idea to keep track from now of what you are doing in regard to data and metadata, as it is often much more difficult to retrieve this information at the end of your project. There are also some technical times involved in getting data properly published.<br />
# Consider creating a [[Researcher-ID | researcher-ID]] to uniquely identify your work.<br />
# Make sure you browse through all the available data sources before you download data, probably even better send us an e-mail to the climate_help to make sure you are not wasting time downloading something which might be already available, or that we could download and manage for you<br />
# Become familiar with the different filesystems available on any server you are using, make sure you know which are meant to be working spaces, where you can archive your data, how to share data with collaborators and best practices to transfer data within and out of the system. If you are using raijin on NCI you can find this information on their [http://nci.org.au/nci-systems/national-facility/data-storage/filesystems/ | user support ]and training sections.<br />
# You can explore a lot of data related information in the [http://ands.org.au | Australian National Data Services] website<br />
# Don't hesitate in contacting climate_help for any question you might have, or also e-mail directly paola.petrelli@utas.edu.au, if you have a more general question or you are confused about data management. We are always available to discuss with you the details of your project and give you advice.<br />
<br />
=How to learn more= <br />
----<br />
==General setup:== <br />
a blog post by a UNSW's PhD student on a few things to know and learn: [http://web.maths.unsw.edu.au/~oangelil/blog/blog2/page_blog2.html | Oliver Angélil's post]<br />
<br />
==CMS:== <br />
this [[home | wiki]]<br />
<br />
==NCI:== <br />
[https://opus.nci.org.au/display/Help/NCI+Help | Guides and reference pages]<br />
[http://nci.org.au/services-support/training/ | Training material] and [https://training.nci.org.au/ | this]<br />
<br />
==Version Control with git:== <br />
(Note: git might also be very useful for your thesis!)<br />
[https://www.youtube.com/playlist?list=PLg7s6cbtAD165JTRsXh8ofwRw0PqUnkVH | Git video tutorial by GitHub]<br />
<br />
==Data management:== <br />
[[Data induction]]<br />
<br />
=Getting help= <br />
----<br />
Your first port of call for help should be the CMS team via the help-desk: [[mailto:climate_help@nf.nci.org.au | climate_help@nf.nci.org.au]]. This help desk is followed by all the members of the CMS team and by some staff from NCI.</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ERA_INTERIM&diff=135ERA INTERIM2018-12-18T21:18:29Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>=====<span style="background-color: #ffffff;">A subset of the ERA Interim re-analysis data ranging from 1979 to date are now hosted on the RDSI fast disk storage at NCI. </span><span style="background-color: #ffffff; font-size: 1em; line-height: 1.5;">This subset is currently managed by the ARCCSS.</span>===== <br />
<span style="background-color: #ffffff; font-size: 1em; line-height: 1.5;">The original grib data is downloaded from the [http://apps.ecmwf.int/datasets/data/interim_full_daily | ECMWF server] , it is regularly updated and specific variables are extracted and converted to monthly netcdf files.</span><br />
For more information on the original ECMWF collection visit their website: [https://confluence.ecmwf.int/display/CKB/What+is+ERA-Interim | ERA-Interim page]<br />
<br />
===Known issues=== <br />
<br />
There are several known quality issues for the ERA Interim data, here is a list of web resources:<br />
<br />
[http://old.ecmwf.int/research/era/do/get/index/QualityIssues | ERA Interim quality issues from ECMWF website]<br />
[https://climatedataguide.ucar.edu/climate-data/era-interim | Climate Data Guide page on ERAI]<br />
[http://reanalyses.org/atmosphere/overview-current-reanalyses | Overview and re-analysis comparison from reanalyses.org]<br />
<br />
Some of these analysis are described more in depth in the following article: [http://onlinelibrary.wiley.com/doi/10.1002/qj.828/full | The ERA-Interim reanalysis: configuration and performance of the data assimilation system]<br />
<br />
We are listing all the issues we are aware of in this[[ERAI-issues | ERAI known issues page]], please feel free to contribute, we just provide the data as it is downloaded from the ECMWF server, we do checks to make sure that our data is the same as the original but we cannot run checks for the original data.<br />
<br />
We found very high values for temperature (~330-350 K) near the lower troposphere in the following periods:<br />
26-31 Dec 2002 and 21-25 Dec 2003. Other variables might be similarly affected.<br />
<br />
===Data access=== <br />
<br />
<span style="background-color: #ffffff;">To comply with the ECMWF licensing you need to be registered with the ECMWF to use this data.</span><br />
<span style="background-color: #ffffff;">To register with the ECMWF you have to </span><span style="background-color: #ffffff; line-height: 1.5;">sign the </span>[http://apps.ecmwf.int/datasets/data/interim_full_daily/licence/ | ERA Interim license and agreement]<span style="background-color: #ffffff; line-height: 1.5;">.</span><br />
<span style="background-color: #ffffff; line-height: 1.5;">You can then request to join the ub4 project by using your NCI account to access </span>[https://my.nci.org.au/ | My NCI portal] , and you can send <span style="background-color: #ffffff; line-height: 1.5;">an e-mail to </span><span style="line-height: 1.5;">[[mailto:climate_help@nf.nci.org.au | climate_help@nf.nci.org.au]] to let us know you have registered. Remember you have to register with ECMWF before.</span><br />
<span style="color: #ed0622;">Be aware that by requesting to be part of the ub4 project you are automatically agreeing to the ECMWF license agreement terms</span>'''<span style="background-color: #ffffff;">.</span>'''<br />
<br />
<span style="background-color: #ffffff;">The data is available on raijin.nci.org.au under /g/data1/ub4/erai/grib for the grib version and /g/data1/ub4/erai/netcdf for the netcdf version.</span><br />
<br />
**<span style="color: #1134e8; font-size: 120%;">[https://docs.google.com/spreadsheets/d/1qnQC_Ki5IAwZPD9viV79tfenemPGoYWKfDa5vEwDl90/pubhtml | Data Inventory]</span>''': what's available and what we are downloading, regularly updated.<br />
NB this is a new version of the inventory, includes more specific information on available variables.<br />
Please note that we download all the available surface fields, both analysis and forecast, but we don't convert all of them to netcdf, unless they are specifically requested.<br />
Just recently we added:<br />
three surface forecast fields: r<span style="background-color: #ffffff; font-family: arial,sans,sans-serif;">unoff, surface thermal radiation downwards, convective available potential energy (CAPE);</span><br />
<span style="background-color: #ffffff; font-family: arial,sans,sans-serif;">three surface analysis fields: total cloud cover, high cloud cover, low cloud cover</span><br />
<br />
<span style="background-color: #ffffff;">The netcdf version is organized in monthly field files while the grib files have all the fields in one file.</span><br />
<br />
<span style="background-color: #ffffff;">The current netcdf version was released in April 2015, the previous version is still available in the ua8 project. If you request access before we moved the data to ub4, you are already part of ub4 and don't need to request access again to use the data. The older version is not updated anymore and we'll be kept there only for users who are at the ned of their project. In January 2016 we'll definitely delete the older version and anyone is encouraged to move to ub4, since this version is updated and is a better data product overall.</span><br />
<br />
**<span style="background-color: #ffffff;">Monthly averaged fields</span>'''<br />
We stopped updating the monthly fields since the BoM has a copy of the same in their rr7 project. You can request access as for the other NCI projects on <span style="line-height: 1.5;">[https://my.nci.org.au/ | My NCI portal] . Anyone working with the ARCCSS should be granted access, if you are having trouble getting access or the data you are looking for is not available there, let us know by e-mailing climate_help@nci.org.au .</span><br />
<br />
===<span style="background-color: #ffffff;">ERAI netcdf v1.0</span>=== <br />
<br />
<span class="s1">The main differences with the previous version, which is still available in the ua8 project, are:</span><br />
<br />
<span class="s1">1) netcdf4 format with compression, which allowed us to get rid of scale and offset</span><br />
<br />
<span class="s1">2) new filenames of the form <varname>_<frequency>_ERAI_historical_<level>_<from-date>_<to-date>.nc</span><br />
<br />
<span class="s1"> Ex. : ta_6hrs_ERAI_historical_an-ml_20110101_20110131.nc</span><br />
<br />
<span class="s1">3) some of the variable names have changed to the correspondent CMIP5 standard name, all names are now lower cases</span><br />
<br />
<span class="s1">4) new directory structure /g/data1/ub4/erai/netcdf/<frequency>/<realm>/<level>/<version>/<variable>/files…</span><br />
<br />
<span class="s1"> Ex. : /g/data1/ub4/erai/netcdf/6hr/atmos/oper_an_ml/v01/ta/</span><br />
<br />
<span class="s1">5) there is more metadata information in the files, in particular we adopted CF conventions and added standard_names wherever possible. We also added an attribute MD5 checksum for each variable, this can be use to check for data corruption by using ncks or nco</span><br />
<br />
<span class="s1">The reason for the changes in filenames and directory structure is to make both as much as possible compliant to the CMIP5 filename and DRS standards, so the entire collection could be accessible from the CWSlab. This should be tested and implemented in the next few months.</span><br />
<br />
<span class="s1">Most of the data will be in the "atmos" realm, with the exception of ERAI land which is is "land" and few ocean fields. And most files have "6hr" frequency, with the exception of the forecast data which has "3hr" frequency. </span><br />
<br />
<span class="s1">We will soon add documentation on the exact procedure we use to create the netcdf files from the original grib.</span><br />
<br />
===<span style="background-color: #ffffff;">Old version on ua8</span>=== <br />
<br />
<span style="background-color: #ffffff;">The two slides below from a presentation given at the 1st ARCCSS winter school show the filenames and directory structure for the old version:</span><br />
<br />
<span style="background-color: #ffffff; line-height: 0px; overflow: hidden;">[[File:Slide1.jpg|648x486px]]</span><br />
<br />
<span style="background-color: #ffffff;">'''NB''': Users of the ECMWF data sets are requested to reference the source of the data in any publication, e.g. "''ECMWF ERA-Interim data used in this study/project have been provided by ECMWF/have been obtained from the ECMWF Data Server''".</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Useful_resources&diff=372Useful resources2018-11-02T02:41:55Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><br />
This place is to create a library of links to external resources that might be useful to the Centre's staff and students.<br />
=Data services= <br />
----<br />
[https://earthdata.nasa.gov/ | NASA Earthdata]: this is a portal to NASA datasets. The User Resources tab also have a list of recorded [https://earthdata.nasa.gov/user-resources/webinars-and-tutorials | webinars] around the use of these datasets.<br />
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[[GIOVANNI | GIOVANNI]] - GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure</span><br />
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[https://dl.tpac.org.au/tpacportal/ | TPAC] Climate and Ocean data portal</span><br />
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[http://www.ands.org.au/ | The Australian National Data Service (ANDS)]</span><br />
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">[http://researchdata.ands.org.au/ | Research Data Australia]</span><br />
* <span style="background-color: #ffffff; color: #000000;">[http://cfconventions.org/ | CF metadata conventions website]</span><br />
* <span style="background-color: #ffffff; color: #000000;">[http://www.bom.gov.au/climate/data/ | BoM Climate data services]</span><br />
* <span style="background-color: #ffffff; color: #000000;">[https://terranova.org.au/ | Terra Nova] - The Australian Climate Change Adaptation Information Hub</span><br />
<br />
=Tools= <br />
----<br />
<span style="background-color: #ffffff; line-height: 1.5;">[http://www.marvl.org.au/ | MARVL - Marine virtual Laboratory] : </span><span style="font-family: Arial,Helvetica,sans-serif; line-height: 1.5;">The Australian Marine Virtual Laboratory (MARVL) is a new development in modelling frameworks for researchers in Australia, co-funded by NeCTAR ([http://www.nectar.org.au/ | www.nectar.org.au]). MARVL allows a non-specialist user to configure and run a model, automating many of the modelling preparation steps needed to bring the researcher faster to the stage of simulation and analysis. The tool is seen as enhancing the efficiency of researchers and marine managers, and is being considered as a educational aid in teaching. Most researchers, students and academic staff with have access to MARVL via their [http://aaf.edu.au/ | AAF](Australian Access Federation) identity, in most cases this is your normal login identity at your local university, institute or federal agency.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">[http://www.unidata.ucar.edu/blogs/news/entry/metpy_an_open_source_python | MetPy] : </span>MetPy is an Open Source project aimed at providing a Pythonic library for meteorological data analysis that meshes well with the rest of the scientific Python ecosystem.<br />
<br />
=Tutorials and other tips and tricks= <br />
----<br />
[http://web.maths.unsw.edu.au/~oangelil/blog/blog2/page_blog2.html | help to start]: a blog post written by Oliver Angélil, a PhD student at the CCRC. It goes through a few practical tips and tricks to help with what to know when starting working in the field of climate science.<br />
<br />
CMS blog: [[https://climate-cms.org]]<br />
<br />
CMS training: [https://docs.google.com/presentation/d/13eCaZCLwh48-3MHg8ks5O3kXTc3Wz4WbkdB0ZqZnZZM/edit?usp=sharing | Publishing]<br />
<br />
CMS training: [https://docs.google.com/presentation/d/1EclbLy3x28F6u821P7_XJT7cfBQpanIo4BxSpJhHcAM/edit?usp=sharing | Storage management] and a separate document with[https://docs.google.com/document/d/17gpAuinpf5DKVhCXABAwgtbwGBzmdgi1diJiZVHyZrc/edit?usp=sharing | links for the training]<br />
<br />
CMS training: [https://docs.google.com/presentation/d/1Bl1SI_v3s-kgqs96FYROO--JhDw_Trujb5TYnxhBaD8/edit#slide=id.g2739569a98_1_58 | Shell tips and tricks]<br />
<br />
CMS training: [https://docs.google.com/presentation/d/1Rvn7a8Y4ghz0SVprwyIN6KHrXqpOYvw2GMcdwFriFEk/edit | Debugging 101]<br />
<br />
CMS training: [https://docs.google.com/presentation/d/1n3b_gRNhb164MQNrFYv0oj12tUCqSnaWgSEzAZNWBL8/edit#slide=id.p | Debugging - Run time checks]<br />
<br />
CMS training: [https://github.com/ScottWales/training | Xarray dask training GitHub]<br />
<br />
CMS training: [http://%20https:''youtu.be/ya-xXY5uzeo | Introduction to 3D visualisation with ParaView] by Martin Jucker (youtube video)<br />
<br />
CMS training: [https://youtu.be/QXIj1nYwGcY | Introduction to CleF: Climate Finder a tool to discover ESGF datasets] (youtube video)<br />
<br />
CMS training: [https://www.youtube.com/watch?v=fHtLbMmgoMY&feature=youtu.be | Python: transitioning from 2 to 3]<br />
<br />
cheat sheets:<br />
* [https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing | Tape ARchive - tar]<br />
* [https://docs.google.com/document/d/1pGgVAoEjErHTn4joiCXbdnZDppEXwq9RcNofedrViz4/edit?usp=sharing | Access Control Lists - acls]<br />
* [https://docs.google.com/document/d/1ugmBLX385YCuEj1Rd1MEyzC85cZEfE6sDzbfEYiv8ZM/edit?usp=sharing | NCI accounting:] data#_report_files, short_report_files, lquota, ncimonitor and mdssdiff<br />
=Professional development= <br />
----<br />
Data science: [https://www.dataquest.io | Dataquest] and [https://www.dataquest.io/blog | its blog].</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Publishing_options&diff=126Publishing options2018-09-17T01:29:35Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>Recently journal editors have updated their data policy and now require that data relating to the submitted paper should be made available by the authors. From the [http://publications.agu.org/author-resource-center/publication-policies/data-policy/ | AGU data policy] : "..all data necessary to understand, evaluate, replicate, and build upon the reported research must be made available and accessible whenever possible ..."<br />
. The aim of this change in the policy is to satisfy the principle that someone reading the paper should be able to reproduce your experiment. Again form the AGU policy :<br />
" For the purposes of this policy, data include, but are not limited to, the following:<br />
* <span style="font-family: Arial,Helvetica,sans-serif;">Data used to generate, or be displayed in, figures, graphs, plots, videos, animations, or tables in a paper.</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif;">New protocols or methods used to generate the data in a paper.</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif;">New code/computer software used to generate results or analyses reported in the paper.</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif;">Derived data products reported or described in a paper. "</span><br />
<br />
There can be practical and even copyright limitations to do this, but these can be taken into account and it should not be an impediment to publication if properly documented.<br />
The JGR-Space Physics editor-in-chief has listed some of these challenges and clarified the scope of the policy on his [https://liemohnjgrspace.wordpress.com/category/publication-policy/ | blog]. This include references to model data which apply to lots of the ARCCSS data as well.<br />
<br />
===How to publish=== <br />
<br />
NCI is now providing web services to publish data and metadata. This include a [https://geonetwork.nci.org.au | geonetwork catalogue] to describe your dates (i.e. a metadata repository) and provide links to other description and to the dataset access point. Once you have a geonetwork record, NCI can mint a DOI for the dataset, as for papers a DOI makes the dataset easy to cite.<br />
The files can be made accessible to the public by using their [http://dap.nci.org.au/thredds/remoteCatalogService?catalog=http:''dapds00.nci.org.au/thredds/catalogs/ua8/catalog.xml | TDS catalogue] or thredds. <br />
We also create a collection record on Research Data Australia (RDA), a metadata catalogue service provided by ANDS. We do this because RDA has more visibility then the NCI catalogue, including being harvested by the new [https://toolbox.google.com/datasetsearch | google dataset search toolbox]. All these records are referencing each other so no matter where a user find your record it will always get to the same DOI, data access point and information.<br />
<br />
There a few necessary steps to make your data available:<br />
# Create a data management plan for your collection using the [http://144.6.225.151:3000 | ARCCSS DMPonline] tool, if you don't have one already. If you do make sure to have filled in the third phase of the plan which deals with the publishing details. You then should share the plan with me when you're ready (share with paola.petrelli@utas.edu.au). I use the dmp to collect the necessary information to create a metadata record on RDA on your behalf. You can look at one of [https://researchdata.ands.org.au/access13b-model-output-experiment-v10/453996 | ARCCSS records] as an example of the kind of information required.<br />
# We will use the information on the plan to automatically generate the NCI and RDA records and a directory in /g/data/ua8/ARCCSS_Data/<your-dataset> which will contained also a draft for a readme file and the license attached to the data.<br />
# move the data to the your dataset directory: /g/data/ua8/ARCCSS_Data/<your-dataset>/tmp . <br />
# get the dataset to a good quality level, necessary to share it successfully: metadata in the files should satisfy [http://cfconventions.org/ | CF conventions], there should be a READ-ME file or some data description sitting with the data and both directory structure and filenames should be understandable and contains information on the data. It sounds more difficult that normally it is, once you moved the data I'll have a look and tell you if there's anything which needs to be done.<br />
# Once the files are ready we will ask NCI to run their QA/QC checker on them, if they pass these quality checks the dataset is added to the thredds catalogue and the metadata records are updated.<br />
Other advice:<br />
# Think carefully about the license and rights terms, you'll find some options on the form itself, contact me if there are other collaborators involved or any other special terms to be kept into account, the license will be virtually null if either the ARCCSS and/or your university don't hold the copyright.<br />
# if you don't have one already to make a ORCID <span class="s1">[[http://orcid.org/]]</span> identity which we can list in the record as well. In the future you might be able to reference all your work through a researcher identity, when applying for a grant.<br />
# give some thought to what you want to publish, as well as satisfying the journal requirements, you want to make sure to include anything that could be useful to other researchers. This will increase the value of your data and potentially get more people to cite you. Look [[which data publish | here]] for guidelines.<br />
<br />
Managing your data is an essential part of the publishing process for more detailed information go to the '''[[Data induction | Data management induction training]]'''<br />
<br />
I''''m still working on this page and the DMPonline tool, any feedback on both is welcome!'''<br />
Look also to the other wiki pages under data services dedicated to the tool, data management in general and researcher identities for more information</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CMEMS_-_Sealevel_GLO_PHY_L4_observations_008_047&diff=311CMEMS - Sealevel GLO PHY L4 observations 008 0472018-09-06T04:41:05Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>==='''<span style="color: #3f3f41; font-family: Arial,Helvetica,sans-serif;">Sealevel_GLO_PHY_L4_REP_observations_008_047</span>'''=== <br />
<br />
This dataset is a satellite global sea level product downloaded from the Copernicus Marine Environment Monitoring Service (CMEMS).<br />
For more information and the official documentation, including the product manual refer to the [http://marine.copernicus.eu/services-portfolio/access-to-products/?option=com_csw&view=details&product_id=SEALEVEL_GLO_PHY_L4_REP_OBSERVATIONS_008_047 | CMEMS data portal]<span style="color: #3f3f41; font-family: Arial,Helvetica,sans-serif;">.</span><br />
<span style="color: #3f3f41; font-family: Arial,Helvetica,sans-serif;">The following brief description is copied over from the original website.</span><br />
<span style="color: #3f3f41; font-family: Arial,Helvetica,sans-serif;"> For the Global Ocean - Multimission altimeter satellite gridded sea surface heights and derived variables computed with respect to a twenty-year mean. Previously distributed by Aviso+, no change in the scientific content. All the missions are homogenized with respect to a reference mission which is currently OSTM/Jason-2. The sla is computed with an optimal and centered computation time window (6 weeks before and after the date). Two kinds of datasets are proposed: filtered (nominal dataset) and unfiltered.</span><br />
<br />
Resolution is 0.25X0.25 degrees and has an irregular temporal resolution. The data starts in 1993 and it's ongoing.<br />
<br />
===Terms of use=== <br />
<br />
**<span style="color: #d40b44;">NB </span>'''You need to register with CMEMS to use this product:<br />
[[http://marine.copernicus.eu/web/56-user-registration-form.php]]<br />
The full terms are listed on the CMEMS website:<br />
[http://marine.copernicus.eu/services-portfolio/service-commitments-and-licence/#licence | http:''marine.copernicus.eu/services-portfolio/service-commitments-and-licence/#licence]<br />
<br />
In particular:<br />
<span style="background-color: #ffffff; color: #4b4b4d; display: block;">... In case of any publication, the Licensee will ensure credit the Copernicus Marine Service in the following manner:</span><span style="color: #000080; display: block;">“This study has been conducted using E.U. Copernicus Marine Service Information” ...</span><span style="color: #000080; display: block;"><br />
</span><br />
===On raijin=== <br />
<br />
This dataset is available in the ua8 project:<br />
<br />
/g/data/ua8/CMEMS_SeaLevel/<version>/<year>/<files><br />
<br />
v4-0 is the most recent version<br />
<br />
Filenames are:<br />
<br />
dt_global_allsat_phy_l4_YYYYMMDD_<production_date>.nc<br />
<br />
Variables:<br />
<br />
* sea_surface_height_above_sea_level (SSH)<br />
* surface_geostrophic_eastward_sea_water_velocity_assuming_sea_level_for_geoid (UVG)<br />
* surface_geostrophic_northward_sea_water_velocity_assuming_sea_level_for_geoid (UVG)<br />
* sea_surface_height_above_geoid (SSH)<br />
* surface_geostrophic_eastward_sea_water_velocity (UVG)<br />
* surface_geostrophic_northward_sea_water_velocity (UVG)</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=JRA55&diff=192JRA552018-08-09T01:24:34Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>'''<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Description</span>'''<br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">The Japan Meteorological Agency (JMA) conducted JRA-55, the second Japanese global atmospheric reanalysis project. It covers 55 years, extending back to 1958, coinciding with the establishment of the global radiosonde observing system. Compared to its predecessor, JRA-25, JRA-55 is based on a new data assimilation and prediction system (DA) that improves many deficiencies found in the first Japanese reanalysis. These improvements have come about by implementing higher spatial resolution (TL319L60), a new radiation scheme, four-dimensional variational data assimilation (4D-Var) with Variational Bias Correction (VarBC) for satellite radiances, and introduction of greenhouse gases with time varying concentrations. The entire JRA-55 production was completed in 2013, and thereafter will be continued on a real time basis.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">(This information was obtained from the NCAR website http://rda.ucar.edu/datasets/ds628.0/)</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">License</span>=== <br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">You need to register by filling the [http://jra.kishou.go.jp/comm/application_en.html | JRA-55 project application form] to use this dataset. A copy of their agreement is available here as [[:File:Application_JRA_Data_Access.pdf]].</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">More information, the JRA55 user manuals for pressure and model levels and quality issues with the data are listed both in the [http://jra.kishou.go.jp/JRA-55/index_en.html#about | JRA-55 project website]</span><br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">and the NCAR [https://climatedataguide.ucar.edu/climate-data/jra-55 | Climate Data Guide] and [http://rda.ucar.edu/#!lfd?nb=y&b=proj&v=JMA%20Japanese%2055-year%20Reanalysis | JRA-55 dataset webpages]</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">JRA55 on raijin</span>=== <br />
====''<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Getting access:</span>''==== <br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">This dataset is part of the climate re-analysis collection, which is stored on raijin in the climate re-analysis collection, which is a collaboration between BoM, CSIRO and ARCCSS.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">To access the data you need to be part of '''rr7 group, request access via [https://my.nci.org.au | my.nci.org.au]''' </span><br />
<br />
====''<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Location of the data</span>''==== <br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Data is located in</span><br />
* <span style="font-size: 120%;">pressure levels: /g/data1/rr7/ana4MIPs/reanalysis/JRA/JRA55/6hr/atmos/<varname>/v1/<filenames></span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">model levels: /g/data1/rr7/ana4MIPs/reanalysis/JRA/JRA55/6hr/atmos/reg-tl319/<varname>/v1/<filenames></span><br />
<span style="color: #f22508; font-family: Arial,Helvetica,sans-serif; font-size: 130%;">We are going to move the data on Monday 13th of August to:</span><br />
<br />
* <span style="color: #f22508; font-family: Arial,Helvetica,sans-serif; font-size: 130%;">pressure levels: /g/data1/rr7/JRA55/6hr/atmos/<varname>/v1/<filenames></span><br />
* <span style="color: #f22508; font-family: Arial,Helvetica,sans-serif; font-size: 130%;">model levels: /g/data1/rr7/JRA55/6hr/atmos/reg-tl319/<varname>/v1/<filenames></span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Filenames are:</span><br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;"> <var>_<mip>_JRA55_<grid>_<YYYYMMDDHH-YYYMMDDHH>.nc</span><br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;"><grid> information is currently present only for model level data and it is "reg-tl319"</span><br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;"><mip> is 6hrPlev and 6hrLev for pressure and model levels respectively</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">NB this is a temporary location, the directory structure will be updated in the future</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Currently available variables on model levels are</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">ta - atmospheric temperature</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">ua - eastward wind component</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">va - northward wind component</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">hus - specific humidity</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">zg - geopotential</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">ps - surface air pressure</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Currently available variables on pressure levels are</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">tas- surface atmospheric temperature</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">uas - surface eastward wind component</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">vas - surface northward wind component</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">hurs - surface relative humidity</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">huss - surface specific humidity</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">zg - geopotential</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">ps - surface air pressure</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">psl - air pressure at sea level</span><br />
* <span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">dptas - Near-Surface Dew Point Depression</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Download & conversion process</span>=== <br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">We downloaded the model analysis level from the NCAR website and the pressure level data from the JMA Data Dissemination System</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">Feedback</span>=== <br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">You can add comment on this collection directly here or on the dedicated [https://opus.nci.org.au/display/CMIP/Observational+and+Reanalysis+Data | Observational and Reanalysis Data page] on NCI confluence</span><br />
<span style="font-family: Arial,Helvetica,sans-serif; font-size: 120%;">For questions use the [https://opus.nci.org.au/display/CC/questions/all | Climate Community confluence questions] .</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=MACC&diff=210MACC2018-06-28T22:59:59Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>The MACC Reanalysis is a global reanalysis data set of atmospheric composition (AC), made available by the Copernicus Atmosphere Monitoring Service (CAMS). The dataset spans the period 2003 to 2012. The spatial resolution of the data set is approximately 80 km (T255 spectral) on 60 vertical levels from the surface up to 0.1 hPa for the analysed species and 1.125° by 1.125° at the same 60 levels for the other chemical species.<br />
For the complete description and technical details see the [https://software.ecmwf.int/wiki/pages/viewpage.action?pageId=81010554 | ECMWF wiki].<br />
We replicated only few surface variables: aerosols at 550 nm, see below for list.<br />
<br />
==='''Known Issues'''=== <br />
A list of known issues is available from the documentation page linked above.<br />
Please note there's a known issue for one of the variables we replicated:<br />
<br />
**Sea salt aerosol mixing ratio above freshwater:''' The model is not distinguishing between sea and freshwater in the context of sea-salt emissions. Unfortunately, this appears to affect all cycles from the MACC reanalysis. A fix is planned to be implemented in a future model release (45r1) and will be effective from the implementation date on. This means that the fix will not be retrospective for the MACC reanalysis but will be in future reanalysis.<br />
<br />
===License=== <br />
If you do not have an account with ECMWF you should create one and agree to their terms and conditions for this dataset.<br />
The license text is available from their [http://apps.ecmwf.int/datasets/data/macc-reanalysis/licence/ | wiki] and contains also the required acknowledgement.<br />
<br />
===MACC on raijin=== <br />
Data is stored as for ERAI and ERA5 in the ub4 project:<br />
<br />
/g/data/ub4/macc/grib/oper_fc_sfc/<files> <br />
<br />
for monthly grib files: <br />
macc_aerosol_oper_fc_sfc_<fromdate>_<todate> <br />
<br />
/g/data/ub4/macc/netcdf/3hr/atmos/oper_fc_sfc/v01/<varname>/<files> <br />
<br />
for monthly netcdf files:<br />
<varname>_3hrs_reanalysis_IFS-Cy36r1_fc-sfc_<fromdate>_<todate>.nc<br />
<br />
where the dates are YYYYMMDD<br />
<br />
Only the following variables are available:<br />
<br />
{| <br />
| varname || param || ECMWF name || Description || standard_name || units <br />
|-<br />
| od550aer || 207.210 || aod550 || Total Aerosol Optical Depth at 550nm [1] || atmosphere_optical_thickness_due_to_ambient_aerosol || 1 <br />
|-<br />
| ssod550aer || 208.210 || ssaod550 || Sea Salt Aerosol Optical Depth at 550nm [1] || atmosphere_optical_thickness_due_to_seasalt_ambient_aerosol || 1 <br />
|-<br />
| duod550aer || 209.210 || duaod550 || Dust Aerosol Optical Depth at 550nm [1] || atmosphere_optical_thickness_due_to_dust_ambient_aerosol || 1 <br />
|-<br />
| omod550aer || 210.210 || omaod550 || Organic Matter Aerosol Optical Depth at 550nm [1] || atmosphere_optical_thickness_due_to_particulate_organic_matter_ambient_aerosol || 1 <br />
|-<br />
| bcod550aer || 211.210 || bcaod550 || Black Carbon Aerosol Optical Depth at 550nm [1] || atmosphere_optical_thickness_due_to_black_carbon_ambient_aerosol || 1 <br />
|-<br />
| suod550aer || 212.210 || suaod550 || Sulphate Aerosol Optical Depth at 550nm [1] || atmosphere_optical_thickness_due_to_sulfate_ambient_aerosol_particles || 1 <br />
|-<br />
|}</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CMORPH&diff=83CMORPH2018-06-25T03:00:19Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>NOAA CPC MORPHing technique: high resolution precipitation (60S-60N) v1.0<br />
<br />
<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">CMORPH produces global precipitation analyses at very high spatial and temporal resolution. This technique uses precipitation estimates that have been derived from low orbiter satellite microwave observations </span>''<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">exclusively</span>''<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">, and whose features are transported via spatial propagation information that is obtained entirely from geostationary satellite IR data.</span><br />
<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">The grid resolution is </span><span style="font-family: Arial,Helvetica,;">0.07277 degrees lat/lon (8 km at the equator) and temporal resolution is 30 minutes. </span><span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">With regard to spatial resolution, although the precipitation estimates are available on a grid with a spacing of 8 km (at the equator), the resolution of the individual satellite-derived estimates is coarser than that - more on the order of 12 x 15 km or so. The finer "resolution" is obtained via interpolation. (source [http://www.cpc.ncep.noaa.gov/products/janowiak/cmorph_description.html | CPC website] )</span><br />
The original data is downloaded from the [[ftp://ftp.cpc.ncep.noaa.gov/precip/CMORPH_V1.0/RAW/8km-30min/ | NOAA CPC ftp server]].<br />
More information is available on the CPC website and the [https://climatedataguide.ucar.edu/climate-data/cmorph-cpc-morphing-technique-high-resolution-precipitation-60s-60n | data climate guide website] .<br />
<br />
==='''Terms of use'''=== <br />
This dataset is freely available, please acknowledge the creators of the dataset. Check their website for more information.<br />
<br />
===<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">References</span>=== <br />
<br />
<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">Ferraro, R. R., 1997: SSM/I derived global rainfall estimates for climatological applications. J. Geophys. Res., 102, 16715-16735. </span><br />
<br />
<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">Ferraro, R. R., F. Weng, N. C. Grody and L. Zhao, 2000: Precipitation characteristics over land from the NOAA-15 AMSU sensor. Geophys. Res. Ltr., 27, 2669-2672. </span><br />
<br />
<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">Joyce, R. J., J. E. Janowiak, P. A. Arkin, and P. Xie, 2004: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution.. J. Hydromet., 5, 487-503. </span><br />
<br />
<span style="font-family: Arial,Helvetica,&#39;San Serif&#39;; font-size: 12px;">Kummerow, C., Y. Hong, W. S. Olson, S. Yang, R. F. Adler, J. McCollum, R. Ferraro, G. Petty, D-B Shin, and T. T. Wilheit, 2001: Evolution of the Goddard profiling algorithm (GPROF) for rainfall estimatin from passive microwave sensors. J. Appl. Meteor., 40, 1801-1820.</span><br />
<br />
==='''CMORPH on raijin'''=== <br />
This dataset is available on raijin under the ua8 project<br />
<br />
/g/data/ua8/CMORPH/CMORPH_V1.0/<br />
--> raw/8km-30min/YYYY/ for the original binary files<br />
<br />
--> netcdf/YYYY/pr_30min_CMORPH_V1_<fromdate>_<todate>.nc for the converted netcdf files<br />
where from date and today are in the format YYYYMMDD<br />
<br />
For CMORPH V0.x data is in<br />
/g/data/ua8/CMORPH/CMORPH_V1.0/netcdf/<br />
<br />
===Update: 25 June 2018=== <br />
<br />
<span style="background-color: #ffffff;">Since there are delays in releasing new data for V1-0 we downloaded its precursor V0.x from July 2017 to date included. The two versions should be compatible but please take into account that V0.x could potentially have issues.</span><br />
<br />
<span style="background-color: #ffffff;">We also concatenated the existing daily files for version v1-0 into monthly files. In two weeks time (09 July) we will delete the daily files and only the monthly will remain available.</span><br />
Finally raw files are now only available as tar or gzip files, i.e. as originally downloaded from the web.</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=GHCN&diff=150GHCN2018-06-08T00:53:41Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>===Global Historical Climatology Network=== <br />
<br />
We replicated the daily station precipitation data from GHCN.<br />
<span style="color: #444444; font-family: &#39;Open Sans&#39;,Helvetica,Verdana,Arial,sans-serif; font-size: 15.215198516845703px;">GHCN-Daily is </span>an integrated database of daily climate summaries from land surface stations across the globe. It is comprised of daily climate records from numerous sources that have been integrated and subjected to a common suite of quality assurance reviews and contains records from over 100,000 stations in 180 countries and territories.<br />
This is an extract of the dataset description from the [https://www.ncdc.noaa.gov/ghcn-daily-description | NOAA website] where you can find more information on this data.<br />
We replicated only the precipitation daily station data organised "by_year". More details on how the data is organised and its status are available from the documentation files stored with the data.<br />
For more information on methods and data quality assurance: [[https://www.ncdc.noaa.gov/ghcn-daily-methods]]<br />
<br />
===License=== <br />
Data is freely available but there are restrictions on commercial use, from their [https://www.ncdc.noaa.gov/ghcnd-data-access | data access policy]:<br />
<br />
<span style="background-color: #ffffff; color: #444444; font-family: &#39;Open Sans&#39;,Helvetica,Verdana,Arial,sans-serif; font-size: 15.215198516845703px;">Some of the data provided here are based on data exchanged under the World Meteorological Organization (WMO) World Weather Watch Program according to </span><span style="color: #0057a5; font-family: &#39;Open Sans&#39;,Helvetica,Verdana,Arial,sans-serif; font-size: 15.215198516845703px; text-decoration: none; vertical-align: baseline;">[http://www.wmo.int/pages/prog/www/ois/Operational_Information/Publications/Congress/Cg_XII/res40_en.html | WMO Resolution 40 (Cg-XII)(link is external)]</span><span style="background-color: #ffffff; color: #444444; font-family: &#39;Open Sans&#39;,Helvetica,Verdana,Arial,sans-serif; font-size: 15.215198516845703px;">. This allows WMO member countries to place restrictions on the use or re-export of their data for commercial purposes outside of the receiving country. Those country's data summaries and products which are available here are intended for free and unrestricted use in research, education, and other non-commercial activities. For non-U.S. locations, the data or any derived product shall not be provided to other users or be used for their re-export for commercial services.</span><br />
<br />
For references: [[https://www.ncdc.noaa.gov/ghcn-daily-references]]<br />
<br />
===GHCN on raijin=== <br />
<br />
GHCN is available on raijin under the ua8 project but is accessible to anyone.<br />
<br />
/g/data/ua8/GHNC/daily/by_year<br />
<br />
The files are ascii format: YYYY.csv<br />
The files format is described in /g/data/ua8/GHCN/ghcn-daily-by_year-format.rtf</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=GSMaP&diff=157GSMaP2018-06-08T00:23:26Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>===Global Satellite Mapping of Precipitation from JAXA=== <br />
<br />
GSMaP is a dataset of hourly global rainfall maps, a re-analysis version of the Near-Real-Time product by JAXA Global Rainfall Watch for meteorological and climate studies. Different products are part of GSMaP we copied the Gauge-calibrated Reanalysis product GSMaP_Gauge_RNL v6. Variable is rainfall rate and gauge-calibrated rain rate in mm/h, original units of raw product are mm/hr we converted to standard units kg m^-2 s^-1 in the netcdf version.<br />
The global grid is limited to 60N-60S and resolution is 0.1 degree latitude/longitude. Temporal resolution is up to 1 hour (averaged from 00 minutes to 59 minutes).<br />
The current algorithm version is 6, version 7 is being released. Product version is v03 which doesn't detect snowfall, although a new version v04 which fixes the issue and use v7 of the algorithm is in preparation.<br />
More information on the upgrade and changes to the product are available [http://www.eorc.jaxa.jp/GPM/doc/product_info/release_note_gsmapv04-v7_en.pdf | here]. The Gauge_RNL product is updated only occasionally currently covers from March 200 to February 2014. We downloaded the daily data (average between 00 to 23 hours) but 1 hourly is available too.<br />
More information is available from the PDF and readme file available in the directory and online and from the [http://sharaku.eorc.jaxa.jp/GSMaP_crest/ | GSMaP webpage]<br />
<br />
===License=== <br />
<br />
Anyone using this product should register with the [http://sharaku.eorc.jaxa.jp/GSMaP/index.htm | JAXA GSMaP products portal]<br />
You should include acknowledgements and citations as directed in the pdf available with the data and from the GSMaP webpage (see above).<br />
JAXA would also like to receive a copy of your manuscript, again see instructions on the webpage.<br />
Signing up is also recommended to receive the frequent updates on the GSMaP products, while they mostly refer to the Near-Real-Time and gauge products, the re-analysis is derived from them so they might contain important information.<br />
<br />
===GSMaP on raijin=== <br />
<br />
The data is available in ua8 project:<br />
<br />
/g/data/ua8/GSMaP/netcdf/v0/rnl_gauge/day/00Z-23Z/<br />
<br />
Filename: gsmap_YYYYMM.nc<br />
<br />
The original binary data is in<br />
<br />
/g/data/ua8/GSMaP/raw/rnl_gauge/day/00Z-23Z/<br />
<br />
NB this directory structure and also the project might change since we are trying to have one structure for all re-analysis products.</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Storage&diff=325Storage2018-05-15T01:48:48Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><br />
During your work at the Centre, you are likely to produce, use and share data on different systems. You will probably have access to two different systems: your University system and NCI.<br />
<br />
=Storage at NCI= <br />
----<br />
NCI provides two types of storage: tape and disk. Tape is for long term storage while disk is more suited to store data you need to access often.<br />
<br />
==Tape at NCI== <br />
----<br />
The tape system at NCI is called /massdata. Please read the [http://nci.org.au/user-support/getting-help/massdata-user-guide/ | Users' guide] to learn how to use this system. Here are a few important points to keep in mind:<br />
* Tape is mostly appropriate for archiving data.<br />
* You should only store big files on tape. If you want to migrate a lot of small files, you should first archive them together. To learn how to do that please have a look at the [http://nci.org.au/user-support/getting-help/massdata-user-guide/ | Users' guide] and [[mailto:climate_help@nci.org.au | email]] your questions to us.<br />
* /massdata is only accessible from the login nodes (interactive) or via a script submitted to the copyq queue. It is generally recommended to use the copyq queue as you then have a much longer run time.<br />
* Tape access (writing and reading) is slow.<br />
* Considering data on /massdata are likely to be un-used for a long time it is quite essential to document your data. For example adding a detailed README file to your data folder can help a lot.<br />
* Storage update for quota is only updated daily, overnight for massdata because of the size of it. It is then recommended to act quickly for clean up and if possible before breaching the quota.<br />
===<span style="line-height: 1.5;">Additional storage</span>=== <br />
<span style="line-height: 1.5;">It might be possible to add quota on massdata for your project to do so please send an [[mailto:climate_help@nci.org.au | email to us]] detailing how much additional space you would like and which NCI project it is for. Again, being tape storage the request might take some days to be processed so please plan ahead.</span><br />
<br />
==Disk at NCI== <br />
----<br />
There are three different disk filesystems at NCI, each with a slightly different purpose. NCI also has a [http://nci.org.au/user-support/getting-help/filesystem-user-guide/ | Users' Guide]. All the disk filesystems are accessible from the login nodes and the compute nodes hence you can read/write to one while currently being in an other filesystem. And all these filesystems have access to massdata either through login nodes connection or sending a script to copyq queue.<br />
===/home=== <br />
* <span style="line-height: 1.5;">This is your home directory. </span><br />
* <span style="line-height: 1.5;">This space is strictly limited at 2GB for each user but it is backed up. </span><br />
* <span style="line-height: 1.5;">It is most suitable for storing source code rather than model outputs or observation datasets. </span><br />
* <span style="line-height: 1.5;">You can monitor your use on home with the "lquota"[1] command.</span><br />
<br />
===/short=== <br />
* <span style="line-height: 1.5;">All projects have some storage on /short. </span><br />
* <span style="line-height: 1.5;">The amount of space varies from a project to another. </span><br />
* <span style="line-height: 1.5;">The management of the space is left to the responsibility of the members of each project. Although automatic emails are sent to the project members when the usage comes close to fill in the quota.</span><br />
* <span style="line-height: 1.5;">When a project fills its quota on /short, the project's members will not be able to use the computing queues except for copyq queues to help with moving data around. </span><br />
* <span style="line-height: 1.5;">To monitor the overall usage, please use the "<span style="font-family:monospace">nci_account}}"[1] command. To monitor the usage per user, please use "{{short_files_report -P $PROJECT</span>" [1].</span><br />
* <span style="line-height: 1.5;">Increasing the quota on /short for a project might be possible but is left to the decision of NCI staff. If you want to try to have an extension, please send us an [[mailto:climate_help@nci.org.au | email]].</span><br />
<br />
===/g/data=== <br />
* <span style="line-height: 1.5;">Most projects now have some storage on /g/data1 or /g/data3. </span><br />
* <span style="line-height: 1.5;">As for /short, the quota on /g/data is per project with management of the usage the sole responsibility of the project's members. </span><br />
* <span style="line-height: 1.5;">/g/data can be less stable than /short. As such it is recommended to use the special PBS resource: </span>#PBS -lother=gdata. Then your job will only start when the /g/data filesystem is accessible.<br />
* Compute nodes have both read and write access to this filesystem.<br />
* Your project's quota on /g/data is not extensible by simple email. There is a review of some of the quotas every year, at which point some projects might be granted an increase. If you need additional storage before then, please consider deleting old or incorrect data, archiving old data to /massdata, using [[Storage#temporary | temporary storage]] or your [[Storage#local | University system]]. If you still want an increase to be considered at review time, please make sure to discuss it with the Lead CI of your project who will be part of the review.<br />
* To monitor usage, please use "<span style="font-family:monospace">nci_account}}" [1]. To see a per-user summary run the command "{{gdata1_files_report -P $PROJECT</span>"<br />
<br />
==<span id="temporary"></span>Temporary storage== <br />
----<br />
The CMS is also managing two projects on NCI that can be used for temporary storage of data. Both are mounted on /g/data and have the same characteristic as other /g/data storage space as explained above.<br />
<br />
**To use any space on these projects''', you need to:<br />
* request [https://my.nci.org.au/mancini/login?next=/mancini/project/ | connection to the project] if you are not yet a member. You can check which projects you are part of with the "groups" command.<br />
* fill a storage request at [http://dmp.climate-cms.org:3000/]] Please [[mailto:climate_help@nci.org.au | email us] if you have any question about this form. Note this form is principally to enable us to monitor the space used, requested and available. It also enables us to prepare a folder for you with appropriate permissions. The forms are very short and quick to fill and the storage is usually ready for use within a few hours. See [[storage-request|this page]] for more detailed instructions on how to fill the form. Note:<br />
** you do not need to email us, please just fill in the form to request the space you'd like.<br />
** Also, you can request space for use by a whole group instead of per user, but all users of the group '''must''' request connection to the project.<br />
The temporary storage projects are:<br />
* /g/data3/hh5: this project is for short temporary use (~3 months). It could be used for example to print your raw model outputs, then you would save a subset or a reformatted version to your project's space and move the raw outputs to /massdata for safekeeping.<br />
* /g/data1/ua8: the main purpose of this project is to store published datasets created by the Centre's staff. For example, some journals now request that researchers publish their data in parallel to their papers. It is also used for small downloaded datasets that are shared across the CoE and do not have their own "project". However, the free space in this project can be used as temporary storage. Our preference would be to store data that might potentially require publication.<br />
<br />
=<span id="local"></span>Storage at Universities= <br />
----<br />
<span style="line-height: 1.5;">at ANU</span><br />
at Monash<br />
at UMelb<br />
at UNSW<br />
at UTas<br />
<br />
=File compression and archiving= <br />
----<br />
For an efficient use of storage, there are a few rules to keep in mind:<br />
* Nowadays it is better to store a few larger files than lots of smaller files. It is hard to define large and small but files of several tens of gigabytes are absolutely acceptable. That said it is best practice to compress your data when possible.<br />
* Netcdf files are now easily compressible, see [[NetCDF Compression Tools | this article]] for detailed explanations on tools available at NCI.<br />
<br />
To store small setup files that define your experiments, think about using the "tar" command. This is a shell command with a manual accessible through<br />
<syntaxhighlight><br />
man tar<br />
</syntaxhighlight><br />
This command will save many files together in a single archive, it can be used on a directory tree and will restore the directory structure when restoring the files from the archive. This means if you have several experiments you need to save the setup of, the best way might be to create a directory tree containing the setup files of all the experiments then create one single archive file for all. The archive files can also easily be compressed/uncompressed using the gzip utility either at the archive creation time or afterwards.<br />
<br />
[1]: see the [https://drive.google.com/drive/folders/0B1KncyYETiZqNGdHZ0VZSk02cUE | 2017 training material] for usage of these commands and acls and tar cheat sheets.</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CLEX_and_ARCCSS_data_collection&diff=55CLEX and ARCCSS data collection2018-05-07T04:16:32Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>=ARCCSS data collection at NCI= <br />
<br />
These are dataset created by the Centre students and researchers and published on the NCI geonetwork catalog under the [http://geonetwork.nci.org.au/geonetwork/srv/eng/metadata.show?id=2&currTab=simple | ARCCSS data collection] .<br />
Each dataset has also a Research Data Australia record, which contains a detailed description of the data and it is the correct way to cite the data.<br />
This collection does not include all of the data published by the centre, only smaller datasets which are not part of a larger collection already. If you want to publish your research results or you need to make your data available because of publishing requirements, we can add your dataset to this collection.<br />
These datasets are available on raijin, they are stored on /g/data1/ua8/ARCSS_Data/ and will be soon available on the web through the [http://dap.nci.org.au/thredds/catalog.html | NCI thredds catalog].<br />
To access them you can join ua8 project, to do so use your NCI account to access [http://%20https:''my.nci.org.au/ | MyNCI portal].<br />
<br />
* <span class="s1">[https://researchdata.ands.org.au/access13b-model-output-experiment-v10/453996 | ACCESS1-3b_AMZDEF v1.0] - </span>ACCESS1.3b model output from the Amazonian Deforestation (AMZDEF) experiment <span style="color: #454545;">(version 1.0)</span><br />
* [https://researchdata.ands.org.au/access13b-model-output-experiment-v20/653710 | ACCESS1-3b_AMZDEF v2.0] - ACCESS1.3b model output from the Amazonian Deforestation (AMZDEF) experiment <span style="color: #454545;">(version 2.0)</span><br />
* [https://researchdata.ands.org.au/lagrangian-drifter-output-forced-tropac01/472299 | CMS_TROPAC01_LeeuwinCurrent v1.0] - <span style="color: #454545;"><span style="color: #454545;">Lagrangian drifter output in the Southeast Indian Ocean using the Connectivity Modelling System output forced with TROPAC01 </span>(version 1.0)</span><br />
* <span style="color: #454545;">[https://researchdata.ands.org.au/derived-optimal-linear-combination-evapotranspiration/815690 | DOLCE v1.0] - Derived Optimal Linear Combination Evapotranspiration (version 1.0)</span><br />
* <span class="s1">[https://researchdata.ands.org.au/time-series-emcei-components-subcomponents/512056 | EmCEI v1.0] - </span>Time series of EmCEI Climate Extremes Index components and subcomponents <span style="color: #454545;">(version 1.0)</span><br />
* [https://researchdata.ands.org.au/area-extent-temperature-observations-reanalysis/754145 | EmCEI-CMIP5 v1.0] - Area-extent of temperature and precipitation extremes in CMIP5 models under different radiative forcing scenarios, observations and reanalysis <span style="color: #454545;">(version 1.0)</span><br />
* [https://researchdata.ands.org.au/loveclim-glacial-maximum-d13c-d14c/792249 | LGMc13 v1.0] - LOVECLIM Last Glacial Maximum oceanic d13C and D14C (version 1.0)<br />
* [https://researchdata.ands.org.au/maritime-continent-austral-climatology-v10/794716 | MCASClimate v1.0] - Maritime Continent Austral Summer Climatology (version 1.0)<br />
* [https://researchdata.ands.org.au/simulations-glacial-climate-mk3l-v10/814964 | MK3L-LGM v1.0] - Simulations of glacial climate and ocean biogeochemistry with the CSIRO Mk3L model (version 1.0)<br />
* <span class="s1">[https://researchdata.ands.org.au/weather-research-forecasting-version-10/461001 | WRF_Sydney2k v1.0] - </span>Weather Research and Forecasting model outputs from the 2-km resolution Sydney experiments (version 1.0)<br />
* Mk3L-BioPump v1.0 - (version 1.0)</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CESM1-CAM5-BGC-LE&diff=74CESM1-CAM5-BGC-LE2018-05-03T03:35:52Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>The CESM Large Ensemble Project is a publically available set of climate model simulations intended for advancing understanding of internal climate variability and climate change. All simulations are performed with the nominal 1-degree latitude/longitude version of the Community Earth System Model version 1 (CESM1) with CAM5.2 as its atmospheric component.<br />
For more information: [http://www.cesm.ucar.edu/projects/community-projects/LENS/ | CESM1 LENS website]<br />
You can find [http://www.cesm.ucar.edu/projects/community-projects/LME/known-issues.html | known issues] with this dataset listed in the same website.<br />
<br />
===<span style="background-color: #ffffff;"><span id="x--Terms of use"></span>Terms of use</span>=== <br />
<br />
<span style="background-color: #ffffff;">From the CESM LENS project website:</span><br />
<span style="background-color: #ffffff; color: #484848; font-family: UbuntuRegular; font-size: 14px;">When presenting results based on the CESM Large Ensemble in either oral or written form, please acknowledge the CESM Large Ensemble Community Project and supercomputing resources provided by NSF/CISL/Yellowstone, and reference Kay et al. (2015):</span><br />
<span style="background-color: #ffffff; color: #484848; font-family: UbuntuRegular; font-size: 14px;">Kay, J. E., Deser, C., Phillips, A., Mai, A., Hannay, C., Strand, G., Arblaster, J., Bates, S., Danabasoglu, G., Edwards, J., Holland, M. Kushner, P., Lamarque, J.-F., Lawrence, D., Lindsay, K., Middleton, A., Munoz, E., Neale, R., Oleson, K., Polvani, L., and M. Vertenstein (2015), The Community Earth System Model (CESM) Large Ensemble Project: A Community Resource for Studying Climate Change in the Presence of Internal Climate Variability, Bulletin of the American Meteorological Society, doi: 10.1175/BAMS-D-13-00255.1, 96, 1333-1349 <span style="color: #fa8128;">[http://journals.ametsoc.org/doi/abs/10.1175/BAMS-D-13-00255.1 | [Article]]</span></span><br />
<br />
===<span style="background-color: #ffffff;"><span id="x--CESM1-LME on raijin"></span>CESM1-CAM5-BGC-LE on raijin</span>=== <br />
<br />
We downloaded a subset of the entire collection, both monthly and daily fields. For each selected variables we downloaded all the available ensembles ??.<br />
The data was downloaded from the NCAR [https://www.earthsystemgrid.org/home.html | earth system website] .<br />
The current directory structure mimics the ESGF official DRS and location on raijin is:<br />
<br />
/g/data/ua6/CESM1-CAM5-BGC-LE/<experiment>/<frequency>/<realm>/<mip_table>/<ensemble>/<variable>/<version>/<files><br />
<br />
For example:<br />
/g/data/ua6/CESM1-CAM5-BGC-LE/atmControl/day/atmos/day/r1/psl/v2/<br />
<br />
Available variables are listed in this [https://docs.google.com/spreadsheets/d/e/2PACX-1vRO6r8zPQK-rk4e9HwSsB5Wi4dlsj_NLazdKw6OM7Xm_gnmzuBi3JfEFH4bmZFo9gZjyDHNn4AXcjmk/pubhtml | google spreadsheet]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS_publishing_procedure&diff=8ACCESS publishing procedure2018-03-13T08:42:01Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>'''<span style="font-family: Cambria; font-size: 16pt;">Publishing ACCESS model simulations to ESGF</span>'''<br />
<br />
<span style="font-family: Cambria;">There are two main phases to prepare an ACCESS simulation for publication on an ESG node: the post-processing and the quality control (QC).</span><br />
**<span style="font-family: Cambria; font-size: 14pt;">Post-processing</span>'''<br />
<span style="font-family: Cambria;">In this phase we extract or calculate, from the raw model output, selected variables as defined by the [http://cmip-pcmdi.llnl.gov/cmip5/docs/standard_output.pdf | CMIP5 standard_output table] . </span><br />
To do so we use the [https://trac.nci.org.au/svn/access_tools/post_processor/branches/APP1-0 | post-processor app] created by Peter Uhe, CAWCR, this is a python wrapper to the [http://cmip-pcmdi.llnl.gov/cmip5/output_req.html#cmor | Climate Model Output Rewriter] (CMOR). CMOR2 is the current version and can be used to satisfy the specifications for CMIP5 model output.<br />
<br />
**APP1-0 description'''<br />
<br />
APP1-0 is roughly made up of<br />
* database/champions – contains all the champions tables the files defining which variables will be calculated, their publishing status, the input variables, input filenames and, if applicable, how they will be calculated. There is a file for each CMIP table, plus a few more for “extra” variables. Fields are: ''cmip variable, definable in access, implemented in post processor, dimensions, access variable name, access file location, realm, calculation, override units, axes modifier, positive, notes.'' See at bottom of page for an example.<br />
* database/experiments.csv – this is the default name for a csv file in which the following attributes/settings are set for each simulation: ''experiment_id, experiment_directory, start_year, end_year, realization, initialization, physics_version, local_experiment name, model name, forcing, parent_id, parent_rip, branch time, land model, notes''<br />
* database/app.db – this is the default name of a sqlite database that contains a combination of the information provided by the experiments.csv file and the champions tables. You can use the same app.db and experiments.csv file for all your simulations, we normally produced one for each experiment adding _local-exp-name to distinguish them. There is a row for each file that needs to be processed. The “status” field is updated after each processing step: ''unprocessed, processing, processed, published''. And in case of errors: ''processing_failed''.<br />
* database/database_manager.py is the python script that produces the app database. Which champions tables are used for different CMIP5 experiments (like historical, amip etc.) are defined here. If a database with the same name exists already it will update it<br />
* app_wrapper.py – this is a wrapper to app.py. It calls app.py and passes information about a specific simulation. It reads input information from the run app database (app.db default name) and set the version (ex. V20130607).<br />
* app.py – the main post-processor scripts, it calls CMOR passing to it all the relevant settings. In app.py some of the global attributes (such as title) are directly defined. App.py can call ''app_functs.py'' if a calculation needs to be performed to output a variable.<br />
* move_to_published.py – this script has two functions update the version and move the files to the “published” directory, currently /g/data1/p66/CMIP5/published/<br />
This will become /g/data/rr3/publications/CMIP5 or <span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">ACCESS_pre-publishing</span>, at the moment the NCI publisher rsync the p66 dir to this one before starting the actual publishing. The “published” directory follows the official CMIP5 DRS. An example:<br />
''..''''/CMIP5/output1/CSIRO-BOM/ACCESS1-3/historicalNat/mon/atmos/Amon/r1i1p1''''/files/cl_20130906/cl_Amon_ACCESS1-3_historicalNat_r1i1p1_185001-187412.nc''<br />
* call.sh, run.sh, setup_env.sh – these are the scripts used to actually submit to the queue an instance of the post-processor and set up the environment to run it<br />
* /um2netcdf/ - this directory contains the script necessary to convert the UM binary output to netcdf<br />
<br />
**Set up of the APP1-0'''<br />
<br />
# svn checkout [[https://trac.nci.org.au/svn/access_tools/post_processor/branches/APP1-0]]<br />
# Update the run.sh, call.sh and setup_env.sh files so you are loading the correct environment variables and NCI project id. See at bottom of page for example scripts.<br />
# Check that the settings in the champions tables are correct for your simulations. In particular: ''definable in access'' should be “yes” if you want to process the variable, ''implemented in post processor'' is a string representing the publishing priority and ''file location'' completes the input file path in the experiments.csv.<br />
# In APP1-0/database there is an example of an experiments.csv file. Update this file with the information relating your simulations, you can use one for all simulation, adding a new row for each simulation, or, as we did, use separate ones for each of them. In particular this is where you set up the first part of the input files path.<br />
# Set up the directory where you are storing the input files. Ex.<br />
** <span style="line-height: 1.5;">../../<local-exp>/history/atm/netCDF/<UM output files></span><br />
** <span style="line-height: 1.5;">../../<local-exp>/history/ocn/<ocean output files></span><br />
** <span style="line-height: 1.5;">../../<local-exp>/history/ice/<sea-ice output files></span><br />
# <span style="font-family: Cambria; font-size: 12pt;">Set up the directory where you are storing the output files and the ancillary files. Ex.</span><br />
** ../../<local-exp>/CMIP5/job_output/<br />
** <span style="line-height: 1.5;">../../<local-exp>/CMIP5/ancillary_files/ {cice_grid_....nc, grid_spec.auscom_….nc, landFrac.nc, lsmask_.....nc, Base-09Ipj.astart-….nc}</span><br />
<br />
**Running the APP1-0'''<br />
# <span style="font-family: Cambria;">source setup_env.sh – to load all the necessary modules and setting up the environment variables</span><br />
# <span style="font-family: Cambria;">python database_manager.py – to create the app database</span><br />
# <span style="font-family: Cambria;">./run.sh – will submit to the queue call.sh which load the environment and run the app_wrapper.py python script.</span><br />
# <span style="font-family: Cambria;">Check the run output and error log files. If there are errors or the process was interrupted and you need to reprocess a variable use the reset_db_status.py script in /database/ to reset the variable status to “''unprocessed''”. In case of error the status will be “''processing_failed''”, in case the job is interrupted before it can update the status, it will be “''processing''”.</span><br />
<br />
**<span style="font-family: Cambria; font-size: 14pt;">Quality Control</span>'''<br />
<br />
The software to manage the QC component has been developed by DKRZ ( [[http://svn-mad.zmaw.de/svn/mad/Model/QualCheck]]). A copy of this is installed on raijin in /g/data1/p66/QC . The software consists of two components:<br />
* a set of c++ executables and scripts (QC) to do the actual checks (min,mean,max, etc)<br />
* a wrapper script (QCWrapper) to upload the results to a central postgressDb at DKRZ.<br />
<br />
**<span style="font-family: Cambria;">Move to publishing directory</span>'''<br />
# <span style="font-family: Cambria;">check settings for “outpath” if you’re not using the path set in the database. In our case we commented all the rows where the outpath was read from the app database, because the files have already being moved once to the standard ACCESS output directory. So we have explicitly defined outpath='/projects/p66/pfu599/'</span><br />
# <span style="font-family: Cambria;">modify the select statement (around line 200 in code) so it select the fields you want to publish (use the priority field Ex. publish1 highest priority, etc. we have also added the publishing date to this field)</span><br />
# <span style="font-family: Cambria;">run first selecting the update_version option (around line 210) if you need to.</span><br />
# <span style="font-family: Cambria;">then run selecting the move_to_published option (around line 211), do this the first time as a dry_run, to check everything is ok, and then publish_var. You set these at lines 160-165 tmp=dry_run and tmp=publish_var</span><br />
<br />
**<span style="font-family: Cambria;">Quality Control set up</span>'''<br />
# <span style="font-family: Cambria;">go to /projects/p66/QC/QCWrapper and create your own running directory </span><br />
# <span style="font-family: Cambria;">In this subdirectory copy the config file: qc_ACCESS_final.conf, and change</span><br />
** <span style="font-family: Cambria;">contact details and QCBD logfile at the start of the script (lines 18-24)</span><br />
** <span style="font-family: Cambria;">$QCBD_VERSION experiment version/s to process (around lines 30-50)</span><br />
** <span style="font-family: Cambria;">$DATA_ROOT_FS for the experiment, which has to end with the “experiment” subdir. Ex. DATA_ROOT_FS=/projects/p66/CMIP5/published/CMIP5/output1/</span><span style="font-family: Cambria; line-height: 1.5;">CSIRO-BOM/ACCESS1-3/historical/ (around lines 50-60)</span><br />
# <span style="font-family: Cambria; line-height: 1.5;">copy qc_call.sh in your own dir , change reference to configuration file (NB --nodb is the </span>''<span style="font-family: Cambria; line-height: 1.5;">“no database”</span>''<span style="font-family: Cambria; line-height: 1.5;"> flag so it doesn't submit files to external database, --noqc is the </span>''<span style="font-family: Cambria; line-height: 1.5;">“no quality control”</span>''<span style="font-family: Cambria; line-height: 1.5;"> flag). An example is shown at bottom of page.</span><br />
# <span style="font-family: Cambria; line-height: 1.5;">submit job to queue by running qsub qc_call.sh</span><br />
<br />
**<span style="font-family: Cambria;">After QC is finished</span>'''<br />
# <span style="font-family: Cambria;">Check log file /projects/p66/QC/QCWrapper/<your-subdir>/qcdb.log</span><br />
# <span style="font-family: Cambria;">Check results under /projects/p66/QC/QCResults.ACCESS/CMIP5/output1/CSIRO-BOM/<<model>>/<<experiment>>/data/<<freq>>/<<realm>>/<<cmip-table>>/<<ensemble>>/<<version>>/<<var>>/</span><br />
** <span style="font-family: Cambria;">qc_<var>_<cmip-table>_<model>_<experiment>_<ensemble>.nc contains stats for the file</span><br />
** <span style="font-family: Cambria;">qc_warning_<var>_<cmip5-table>_<model>_<experiment>_<ensemble>_<period>.txt contains warning messages. You can safely ignore all the warnings about ACCESS1-3/ACCESS1-0 name.</span><br />
** <span style="font-family: Cambria;">tid_<var>_<cmip-table>_<model>_<experiment>_<ensemble>.txt contains the file tracking id and the error code (if 0 no errors)</span><br />
# <span style="font-family: Cambria;">You can plot the file stats by copying in your area /g/data1/p66/QC/QCWrapper/pxp581/QC_scripts/qc_plot.py, this will produce .png files and store them in …/QC_scripts/qc_plots . On raijin you can use display <filename>.png to plot them. You can choose a database_plot option and you set the version of the experiment you want to process or a manual option that allows you to add more constraints to your selection.</span><br />
<br />
**<span style="font-family: Cambria;">Example of APP1-0/run.sh</span>'''<br />
<br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#!/bin/bash -l</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">date=`date +%F_%H-%M-%S`</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">cd $HOME/APP1-0</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#set up environment</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">. ./setup_env.sh</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#Inputs</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">Exp=testHG3-HT # local experiment name</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">N=200 # maximum number of files that will be processed</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">Table=Limon # CMIP table to be processed </span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#Output files</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">name=a${Table}_${Exp}</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">dir=${APP_OUTPATH}/CMIP5/job_output/${Exp}/${Table}/</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">output=${dir}${date}_output.txt</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">error=${dir}${date}_error.txt</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">mkdir -p $dir</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#qsub command</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">qsub -P w97 -q normal -l walltime=8:00:00,mem=16GB -lother=gdata1 -N $name -o $output -e $error -v Table=$Table,Exp=$Exp,N=$N ./call.sh</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#command to use without environment variables</span><br />
<span style="font-family: &#39;Menlo Regular&#39;; font-size: 11pt;">#qsub -P p66 -q normal -l walltime=8:00:00,mem=3500MB -wd -v ./call.sh</span><br />
<br />
**Example of APP1-0/call.sh'''<br />
<br />
#!/bin/bash -l<br />
cd $HOME/APP1-0<br />
#set up environment<br />
. ./setup_env.sh<br />
#Call python script<br />
python app_wrapper.py<br />
<br />
**Example of APP1-0/setup_env.sh'''<br />
<br />
#!/bin/bash -l<br />
#Set up environment for post-processor<br />
# Global environment<br />
module use ~access/modules<br />
module load netcdf/4.3.0<br />
module load python/2.7.3<br />
module load python/2.7.3-matplotlib<br />
module load pythonlib/cdat-lite/6.0rc2-fixed<br />
module load pythonlib/ScientificPython/2.8<br />
module load pythonlib/cmor<br />
module load udunits<br />
export UDUNITS2_XML_PATH=/apps/udunits/2.1.24/share/udunits/udunits2.xml<br />
export PYTHONPATH=$PYTHONPATH:~/pythonlibs/lib/python2.7/site-packages/<br />
# APP environment variables<br />
export APP_OUTPATH=/g/data1/ua8/tmp/testHG3-HT/ #set this to where you want output data and logs to go<br />
export APP_CHAMPIONS_DIR=$HOME/APP1-0/database/champions/<br />
export APP_EXPERIMENTS_TABLE=experiments.csv<br />
export APP_DATABASE=$HOME/APP1-0/database/app.db<br />
export CDAT_LOCATION=~access/apps/pythonlib/cdat-lite/i6.0rc2-fixed/lib/python2.7/site-packages/cdat_lite-6.0rc2-py2.7-linux-x86_64.egg<br />
<br />
**<span style="font-family: Cambria;">Example of a champions table: day_limited.csv</span>'''<br />
<br />
#Champions table: day_limited,,,,,,,,,,,<br />
#cmip variable,definable in access,implemented in post processor,dimensions,access variable name,access file location,realm,calculation,override units,dummy file,positive,notes<br />
rhs,yes,pub2013-11,2Datmos,hurs,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
rhsmin,yes,pub2013-11,2Datmos,hursmin,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
rhsmax,yes,pub2013-11,2Datmos,hursmax,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
snc,no,unknown,2Datmos,,,landIce,,,,,<br />
clt,yes,pub2013-11,2Datmos,cldlw,/atm/netCDF/*_dai.nc,atmos,,1,,,<br />
tslsi,yes,failed,2Datmos,ts ts_sea<br />
snw,yes,unknown,2Datmos,snw_land,/atm/netCDF/*_dai.nc,landIce,var[0].filled(0),,,,<br />
prc,yes,pub2013-11,2Datmos,prc1 prc2,/atm/netCDF/*_dai.nc,atmos,var[0]+var[1],,,,<br />
prsn,yes,pub2013-11,2Datmos,prsn,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
mrro,yes,need_check,2Datmos,mrros smrros,/atm/netCDF/*_dai.nc,land,var[0]+var[1],,,,"STASHmapping: m01s08i234 + m01s08i235, bad units: kg m-2 a-2"<br />
uas,yes,pub2013-11,2Datmos,uas,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
vas,yes,pub2013-11,2Datmos,vas,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
sfcWindmax,yes,pub2013-11,2Datmos,wndmax,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
hfls,yes,pub2013-11,2Datmos,hfls,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
hfss,yes,pub2013-11,2Datmos,hfss,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
rlds,yes,pub2013-11,2Datmos,rlds,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
rlus,yes,pub2013-11,2Datmos,rls rlds,/atm/netCDF/*_dai.nc,atmos,(var[0]-var[1])*-1,,,,(m01s02i201-m01s02i207)*-1<br />
rsds,yes,pub2013-11,2Datmos,rsds,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
rsus,yes,pub2013-11,2Datmos,rss rsds,/atm/netCDF/*_dai.nc,atmos,(var[0]-var[1])*-1,,,,(m01s01i201-m01s01i235)*-1<br />
rlut,yes,pub2013-11,2Datmos,rlut_bl,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
usi,yes,pub2July,2Docean,uvel_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5<br />
vsi,yes,pub2July,2Docean,vvel_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5<br />
sic,yes,pub2July,2Docean,aice_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5<br />
sit,yes,pub2July,2Docean,hi_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5<br />
ta,yes,pub24_01_13,3Dp8,ta,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
hur,yes,pub24_01_13,3Dp8,hur<br />
hus,yes,pub24_01_13,3Dp8,hus,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
wap,yes,pub24_01_13,3Dp8,wap,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
va,yes,pub24_01_13,3Dp8,va,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
ua,yes,pub24_01_13,3Dp8,ua,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
zg,yes,pub24_01_13,3Dp8,zg,/atm/netCDF/*_dai.nc,atmos,,,,,<br />
<br />
**Example of qc_call.sh to run QC check'''<br />
<br />
#!/bin/bash<br />
#PBS -l walltime=4:00:00<br />
#PBS -l mem=1900mb<br />
#PBS -j oe<br />
#PBS -l wd<br />
#PBS -o logs<br />
module load python/2.7.3<br />
module load python/2.7.3-matplotlib<br />
module use ~access/modules<br />
export PYTHONPATH=$PYTHONPATH:~pfu599/pythonlibs<br />
export QCDB_TYPE=ORACLE<br />
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/g/data1/p66/QC/Oracle/instantclient_11_1<br />
python ./src/qcWrapper.py --configure=qc_ACCESS.conf --nodb</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=OSTIA-SST&diff=274OSTIA-SST2018-01-31T02:30:51Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>=Global Ocean <span class="highlighto">OSTIA</span> Sea Surface Temperature and Sea Ice Reprocessed= <br />
<br />
For the Global Ocean- the OSTIA global foundation Sea Surface Temperature product provides daily gap-free maps of:<br />
Foundation Sea Surface Temperature at 0.05x 0.05 degrees horizontal resolution, using in-situ and satellite data from both infra-red and micro-wave radiometers.<br />
* Sea Surface Temperature anomaly from the Pathfinder climatology at 0.25 x 0.25 degrees horizontal resolution. <br />
* Estimates of SST bias in each of the satellites contributing to the OSTIA SST analysis at 0.25 x 0.25 degrees horizontal resolution.<br />
* Monthly and seasonal means of the daily Sea Surface Temperature product at 0.25 x 0.25 degrees horizontal resolution are also available. <br />
* This product provides the foundation Sea Surface Temperature which is the surface temperature free of diurnal variability.<br />
Downloaded from the [http://marine.copernicus.eu/services-portfolio/access-to-products/?option=com_csw&view=details&product_id=SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001 | Marine Copernicus portal] . <span style="background-color: #ffffff;">This dataset is updated regularly every week, data temporal coverage is about one week delayed from present date.</span><br />
<br />
===License and terms of use=== <br />
<br />
This dataset is provided as a local cache for ARCCSS researchers only. Please register with the Marine Copernicus portal if you are using this product<br />
<br />
===OSTIA-SST on raijin=== <br />
<br />
Directory structure mirrors the Copernicus ftp server layout, if you wish to use files that we don't have available please contact the CMS helpdesk.<br />
data is available from the ua8 project:<br />
<br />
* /g/data/ua8/ostia/Core/SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001/METOFFICE-GLO-SST-L4-NRT-OBS-SST-V2/<YYYY>/<MM>/<files><br />
filenames are: <br />
* YYYYMMDDhhmm00-UKMO-L4_GHRSST-SSTfnd-OSTIA-GLOB-v02.0-fv02.0.nc<br />
For example:<br />
* 20180101120000-UKMO-L4_GHRSST-SSTfnd-OSTIA-GLOB-v02.0-fv02.0.nc<br />
> <br />
===Available data=== <br />
<syntaxhighlight><br />
Years Collection Dataset Variable<br />
---------|---------------------------------------|------------------------------------|--------<br />
1985-2007 SST_GLO_SST_L4_REP_OBSERVATIONS_010_011 METOFFICE-GLO-SST-L4-RAN-OBS-SST sst<br />
1985-2007 SST_GLO_SST_L4_REP_OBSERVATIONS_010_011 METOFFICE-GLO-SST-L4-RAN-OBS-SST anom<br />
2007-2014 SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001 METOFFICE-GLO-SST-L4-NRT-OBS-SST-V2 sst<br />
2007-2014 SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001 METOFFICE-GLO-SST-L4-NRT-OBS-ANOM-V2 sst<br />
</syntaxhighlight><br />
===References=== <br />
<br />
Bell, M.J., R.M. Forbes and A. Hines, 2000, Assessment of the FOAM global data assimilation system for real-time operational ocean forecasting. J. of Marine Systems, 25(2000) 1-22.<br />
<br />
Donlon, C.J., Martin, M., Stark, J., Roberts-Jones, J., Fiedler, E., and Wimmer, W., 2012, The Operational Sea Surface Temperature and Sea Ice Analysis (OSTIA) system. Remote Sensing of the Environment. doi: 10.1016/j.rse.2010.10.017 2011.<br />
<br />
John D. Stark, Craig J. Donlon, Matthew J. Martin and Michael E. McCulloch, 2007, OSTIA : An operational, high resolution, real time, global sea surface temperature analysis system., Oceans 07 IEEE Aberdeen, conference proceedings. Marine challenges: coastline to deep sea. Aberdeen, Scotland.IEEE.</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=GPCP&diff=155GPCP2018-01-30T23:46:22Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>The Global Precipitation Climatology Project, produced by the NASA Mesoscale Atmospheric Processes Laboratory, comprises threee products: One-Degree Daily (1DD), monthly product and Pentad data. We downloaded the daily product version 1.2 and 1.3 which correspond respectively to the v2.2 and v2.3 for monthly data. <span style="color: #f0370e;">Version 1.3 is the most recent but is still a beta test so you should expect changes. </span><br />
The GPCP One-Degree Daily Precipitation Data Set (1DD Data Set) provides daily, global 1x1-deg gridded fields of precipitation totals for October 1996 through the delayed present. The 1DD draws upon several different data sources, surface and satellite measurements, covering different areas of the globe.<br />
This product is updated sporadically, about once a month.<br />
We downloaded v1.3 from the [https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/ | NCEI server], and v1.2 from the [https://rda.ucar.edu/datasets/ds728.3/#!description | NCAR Research Data Archive] . More information is available from the [https://precip.gsfc.nasa.gov/ | GPCP webpage] .<br />
<br />
===License=== <br />
<br />
Access is free but users are requested to add the official citation as below and email the citation to george.j.huffman@nasa.gov or david.t.bolvin@nasa.gov.<br />
<br />
===Citation=== <br />
v1.2<br />
Huffman, G. J., D. T. Bolvin, and R. F. Adler. 2016. GPCP Version 1.2 One-Degree Daily Precipitation Data Set. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory. https://doi.org/10.5065/D6D50K46. Accessed† 30 Jan 2018.<br />
<br />
v1.3<br />
Simply use references, for the moment no specific indication has been given.<br />
<br />
===References=== <br />
<br />
Huffman, G. J., R. F. Adler, M. M. Morrissey, D. T. Bolvin, S. Curtis, R. Joyce, B. McGavock, and J. Susskind, 2001: [https://doi.org/10.1175/1525-7541(2001)002%3C0036:GPAODD%3E2.0.CO;2 | Global precipitation at one-degree daily resolution from multisatellite observations]. J. Hydrometeor., 2, 36-50 (DOI: 10.1175/1525-7541(2001)002<0036:GPAODD>2.0.CO;2).<br />
<br />
===GPCP on raijin=== <br />
<br />
We downloaded the daily product v1.2 and v1-3 which are available from the ua8 project<br />
<br />
* /g/data/ua8/GPCP/day/v1-#/<YYYY>/<files><br />
**v1-2'''<br />
Files are in netcdf4 format and filenames are<br />
* gpcp_1dd_v1.2_p1d.YYYYMMDD.nc<br />
where the date is the represented day<br />
For example:<br />
* gpcp_1dd_v1.2_p1d.199610.nc<br />
**v1-3'''<br />
Files are in netcdf4 format and filenames are<br />
* gpcp_v01r03_daily_dYYYYMMDD_cYYYYMMDD<br />
where the first date is the represented day and the second is the creation date for the file.<br />
For example:<br />
* gpcp_v01r03_daily_d20170323_c20171211</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=NASA-TRMM&diff=224NASA-TRMM2018-01-25T03:49:40Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><span style="font-family: Arial,Helvetica,sans-serif;">TRMM-TMPA is a 3 hours Level 3 product from the Tropical Rainfall Measuring Mission (TRMM 3B42) providing a rainfall estimate.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">This dataset is the output from the TMPA (TRMM Multi-satellite Precipitation) Algorithm, and provides precipitation estimates in the TRMM regions that have the (nearly-zero) bias of the ”TRMM Combined Instrument” precipitation estimate and the dense sampling of high-quality microwave data with fill-in using microwave-calibrated infrared estimates.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">The spatial resolution is 0.25ºx0.25º latitude--longitude (on a Cylindrical Equal Distance grid) , covering -180E to 180W and -50S to 50N degrees.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Current version is V7 and the dataset is published by Goddard Earth Sciences Data and Information Services Center (GES DISC).</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">A summary for the dataset is available on the [https://disc.gsfc.nasa.gov/uui/datasets/TRMM_3B42_7/summary | GESC DISC website]</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">A full technical description of this and other TRMM products can be found in the [https://pmm.nasa.gov/sites/default/files/document_files/3B4XRT_doc_V7_4_19_17.pdf | Real-Time TRMM Multi-Satellite Precipitation Analysis Data Set] documentation. From this document:</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">"... *3B42RT* is the official PPS identifier of the HQ+VAR data set. The identifier indicates that it is a level 3 (gridded) product with input from multiple sensors ("B") using non-TRMM data ("40"-series), running in Real Time.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">The *units of the TMPA-RT estimates* as stored in the data files is 0.01 mm/h for both precipitation and random error. To recover the original floating-point values in mm/h, divide by 100. The precipitation values are based on satellite snapshots. ...</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">The *period of record* for the TMPA-RT is 1 March 2000 through the present. The start is based on the start of the CPC merged 4 Km IR Tb data set. In contrast, the Version 7 TRMM product 3B42 provides after-real-time processing of the TMPA from 1 January 1998 to the delayed present. ..."</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif;">License</span>=== <br />
<span style="font-family: Arial,Helvetica,sans-serif;">Tropical Rainfall Measuring Mission (TRMM) (2011), TRMM (TMPA) Rainfall Estimate L3 3 hour 0.25 degree x 0.25 degree V7, Greenbelt, MD, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] [[https://disc.gsfc.nasa.gov/datacollection/TRMM_3B42_7.html]]</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Access to GES DISC data requires all users to be registered with the Earthdata Login system, even if we downloaded the data for you you should register. It is important for the data providers to have an estimate of users to justify the work that goes with providing the data services.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Complete terms of use and information on how to sign up are available on the [https://wiki.earthdata.nasa.gov/display/EL/Earthdata+Login+Overview+and+Policy%2C+v1.2 | GEOS-DISC data policy webpage] .</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif;">TRMM 3B42 on raijin</span>=== <br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Directory structure is:</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">/g/data1/ua8/NASA_TRMM/TRMM_L3/TRMM_3B42/<YEAR>/<files></span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Data format is HDF-EOS and filenames are</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;"> 3B42.<YYYYMMDD>.<HH>.<version>.HDF</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"> for example: 3B42.19980203.21.7.HDF</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">We are using the CRC32 checksum to check the files integrity, the script to download, update and check the files is available in the CMS github:</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">[https://github.com/coecms/CollectionsScripts/blob/master/NASA_TRMM/trmm-download-checks.py | trmm-download-checks.py]</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=MERRA2&diff=212MERRA22018-01-25T03:49:04Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>The Modern Era Retrospective-Analysis for Research and Applications (MERRA) is a NASA reanalysis integrating satellite observations using a major new version of the Goddard Earth Observing System Data Assimilation System Version 5 (GEOS-5).<br />
MERRA focuses on historical analyses of the hydrological cycle on a broad range of weather and climate time scales covering the modern satellite era from 1979 to present.<br />
The state-of-the-art GEOS-5 data assimilation system includes many modern observing systems (such as EOS) in a climate framework Specifically, the GEOS-DAS Version 5 implements Incremental Analysis Updates (IAU) to slowly adjust the model states toward the observed state.<br />
<br />
MERRA-2 is produced with the GMAO/GEOS-5 Data Assimilation System Version 5.12.4 and its native resolution is 0.5° x 0.625° x 72 hybrid sigma/pressure levels. Hourly data intervals are used for two-dimensional products, while 3-hourly intervals are used for three-dimensional products. These may be on the model’s native 72-layer vertical grid or at 42 pressure surfaces extending to 0.1 hPa.<br />
MERRA-2 spans from 1979 to present (with about 5 months lag).<br />
<br />
The <span style="background-color: #ffffff; color: #252525; font-family: sans-serif; font-size: 14px; text-align: -webkit-center;">42 pressure levels are: </span>1000, 975, 950, 925, 900, 875, 850, 825, 800, 775, 750, 725, 700, 650, 600, 550, 500, 450, 400, 350, 300, 250, 200, 150,100, 70, 50, 40, 30, 20,10, 7, 5, 4, 3, 2, 1,0.7, 0.5, 0.4, 0.3, 0.1 hPa<br />
The 72 native grid levels are:<br />
<br />
Information retrieved from [https://climatedataguide.ucar.edu/climate-data/nasa-merra | NCAR climate re-analysis guide] and<br />
[http://wiki.seas.harvard.edu/geos-chem/index.php/MERRA-2 | GEOS-CHEM MERRA-2 wiki]. The data was downloaded from the [https://disc.sci.gsfc.nasa.gov/information/glossary/581b59c65fa71421a68e64fa/merra-2?tags=mission | GES DISC data store] where<br />
information on all the MERRA-2 products is available.<br />
<br />
==='''License and Acknowledgements'''=== <br />
<span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">Access to GES DISC data requires all users to be registered with the Earthdata Login system, even if we downloaded the data for you you should register. It is important for the data providers to have an estimate of users to justify the work that goes with providing the data services.</span><br />
<span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">Complete terms of use and information on how to sign up are available on the [https://wiki.earthdata.nasa.gov/display/EL/Earthdata+Login+Overview+and+Policy%2C+v1.2 | GEOS-DISC data policy webpage] .</span><br />
Example of citation including the DOI, each product has its own (see below).<br />
Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 inst6_3d_ana_Np: 3d,6-Hourly,Instantaneous,Pressure-Level,Analysis,Analyzed Meteorological Fields V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/A7S6XP56VZWS<br />
==='''<span style="background-color: #ffffff; color: #252525; font-family: sans-serif; font-size: 14px;">MERRA-2 on raijin</span>'''=== <br />
<br />
We are hosting a copy of some of the MERRA-2 products on raijin in new "Global Observations and Re-analysis Datasets" group rr7, in collaboration with CAWCR.<br />
The data is available in <span style="font-size: 110%;">/g/data1/rr7/MERRA2/raw/<MERRA-product>/<year>/<files></span><br />
Filenames are <span style="font-size: 110%;"><runid>.<collection>.<timestamp>.nc4</span><br />
<br />
Last update on 24th of October 2016: updated to September 2016 included.<br />
<br />
We started with the following products, that will be regularly updated:<br />
* <span class="s1">'''tavg1_2d_rad_Nx'''</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">MERRA 2D,1-hourly,time-averaged,single-level,assimilation,radiation diagnostics</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">/M2T1NXRAD.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_rad_Nx.yyyymmdd.nc4</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">doi: 10.5067/Q9QMY5PBNV1T</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">file size: 209 Mb</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">citation: </span>Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_rad_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Radiation Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [<span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">Data Access Date] 10.5067/Q9QMY5PBNV1T</span><br />
* '''<span class="s1">tavg1_2d_flx_Nx</span>'''<br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">MERRA2 2D 1-hourly,time-averaged,single-level,assimilation,surface flux diagnostics</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">/M2T1NXFLX.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_flx_Nx.yyyymmdd.nc4</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">doi: 10.5067/7MCPBJ41Y0K6</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">file size: 380 Mb</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">citation: </span><span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_flx_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Surface Flux Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/7MCPBJ41Y0K6</span><br />
* '''<span class="s1">tavg1_2d_int_Nx</span>'''<br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">MERRA 2D ,1-hourly,time-averaged,single-level,assimilation,vertically integrated diagnostics</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">/M2T1NXINT.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_int_Nx.yyyymmdd.nc4</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">doi: 10.5067/Q5GVUVUIVGO7</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">file size: 1.3 Gb</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">citation: </span><span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_int_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Vertically Integrated Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/Q5GVUVUIVGO7</span><br />
* '''inst6d_3d_ana_Np'''<br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">MERRA 3D,6-hourly,instantaneous,pressure-level,analysis,analyzed meteorological fields</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">/M2I1NPANA.5.12.4/yyyy/mm/MERRA2_100.inst6_3d_ana_Np.yyyymmdd.nc4</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">resolution: lat/lon - 0.5 /0.625 degrees Timestep: 6 hourly Levels: 42</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">doi: 10.5067/A7S6XP56VZWS</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">file size: 511 Mb</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">citation: Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 inst6_3d_ana_Np: 3d,6-Hourly,Instantaneous,Pressure-Level,Analysis,Analyzed Meteorological Fields V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/A7S6XP56VZWS</span><br />
* '''tavg1_2d_slv_Nx'''<br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">MERRA 2D,1-hourly,Time-Averaged,Single-Level,Assimilation,Single-Level Diagnostics</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">/M2T1NXSLV.5.12.4/yyyy/mm/MERRA2_100.tavg1_2d_slv_Nx.yyyymmdd.nc4</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">resolution: lat/lon - 0.5 /0.625 degrees Timestep: 1 hourly</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">doi: 10.5067/VJAFPLI1CSIV</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">file size: 400 Mb</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">citation: </span>Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level Diagnostics V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [<span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">Data Access Date] 10.5067/VJAFPLI1CSIV</span><br />
* '''inst3d_3d_asm_Np'''<br />
** <span style="font-size: 90%;">MERRA 3D,3-hourly,Instantaneous,Pressure-Level,Assimilation,Assimilated Meteorological Fields</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">/M2I3NPASM.5.12.4/yyyy/mm/MERRA2_100.inst3_3d_asm_Np.yyyymmdd.nc4</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">resolution: lat/lon - 0.5 /0.625 degrees Timestep: 3 hourly Levels: 42</span><br />
** <span style="font-size: 90%;">doi: 10.5067/QBZ6MG944HW0</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">file size: 1.2 Gb</span><br />
** <span style="font-family: Arial,Helvetica,sans-serif; font-size: 11.7px;">citation: </span><span style="font-family: Arial,Helvetica,sans-serif; font-size: 90%;">Global Modeling and Assimilation Office (GMAO)(2015), MERRA-2 inst3_3d_asm_Np: 3d,3-Hourly,Instantaneous,Pressure-Level,Assimilation,Assimilated Meteorological Fields V5.12.4, Greenbelt, MD, USA, Goddard Earth Sciences Data and Information Services Center (GES DISC), Accessed [Data Access Date] 10.5067/QBZ6MG944HW0</span><br />
>> <br />
* '''<span style="font-size: 1.066em;">List of variables available for each product</span>'''<br />
[[media type="custom" key="28011971"]]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Leaving_the_Centre_guidelines&diff=202Leaving the Centre guidelines2017-11-29T00:24:45Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">When leaving the Centre talk to your supervisor or colleagues on how your storage space should be managed.</span><br />
<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">Questions to consider include</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">Will you require the files after you leave (e.g. as you finish papers)?</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">What files will be useful to others and should be published?</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">What files won't be useful to others and should be deleted?</span><br />
<br />
<span style="font-family: Arial; font-size: 12pt; vertical-align: baseline;">If you are leaving, or even if you are only changing position in the Center, one or more of the following might apply to you.</span><br />
<br />
**<span style="background-color: #ffffff; color: #333333; font-family: Arial; font-size: 11pt; vertical-align: baseline;">NCI account</span>'''<br />
<br />
<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">If you change institution but you are still working with the same collaborators/projects then your user-id will stay the same unless you (or your new institution) specifically ask for it to be changed.</span><br />
<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">NCI will suspend your account once you are no longer a member of any group. </span><br />
<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">Occasionally projects review their active members by sending e-mails: keep your contact e-mail updated through my.nci.org.au</span><br />
<br />
**<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">Leaving a project</span>'''<br />
<br />
<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">If you leave a specific project:</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">tidy up and document the files and directory structure and contact the lead-CI or project representative</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">If you haven’t already, set r-x group access to all your files</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">If the project representative agrees, ask help@nci.org.au to transfer ownership of your files to someone else in the project (specify the project and filesystem)</span><br />
* <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">If you want to transfer file externally</span><br />
** <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">use sftp, scp or rsync to transfer files securely ( rsync can be resumed )</span><br />
** <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">use the dedicated data-mover nodes, r-dm.nci.org.au for large file transfers</span><br />
** <span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">use copyq if you want to queue a job</span><br />
<br />
**<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">Leaving your institution</span>'''<br />
<br />
<span style="font-family: Arial; font-size: 11pt; vertical-align: baseline;">Most universities will close your university and e-mail account. Often university data services are accessible only via your university account. If that is the case, you need to arrange access for yourself by contacting the IT services or the CI of projects that you used to deposit data before leaving. Specifics on university data services and advice on what happens when you leave can be found </span><span style="color: #1155cc; font-family: Arial; font-size: 11pt; vertical-align: baseline;"><u><span style="text-decoration: none;">[[Data#x--University%20data%20policies%20and%20services | here]]</span></u></span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=How_to_use_MDSS_tape_storage_at_NCI&diff=231How to use MDSS tape storage at NCI2017-11-29T00:20:57Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation and can be checked using the nci_account command.<br />
<br />
There is a comprehensive NCI guide to using MDSS<br />
<br />
===MDSS proper usage=== <br />
<br />
MDSS is designed for medium to long-term archive of large files, so it is suitable for<br />
<br />
* Files you are required to keep, for example model outputs or configurations from published datasets, publications, PhD thesis etc.<br />
* Files that you or someone else are likely to reuse or analyse again in the future but not in the next few months. For example restart files or other model output you are not immediately using should be moved from disk to mdss as soon as possible.<br />
<br />
===Guidelines for storage=== <br />
<br />
* Big files: if your files are small in size (less than 20Mb) then use tools like tar to bundle them into a single archive file<br />
* Files should be group readable, with group execute permissions for directories. This helps with long term maintenance, allowing administrators to track the type and size of archived data.<br />
<br />
MDSS is not a backup service and it is not suitable for code files you might want to keep, for this you might prefer to use online services as Github or Bitbucket.<br />
<br />
===Accessing MDSS=== <br />
Massdata cannot be accessed directly via a directory path.<br />
All access of MDSS is via the command mdss<br />
mdss -P project-id + command (ls, put, get ..)<br />
See mdss -help to get a full list of the subcommands and<br />
Mdss --help subcommand to get specific help<br />
Please note mdss commands work only interactively or with ‘copyq’<br />
* Users connected to the project have rwx permissions in that directory and so may create their own files in those areas.<br />
<br />
===Preparing your data for mdss=== <br />
<br />
# Organise your files and delete anything which you won’t be re-using. It is tempting to copy entire directories as they are to mdss thinking you’ll be getting back to them again later. There is currently no easy way to list what you are storing on massdata and so trying to tidy up after you uploaded your files would be slow and painful. Even more than with other storage options, it is really important to put there only suitable files and make sure that they have been compressed and tarred together if necessary<br />
# NCI guidelines suggest a minimum size of 20MB per file and an average size of 250MB.<br />
<br />
While you are preparing your data to be moved it is an opportunity to also document, if you haven’t done so already, what you are archiving and how. Even a simple readme file added to your main directory can help others and your future self.<br />
If you are archiving data underlying a publication or published dataset then it is important a summary of what is stored in /massdata and how is part of the dataset management plan.<br />
<br />
Useful tools:<br />
[https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing | TAR - to create archives]<br />
[[NetCDF Compression Tools | Compressing tools]]<br />
<br />
===Monitoring mdss usage=== <br />
<br />
nci_account -P <project-id> will give an account of the total massdata allocation, usage and availability both as size and i-nodes (i.e. number of files and directories)<br />
<syntaxhighlight><br />
Usage Report: Project=w35 Storage Period=2017.9 (01/07/2017-30/09/2017)<br />
=======================================================================<br />
-------------------------------------------------------------------------------------------------<br />
System StoragePt Grant Usage Avail iGrant iUsage iAvail<br />
-------------------------------------------------------------------------------------------------<br />
dmf massdata 4048.00GB 1209.00GB 2839.00GB 323.00K 8.88K 314.12K<br />
global gdata1 76.00TB 59.69TB 16.31TB 3883.00K 2299.01K 1583.99K<br />
global gdata1a 76.00TB 60.47TB 15.53TB 3883.00K 2313.58K 1569.42K<br />
raijin short 15.00TB 7.54TB 7.46TB 3280.00K 2767.07K 512.93K<br />
-------------------------------------------------------------------------------------------------<br />
Total 170.95TB 128.88TB 42.08TB 11.37M 7.39M 3.98M<br />
</syntaxhighlight><br />
Unfortunately there is not a command to check quickly usage by user-id as for /g/data and /short.<br />
The only way to get this information currently is to ask help@nci.org.au, administrators can access this information for any CI of the group.<br />
<br />
===Transferring data to and from MDSS=== <br />
NCI supports different commands to work with MDSS as it is explained on their User Guide. The CMS team has also developed a utility called mdssdiff. This utility allows users to compare the contents of the local directory and a directory under /massdata. It will also recursively update the content on the massdata directory to copy the local directory or vice versa.<br />
<br />
===Modifications to MDSS datasets=== <br />
Contact NCI at help@nci.org.au if large metadata operations are needed on massdata, as changing ownership, project code, permissions etc. of existing datasets<br />
<br />
[[include component="page" wikiName="climate-cms" page="nci_guidelines_index" editable="1"]]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=UNSW_data_requirements_and_tools&diff=370UNSW data requirements and tools2017-11-28T00:16:29Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>'''Archive data at UNSW'''<br />
UNSW has just recently launched a new website that allows students and researchers to managed and store their data completely through an online workflow.<br />
First step is the submission of a Research Data Management Plan (RDMP) and then you can upload your data directly on the [https://rds.unsw.edu.au/ | RDS web portal] or by sftp or using a command line, depending on size and complexity of data. Data can then be accessed by the web portal. All the necessary information, links and video tutorials are available on the <span class="s1">[http://www.dataarchive.unsw.edu.au/ | UNSW data archive services] website.</span><br />
<br />
**<span class="s1">Student requirements in regard to data storing</span>'''<br />
<br />
<br />
<span class="s1">ARC funding rules now have various stipulations about correct handling of data and collection and storage of metadata. The UNSW ResData system provides a means to do this. As such, all supervisors of PhD students probably ought to have RDMPs linked to their grants. If so, there’s an option to link plan/s to any students they supervise if the data is shared. </span><br />
<br />
<span class="s1">HDR students can also set up their own accounts and data plans. It’s a discussion best had between students and supervisors as to whether it’s beneficial for students to have their own separate plan e.g. If you the student is working with and/or generating data that is quite distinct to the supervisor’s. </span><br />
<br />
<span class="s1">In the meantime, it is not detrimental to upcoming PG reviews to tick “no” to the question about RDMPs, but it should be a prompt to look into this (staff and students) as it is an emerging compliance issue.</span><br />
<br />
**<span class="s1">When you leave</span>'''<br />
<br />
<span style="color: #333333; font-family: Arial,Helvetica,'Nimbus Sans L',sans-serif; font-size: 18px;">You need a valid and active zID and zPass to access and use the UNSW Data Archive. If you leave UNSW you can gain access via the <span style="font-family: Arial,Helvetica,;">Lead Chief Investigator (LCI) and the Research Project Manager as defined in the Research Data Management Plan (RDMP) </span></span><br />
<span style="color: #333333; font-family: Arial,Helvetica,'Nimbus Sans L',sans-serif; font-size: 18px;">If the Lead Chief Investigator (LCI) and the Research Project Manager have also left UNSW, access to the project can be gained through the Head of School.</span><br />
<span style="color: #333333; font-family: Arial,Helvetica,'Nimbus Sans L',sans-serif; font-size: 18px;">You should contact your local IT support or the UNSW IT Service Centre to arrange access to data stored in the Data Archive. </span><br />
<br />
<span class="s1">'''Known issues with the transfer to data storage''' </span><br />
<br />
<span class="s1">Some users have reported having trouble using the sftp command line option to transfer data to the UNSW Data Storage.</span><br />
<span class="s1">The official documentation is following the instruction for the [http://www.dataarchive.unsw.edu.au/help/sftp-client-guide#UploadDownload | sftp-client-guide]</span><br />
<br />
<span class="s1">There are two issues we've been told about: </span><br />
<span class="s1">1) If you're trying to connect from one of the CCRC server like Storm or Maelstrom using the command in the instructions</span><br />
<br />
<span class="s1"> sftp -oPort=8022 -r <span class="s2">UNSW_RDS:<z-user-id>@rds.unsw.edu.au</span></span><br />
<br />
<span class="s1"><span class="s2">you will get the following error message:</span></span><br />
<br />
<span class="s3"> sftp: illegal option -- r</span><br />
<span class="s3"> usage: sftp [-1Cv] [-B buffer_size] [-b batchfile] [-F ssh_config]</span><br />
<span class="s3"> [-o ssh_option] [-P sftp_server_path] [-R num_requests]</span><br />
<span class="s3"> [-S program] [-s subsystem | sftp_server] host</span><br />
<span class="s3"> sftp [user@]host[:file ...]</span><br />
<span class="s3"> sftp [user@]host[:dir[/]]</span><br />
<span class="s3"> sftp -b batchfile [user@]host</span><br />
<br />
<span class="s3">This is because the servers are using an old version of sftp that doesn't have the -r flag.</span><br />
<span class="s3">The -r flag is used to transfer files recursively, so you can transfer entire directories in one go.</span><br />
<span class="s3">If you omit this flag than you shouldn't have any issues connecting. If you need to transfer lots of files alternative option are listed below.</span><br />
<br />
<span class="s1">2) Depending on the ssh/sftp and unix/linux distribution you're using you might experience a bug that will stop you connecting to the server.</span><br />
<span class="s1"> A couple of users experienced this when trying to connect from their laptop/desktops</span><br />
<span class="s1">Here's an example:</span><br />
<br />
sftp -oPort=8022 -r <span class="s2">UNSW_RDS:<z-user-id>@rds.unsw.edu.au</span><br />
<span class="s1"> hash mismatch</span><br />
<span class="s1"> key_verify failed for server_host_key</span><br />
<span class="s1"> Couldn't read packet: Connection reset by peer</span><br />
<br />
The IT services are working to solve this and the new release is scheduled for the 17th-18th of November.<br />
If you can't or don't want to wait you still have other options to transfer the data. As well as the web interface, which should work just fine from your own computer, you can use filezilla or lftp<br />
<br />
lftp is actually available on the "storm" server and can be used in the following way<br />
<br />
<span class="s1">lftp -p 8022 s<span class="s2">[[ftp://rds.unsw.edu.au]]</span></span><br />
<span class="s3">Then at the lftp prompt:</span><br />
<span class="s3">user UNSW_RDS:<your zID></span><br />
<span class="s3">You are then prompted for your password.</span><br />
<span class="s3">Then you can transfer data. To upload:</span><br />
<span class="s3">mirror –R <local dir> <archive dir></span><br />
<br />
<span class="s3">To download:</span><br />
<span class="s3">mirror <archive dir> <local dir></span><br />
<br />
<span class="s1">NB you need to put a / after the destination directory if you want it to create the source directory in the target, otherwise it puts the contents of the source in the target</span><br />
<br />
NB lftp can have issues when using ls which cause lftp to hang!!<br />
Here: [[http://www.mail-archive.com/lftp@uniyar.ac.ru/msg03949.html]] there's a suggestion of using "ftp" instead of "sftp"<br />
ie<br />
lftp -p 8022 <span class="s2">[[ftp://rds.unsw.edu.au]]</span><br />
The IT service responsible for the data storage is testing other ways to use lftp, which has a lot of available options. Finally he can help anyone who needs a more complex data transfer to be automated to use set up a java script which is currently available on storm. If you're interested let me know (Paola) and I will put you in contact. Or send an e-mail to their helpdesk (details should be in their documentation).</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=NOAA_OISST&diff=257NOAA OISST2017-07-04T03:23:42Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><span style="color: #444444; font-family: Arial,Helvetica,sans-serif;">The NOAA 1/4° daily Optimum Interpolation Sea Surface Temperature (or daily OISST) is an analysis constructed by combining observations from different platforms (satellites, ships, buoys) on a regular global grid. A spatially complete SST map is produced by interpolating to fill in gaps. More information from the [https://www.ncdc.noaa.gov/oisst | NOAA website].</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">We have a local copy, updated weekly of the AVHRR-Only OISST, which uses infrared satellite data from the Advanced Very High Resolution Radiometer (AVHRR).</span><br />
<br />
The data covers from September 1981 till present, with daily temporal resoultion.<br />
<br />
The scripts used to download and concatenate the data are available from the [https://github.com/coecms/CollectionsScripts/tree/master/OISST | CMS github] .<br />
<br />
<span style="color: #ee3c10;">NOAA is changing the delivery mode of this dataset and also switching to netcdf4 format: "data from 1 April 2017 onward are in netCDF4 (with internal compression, so files will not be named *.gz) format. Older files are still netCDF3, until further notice."</span><br />
<span style="color: #ee3c10;">I'm currently reviewing our download procedure and we'll finally have a new raw and modified version for this data.</span><br />
<span style="color: #ee3c10;">To help transitioning the v2-0_modifed version will stay in place.</span><br />
<br />
===License=== <br />
This dataset is freely available, there are no use limitations or accounts needed. Although there is an optional registration form that can be reached from the documentation page in the NOAA website under the "data access" session. Please consider register, so you'll receive updates on the data and as this also helps the data publishers to get an idea of the data usefulness and continue providing for public access.<br />
Use the following publications to cite the dataset:<br />
<br />
<span style="color: #444444; font-family: 'Open Sans',Helvetica,Verdana,Arial,sans-serif; font-size: 15.2152px; vertical-align: baseline;">Banzon, V., Smith, T. M., Chin, T. M., Liu, C., and Hankins, W., 2016: A long-term record of blended satellite and in situ sea-surface temperature for climate monitoring, modeling and environmental studies. ''<span style="vertical-align: baseline;">Earth Syst. Sci. Data</span>'', '''<span style="vertical-align: baseline;">8</span>''', 165–176, <span style="color: #0057a5; vertical-align: baseline;">[http://www.earth-syst-sci-data.net/8/165/2016/essd-8-165-2016.html | doi:10.5194/essd-8-165-2016(link is external)]</span>.</span><br />
<span style="color: #444444; font-family: 'Open Sans',Helvetica,Verdana,Arial,sans-serif; font-size: 15.2152px; vertical-align: baseline;">Reynolds, R. W., T. M. Smith, C. Liu, D. B. Chelton, K. S. Casey, and M. G. Schlax, 2007: Daily high-resolution-blended analyses for sea surface temperature. ''<span style="vertical-align: baseline;">Journal of Climate</span>'', '''<span style="vertical-align: baseline;">20</span>''', 5473–5496, <span style="color: #0057a5; vertical-align: baseline;">[http://dx.doi.org/10.1175/2007JCLI1824.1 | doi:10.1175/JCLI-D-14-00293.1(link is external)]</span>.</span><br />
<br />
===OISST on raijin=== <br />
<br />
This dataset is available on raijin under the ua8 project (NB this might change in the near future because of the NCI data re-organisation).<br />
The original data is located in<br />
<br />
<span class="s1">/g/data1/ua8/NOAA_OISST/AVHRR/raw/<year>/<files.gz></span><br />
<br />
<span class="s1">NB these files are zipped as when downloaded.</span><br />
<br />
<span class="s1">The files are then concatenated for ease of use into yearly files, the resulting modified dataset is available under</span><br />
<br />
<span class="s1">/g/data1/ua8/NOAA_OISST/AVHRR/v2-0_modified/<files></span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Data_Management_Plan&diff=122Data Management Plan2017-04-14T02:23:59Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>[[media type="custom" key="27347906"]]<br />
<br />
Some tips about research papers that can be applied to DMP too:<br />
<br />
...<span style="background-color: #fbfbf9; color: #04060f; font-family: Corsiva; font-size: 12pt; vertical-align: baseline;">Pitfalls include using complicated jargon, including unnecessary details, and writing for your highly specialized colleagues instead of a wider audience. ...</span><br />
<br />
<span style="background-color: #fbfbf9; color: #04060f; font-family: Syncopate; font-size: 12pt; vertical-align: baseline;">Only abbreviations firmly established in the field are eligible, … avoiding those which are not broadly used ...</span><br />
<br />
<span style="background-color: #fbfbf9; color: #0294a5; font-family: 'Trebuchet MS'; font-size: 12pt; vertical-align: baseline;">… when looking for keywords, avoid words with a broad meaning and words already included in the title.</span><br />
<br />
<span style="background-color: #ffffff; color: #505050; font-family: 'Comic Sans MS'; font-size: 12pt; vertical-align: baseline;">… you need to include detailed information so a knowledgeable reader can reproduce the experiment. However, do not repeat the details of established methods; use References and Supporting Materials to indicate the previously published procedures.</span><br />
<br />
<span style="background-color: #ffffff; color: #505050; font-family: Calibri; font-size: 12pt; vertical-align: baseline;">… indicate uses and extensions if appropriate. Moreover, you can suggest future experiments and point out those that are underway ...</span><br />
<br />
<span style="background-color: #ffffff; color: #505050; font-family: 'Courier New'; font-size: 12pt; vertical-align: baseline;">Effective research articles are interesting and useful to a broad audience, including scientists in other fields.</span><br />
<br />
<span style="background-color: #ffffff; color: #505050; font-family: Impact; font-size: 14pt; vertical-align: baseline;">use standard systems … conventions</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Why_manage_data&diff=386Why manage data2017-04-14T01:43:13Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>==<span style="background-color: #ffffff; font-size: 14px;">Why should I manage my data output?</span>== <br />
<br />
===<span style="font-size: 14px;">In your own interest:</span>=== <br />
* <span style="font-size: 14px;">making sure your data is safe</span><br />
* <span style="font-size: 14px;">making sure you produce good quality data</span><br />
* <span style="font-size: 14px;">sharing your data increases your chance of being cited and hence your research has more impact in your field and potentially in other research areas too</span><br />
* <span style="font-size: 14px;">data and software you produce is starting to count more when you apply for grant or a new job, funding agencies are looking at ways to include it in their metrics</span><br />
* <span style="font-size: 14px;">avoid wasting time down the track to fix errors and collect last minute information when you need to publish your results</span><br />
* <span style="font-size: 14px;">making it easier for other researchers to collaborate with you and for technical staff to help you</span><br />
* <span style="font-size: 14px;">over time improving your knowledge of data management itself will make it easier to comply with new requirements from funding agencies and universities</span><br />
===<span style="font-size: 14px;">For our sake:</span>=== <br />
* <span style="font-size: 14px;">Keeping track of what data is being produced helps researchers to share data and procedures effectively inside the ARCCSS, one of the main reason to unite climate researchers in a centre of excellence across institutions was to aid collaboration across filed and institutions.</span><br />
* <span style="font-size: 14px;">It helps the CMS team to keep track of which datasets and models are worth maintaining, which tasks should be prioritised. Knowing what you're working on, helps us planning and managing our limited time and resources in a more efficient way.</span><br />
* <span style="font-size: 14px;">Your data output is an asset for your own institution and for the Centre, it helps showing that we are doing a good job as much as journal publications do</span><br />
<br />
===<span style="font-size: 14px;">Finally DMPs and sharing data are becoming a requirement by funding agencies and journal publishers:</span>=== <br />
* <span style="font-size: 14px;">The ARC Open Access policy that now applies only to publications will soon be extended to data output. While a DMP is not compulsory yet, it will add weight to your application. And from this year also the use of a researcher ID is encouraged, a researcher ID offers a single link to all your papers, cv, research projects and data and software output. From the ARC funding rules:Recently journal editors have updated their data policy and now require that data relating to the submitted paper should be made available by the authors. AGU, Nature, PLOS are examples of publishers that are already requesting the data to be published. We are helping our staff and students by creating a record for their data on the ANDS Research Data Australia repository, and serving their data as part of an ARCCSS collection on the NCI web data services. To do this we need information which is already collected in a DMP, so having a DMP will help you being ready for this most important part of your research project.</span><br />
** A11.5.2 Researchers and institutions have an obligation to care for and maintain research data in accordance with the NHMRC/ARC/UA Australian Code for the Responsible Conduct of Research (2007). The ARC considers data management planning an important part of the responsible conduct of research and strongly encourages the depositing of data arising from a Project in an appropriate publically accessible subject and/or institutional repository.<br />
** A11.5.3 The ARC encourages all researchers applying for funding to have an ORCID identifier.<br />
** [http://www.arc.gov.au/research-data-management | ARC Research Data Management]<br />
* <span style="font-size: 14px;">A DMP helps satisfy NCI that the allocated cpu time and/or storage are being efficiently used, which can help future allocation requests. The same is true also if you need to use some of the NECTAR or RDSI (Research Data Infrastructure Initiative) resources.</span><br />
* <span style="font-size: 14px;">most Australian universities are introducing compulsory data management plans.</span><br />
<br />
<span style="font-size: 14px;">'''Common concerns'''</span><br />
<br />
* <span style="color: #04060f; font-family: Calibri; font-size: 12pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">My data could be misused or misinterpreted</span><br />
> </span><br />
** <span style="color: #04060f; font-family: Calibri; font-size: 14pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">provide clear information </span><br />
>> </span><br />
* <span style="color: #04060f; font-family: Calibri; font-size: 12pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">My data is too complicated</span><br />
> </span><br />
** <span style="color: #04060f; font-family: Calibri; font-size: 14pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">document and organise it properly</span><br />
>> </span><br />
* <span style="color: #04060f; font-family: Calibri; font-size: 12pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">My data is not very interesting! </span><br />
> </span><br />
** <span style="color: #04060f; font-family: Calibri; font-size: 14pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">you never know how your data can be re-used and for whom will be interesting</span><br />
>> </span><br />
* <span style="color: #04060f; font-family: Calibri; font-size: 12pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">I want to use my data and write a paper</span><br />
> </span><br />
** <span style="color: #04060f; font-family: Calibri; font-size: 14pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">use an embargo and start by publishing the data you are not using </span><br />
>> </span><br />
* <span style="color: #04060f; font-family: Calibri; font-size: 12pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">I’m not sure if I own the data</span><br />
> </span><br />
** <span style="color: #04060f; font-family: Calibri; font-size: 14pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">let’s find out</span><br />
>> </span><br />
* <span style="color: #04060f; font-family: Calibri; font-size: 12pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">People will contact me to ask about it</span><br />
> </span><br />
** <span style="color: #04060f; font-family: Calibri; font-size: 14pt; vertical-align: baseline;"><span style="font-size: 14pt; vertical-align: baseline;">Not if it is well documented, … maybe if they want to collaborate</span><br />
>> </span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=UTAS_data_requirements_and_tools&diff=382UTAS data requirements and tools2017-04-13T23:54:59Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>===<span style="background-color: #ffffff;">Data management</span>=== <br />
<br />
<span style="background-color: #ffffff;">UTAS library website provides guidelines on data management.</span><br />
It also provides [https://vivo.utas.edu.au/vivo/ | VIVO] as a metadata publishing tool to create metadata records on ANDS RDA.<br />
<br />
===<span style="background-color: #ffffff;"><span id="x--Data storage and publishing"></span>'''Data storage and publishing'''</span>=== <br />
<br />
UTAS is still working towards providing one data storage for all the university. Since most of the researchers and students at UTAS affiliated to the ARCCSS are part of IMAS, you can use the [http://data.imas.utas.edu.au/static/landing.html | IMAS Data Portal] . The portal provides a data submission tool to gather the necessary metadata. There's a limit to the file size you can directly upload but you can request to host bigger files which will be uploaded by a portal manager.<br />
<br />
===<span style="background-color: #ffffff;"><span id="x--When you leave"></span><span style="vertical-align: baseline;">When you leave</span></span>=== <br />
<span style="background-color: #ffffff; vertical-align: baseline;">I couldn't find specific answers to what happen to your access options when you don't hold anymore an account. As for others university it might be better to manage data on University repositories as a group or project, so you can always ask someone from the group to help you retrieving the data or managing the records. If your data has public access then you should be able to retrieve it from the IMAS portal.</span><br />
<span style="background-color: #ffffff; vertical-align: baseline;">Below are the requirements about your "working data" set by the university. </span><br />
<span style="background-color: #ffffff; vertical-align: baseline;">“ </span><span style="background-color: #ffffff; color: #3a3a3a; font-family: 'Open Sans',Arial,Helvetica,sans-serif;">Before leaving the University, you should arrange access for at least one other researcher or your Head of School or Research Centre to the data and any documentation relating to it. Master copies of any working data that belongs to the University or to a third party with which the University has an agreement must not be removed from the University.</span><span style="background-color: #ffffff; vertical-align: baseline;">”</span><br />
<br />
===<span style="background-color: #ffffff;"><span id="x--e-research services"></span>'''e-research services'''</span>=== <br />
<br />
UTAS relies on [http://www.tpac.org.au/rdsi-research-data-collection-application-2/ | TPAC] for e-research services in general and to store bigger data collections.<br />
TPAC is also responsible locally for the NeCTAR cloud services and have its own [http://portal.sf.utas.edu.au/thredds/catalog.html | THREDDS server data portal]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ANU_data_requirements_and_tools&diff=54ANU data requirements and tools2017-04-13T23:30:25Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>===<span style="background-color: #ffffff;">Data management</span>=== <br />
<br />
<span style="background-color: #ffffff;">ANU library provides [http://libguides.anu.edu.au/c.php?g=465056&p=3180073 | Data Management guidelines] on their website. Metadata for dataset is collected via ANU Data Commons.</span><br />
<span style="background-color: #ffffff;">They have a two steps online template that you can fill in. In the second step you can attached documents related to your data as a DMP.</span><br />
<span style="background-color: #ffffff;">The [https://services.anu.edu.au/files/DataManagement.pdf | Data Mangement Manual] provides all details.</span><br />
<br />
===<span style="background-color: #ffffff;"><span id="x--Data storage and publishing"></span>'''Data storage and publishing'''</span>=== <br />
[https://datacommons.anu.edu.au/DataCommons/ | ANU Data Commons]<span style="background-color: #ffffff;"> is the </span>ANU’s repository for long-term storage and dissemination of data.<br />
<br />
====== <br />
===<span style="background-color: #ffffff;"><span id="x--When you leave"></span><span style="vertical-align: baseline;">When you leave</span></span>=== <br />
<span style="background-color: #ffffff; vertical-align: baseline;">I couldn't find yet specific information on this. I suppose that anything that requires a university account access won't be accessible anymore directly. If your data is managed as a project data you might be able to retrieve it going through someone else in the project unless the access is public.</span><br />
<br />
===<span style="background-color: #ffffff;"><span id="x--e-research services"></span>'''e-research services'''</span>=== <br />
For other services and bigger storage options ANU relies on NCI data services'''.'''<br />
ANU provides [https://alliance.anu.edu.au/welcome/ | Alliance] as an online collaboration environment. It can be accessed through any web browser and provides tools such as a wiki, forums, and calendar. All ANU staff and students can log into Alliance and create a ‘project’. You can only upload files up to 25 Mb.</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Monash_data_requirements_and_tools&diff=217Monash data requirements and tools2017-04-12T03:45:52Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>===<span style="background-color: #ffffff;">Data management</span>=== <br />
<br />
<span style="background-color: #ffffff;">Monash has a [https://www.monash.edu/library/researchdata | data management section] on their website with lots of information about data management including [https://www.monash.edu/library/researchdata/guidelines/data-planning#Data | checklists] for researchers and students.</span><br />
<br />
===<span style="background-color: #ffffff;"><span id="x--Data storage and publishing"></span>'''Data storage and publishing'''</span>=== <br />
<span style="background-color: #ffffff; color: #000000;">[http://research.unimelb.edu.au/infrastructure/doing-data-better/figshare | Monash Figshare]</span><span style="background-color: #ffffff;"> is the tool used to store and share data, as well DOIs can be created for the datasets. Storage is limited initially to 10 Gb but you can request more.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">You will need a staff Monash username and password and staff email account to use monash.figshare. All graduate research students can apply for a staff account through their supervisors.</span><br />
<br />
===<span style="background-color: #ffffff;"><span id="x--When you leave"></span><span style="vertical-align: baseline;">When you leave</span></span>=== <br />
<span style="background-color: #ffffff; vertical-align: baseline;">“ ... </span>If the project has been set up with group allocation of storage, the ownership of files uploaded by a project member will transfer to the project manager allowing the project manager full access to modify the files.<br />
<br />
If the project has been set up with individual allocation of storage, the ownership of the files stays with the project member meaning they take their files with them when they leave the project. ...<span style="background-color: #ffffff; vertical-align: baseline;">”</span><br />
The figshare administrators advise is to always set up a group project, if you can, rather than an individual one. This is as also the advantage that you can still use your individual one for other files as well as making it easier to be allocated more resources.<br />
I'm assuming (but I need to verify this) that anything which is "published" and has a DOI assigned will still be available, although you might not be able to manage it yourself.<br />
===<span style="background-color: #ffffff;"><span id="x--e-research services"></span>'''e-research services'''</span>=== <br />
<br />
<span style="background-color: #ffffff;">VicNode is a joint venture between Monash University and The University of Melbourne on behalf of all Victorian universities aimed at providing storage, sharing and long-term retention of research data for all researchers.</span><br />
<span style="background-color: #ffffff;">They use the </span><span style="background-color: #ffffff; color: #000000;">[https://vicnode.org.au/2016/05/11/mediaflux-service-available-at-vicnode-university-of-melbourne-operating-centre/ | Mediaflux service]</span><span style="background-color: #ffffff;"> to manage data </span><br />
<span style="background-color: #ffffff;">You can check more information and keep updated on similar services visiting the </span><span style="background-color: #ffffff; color: #000000;">[http://blogs.unimelb.edu.au/researchplatforms/ | [[https://platforms.monash.edu/eresearch/ | Monash eResearch Centre]]</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Melbourne_University_data_requirements_and_tools&diff=211Melbourne University data requirements and tools2017-04-12T02:49:56Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>===Data management=== <br />
<br />
Melbourne University provides a program training to guide researchers and students on creating a DMP for their research data.<br />
<span style="vertical-align: baseline;">[http://library.unimelb.edu.au/Digital-Scholarship/training_and_outreach/data | Managing Data@Melbourne] is the university data management program training. It consists of 6 short modules which will guide you through drafting your data management plan</span><br />
<br />
==='''<span style="vertical-align: baseline;">Data storage and publishing</span>'''=== <br />
[http://research.unimelb.edu.au/infrastructure/doing-data-better/figshare | Figshare] is the tool used to store and share data, as well DOIs can be created for the datasets. Storage is limited to 100 Gb and it’s not available for PhD students (it should soon)<br />
<br />
===<span style="vertical-align: baseline;">When you leave</span>=== <br />
<span style="vertical-align: baseline;"> “The institutional figshare service will end on the same day you leave the University. Everything that has been published through figshare will be permanently available and the DOIs will be maintained and managed by the University. Any unpublished files and data that you wish to keep must be exported or transferred to an alternative storage solution before your end date at the University.”</span><br />
<br />
==='''<span style="vertical-align: baseline;">e-research services</span>'''=== <br />
<br />
VicNode is a joint venture between Monash University and The University of Melbourne on behalf of all Victorian universities aimed at providing storage, sharing and long-term retention of research data for all researchers.<br />
They use the [https://vicnode.org.au/2016/05/11/mediaflux-service-available-at-vicnode-university-of-melbourne-operating-centre/ | Mediaflux service] to manage data <br />
You can check more information and keep updated on similar services on the [http://blogs.unimelb.edu.au/researchplatforms/ | research platforms blog]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=DATA_NEWS&diff=125DATA NEWS2016-05-24T01:18:16Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>- The ARC has added a new [http://www.arc.gov.au/research-data-management | Research Data Management] page to their website<br />
<br />
- CMIP5 update: most of the ESGF nodes are back online, still missing are NCI and the Chinese nodes.<br />
<br />
- CMIP5: the [http://arccssive.readthedocs.org/en/latest/CMIP5.html#examples | ARCCSSive] module has now a new stable version and the database has been updated to include all the data replicated on raijin, both by users and by NCI as a bulk transfer.<br />
<br />
- There is a new [https://opus.nci.org.au/display/CMIP/CMIP+Community+Home | CMIP community page] on NCI confluence, for information and updates relating to the CMIP data and activities at NCI. This includes regular updates on CMIP6 progress. Anyone belonging to ua6 can access and also edit or add comments to some of the pages. This is also the right way to communicate issues with the data and its management on NCI.<br />
<br />
- We are developing a new python module to help accessing CMIP5 data, [http://arccssive.readthedocs.org/en/latest/CMIP5.html#examples | ARCCSSive],<br />
please give it a go and send us feedback.<br />
<br />
===<span style="color: #0f0f0e; font-family: Arial,Helvetica,sans-serif;">- CMIP5 update - ESGF nodes will be down end of January and citation info</span>=== <br />
<br />
<span style="color: #f52306; font-family: Arial,Helvetica,sans-serif;">[[CMIP5_announcement | See Karl Taylor e-mail to CMIP5 users]]</span><br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif;">- New data management plans requirements and tools for UNSW students and researchers</span>=== <br />
<span style="font-family: Arial,Helvetica,sans-serif;"> from Stephen Gray e-mail </span><br />
<br />
''Dear all, (especially students with upcoming formal reviews)''<br />
''There is now a question on the online PG review form that asks students to indicate if they have a Research Data Management Plan (RDMP) associated with their project. Information about what this is and what it means (among other things, access to archival storage which will help clear space on CCRC computational servers…) can be found here: [[https://resdata.unsw.edu.au/pages/authenticate.faces]]//<br />
''ARC funding rules now have various stipulations about correct handling of data and collection and storage of metadata. The UNSW ResData system provides a means to do this. As such, all supervisors of PhD students probably ought to have RDMPs linked to their grants. If so, there’s an option to link plan/s to any students they supervise if the data is shared.''<br />
''HDR students can also set up their own accounts and data plans. It’s a discussion best had between students and supervisors as to whether it’s beneficial for students to have their own separate plan e.g. If you the student is working with and/or generating data that is quite distinct to the supervisor’s.''<br />
''In the meantime, it is not detrimental to upcoming PG reviews to tick “no” to the question about RDMPs, but it should be a prompt to look into this (staff and students) as it is an emerging compliance issue.''<br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">We are in the process of preparing a wiki page with more details about this process and how fits with our own Center requirements.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">You can access that from the main data wiki page.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">- NASA seminar on the new GIOVANNI interface [https://events-na8.adobeconnect.com/content/connect/c1/1136900108/en/events/event/shared/1563480278/event_landing.html?sco-id=1565675436&_charset_=utf-8 | Redesigning Giovanni: More Data, More Plots, Faster Results]</span><br />
<span class="s1">'''When:''' Wednesday 6 May 2015, 02:00 PM - 03:00 PM</span><br />
<br />
<span class="s1">'''Time Zone:''' Eastern Daylight Savings Time (UTC/GMT-4)</span><br />
<br />
- UTAS based PhD student Amelie Meyer developed the "[http://www.mathworks.com.au/matlabcentral/fileexchange/47595-mixing--mx--oceanographic-toolbox-for-em-apex-float-data | Mixing (MX) Oceanographic Toolbox for EM-APEX float data]" with ARCCSS funding.<br />
The Mixing Oceanographic Toolbox provides a framework to estimate the dissipation rate and diffusivity from Electromagnetic Autonomous Profiling Explorer (EM-APEX) float observational data. The EM-APEX floats measure the temperature, salinity, pressure, and horizontal velocity of the current. Vertical gradients of velocity and buoyancy are estimated and a finescale parameterization is applied. This method provides order of magnitude estimates of mixing as well as estimates of the regional variability of mixing.<br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif;">- '''ANDS Events'''</span>=== <br />
Have regularly a look at the [http://ands.org.au/events/index.html | ANDS events webpage].<br />
The Australian National Data Service offers frequent webinars and other online events to help researchers with their data management issues. If this sounds like something which doesn't interest you, then maybe you should start from the next one coming:<br />
====<span class="event-when">Thu, August 7, 12:30pm – 1:30pm '''Can sharing your research data raise your research profile?'''</span>==== <br />
<span style="background-color: #ffffff; display: block; font-family: Arial,sans-serif; font-size: small;"><span class="event-description">Research data is a valuable asset that can be a used to raise your research profile, increase your citation rates and identify opportunities for collaboration. Dr Anita Smyth is a researcher and Data Facilitator with the Terrestrial Ecosystem Research (TERN) Eco-informatics Facility. In this webinar, Anita will talk about the growing body of evidence that suggests social media is an effective way to raise your research profile. Learn about the benefits of using social media to encourage reuse of your research outputs and identify collaboration opportunities. Understand the relationship between social media and “altmetrics” (or alternative metrics) as an indicator of research impact and reach. There will be plenty of time for your questions.</span></span><br />
If you miss it you can always have a look at this and past webinars on youtube [[http://www.youtube.com/user/andsdata]]<br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif;">- New AGU publications data policy</span>=== <br />
<span style="font-family: Arial,Helvetica,sans-serif;">AGU is now requiring authors to identify and archive their data in approved data centers. This applies to all data necessary to understand, evaluate, replicate, and build upon the reported research. Data must be made available and accessible whenever possible together with </span><span style="color: #464646; font-family: Arial,Helvetica,sans-serif;">detailed information describing data or methodology used when the data or methods are new.</span><br />
<span style="color: #464646; font-family: Arial,Helvetica,sans-serif;">AGU reserves the right to refuse publication when authors are unwilling to make the underlying data available or otherwise refuse to comply with the [http://publications.agu.org/author-resource-center/publication-policies/data-policy/ | AGU Data Policy].</span><br />
<span style="color: #464646; font-family: Arial,Helvetica,sans-serif;">Here are some [http://liemohnjgrspace.wordpress.com/2014/02/13/the-new-data-policy-is-here/ | notes on the new policy] from one of their Editor-in-Chief</span><br />
Nature has also updated their [http://www.nature.com/authors/policies/availability.html | publications data policy].<br />
<br />
===- New IMOS ( Integrated Marine Observing System) data portal=== <br />
IMOS has just released (on the 11 of February) the first version of their new data portal for feedback. The main aim of this new release was to simplify the data download in three easy steps: search, subset and download.<br />
If you're a marine observation data user this is a good chance to give feedback and ask for new features to be added to the current version.<br />
An extract from their announcement follows:<br />
"" Please note that while there are many features that could be added, we would like you to drive this development by telling us what it is you need. We encourage you to use the new portal and provide feedback and suggestions. You may even identify a bug or two, which we can resolve before the release in February.<br />
<br />
**Portal Access Details'''<br />
From '''Tuesday 11 February''' the new Portal can be accessed via the following link –<br />
[[http://imos.aodn.org.au/imos/]]<br />
Should you have any comments or require assistance please review the ‘Help’ link at the top right hand of the Portal – you may find your answer in our '''User Guide''', or send details directly using the ‘Contact Us’ link or email <span class="s1">[[mailto:info@emii.org.au | info@emii.org.au]]</span>. ""<br />
<br />
Together with the portal IMOS programmers shared the [https://github.com/aodn/imos-user-code-library | IMOS user code git repository] .<br />
The purpose of this library is to provide a ready to go code solution to incorporate data from NetCDF files in their working environment, starting with a NetCDF parser. Codes are provided in python, R and Matlab. For more information you can visit [http://%20https:''github.com/aodn/imos-user-code-library/wiki | user code library wiki].<br />
<br />
===<span style="font-family: Arial,Helvetica,sans-serif;">- AeroStat: Online Platform for Statistical Intercomparison of Aerosols announcement</span>=== <br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">[[:File:extract_from-Giovanni_news]]</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Which_data_should_I_publish&diff=385Which data should I publish2016-04-18T00:08:09Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>==Which data should I publish?== <br />
<br />
<span style="font-size: 14px;">While there's not a ready-made answer to this there are guidelines and principles to help you formulate your own answer. A few are listed here.</span><br />
* <span style="font-size: 14px;">'''''Sharing and publishing some data is better than none''''', so start from what's easier, look at what are the practices and which services are on offer at your institution to find examples and guidelines.</span><br />
* <span style="font-size: 14px;">'''''Provide the information needed to interpret, reuse and reproduce your results'''''. This is what journal publishers usually require, most of them provides guidelines and examples of which data you should share (see below for some links).</span><br />
* <span style="font-size: 14px;">'''''If the output is big publish only a subset.''''' If your methods are well described, the codes you used are easily available, than in most cases you can publish only the subset of data that underlines your publication. So, for example if you run a model but then used only some of the variables in the output, publishing only the post-processed output of the model is generally sufficient from a journal point of view. You still need though to document clearly which model version and configuration you used, which input data and wherever possible point to an online source for them. Some thoughts on what kind of data you should be publishing along these lines are shared in the [[:File:ARCCSS data perspectives-21st Jan 2014.docx | Centre position on data publishing]] .</span><br />
* <span style="font-size: 14px;">Conversely dumping the data somewhere and not providing any information on the process makes the data available but completely useless. T'''''ry to think of what you look for then you are considering using a dataset for your own research'''''. What kind of information do you consider essential for the data to be usable and which additional information would make its use easier?</span><br />
* <span style="font-size: 14px;">While you might be required to share just part of your data output, there is often lots of data (especially if you run a model simulation) which you will end up not using but that might be useful to someone else. So even if you have not time or resources to publish all of your data output, '''''providing information on its existence and on how to request access to it, it is often enough'''''. This should include details on license, possible use restrictions and details on how to create accounts with other institutions if necessary.</span><br />
* <span style="font-size: 14px;">'''''Any data you share is an asset for your institution'''''. Remember we have a data expert in the ARCCSS and there will be others in your institution, do not hesitate to ask for advice and practical help. If you share data both internally or externally you are creating a new asset for other researchers and it is in our interest to help you.</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ERAI-issues&diff=138ERAI-issues2015-09-15T22:43:52Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><span style="background-color: #ffffff;">We found very high values for temperature (~330-350 K) near the lower troposphere in the following periods:</span><br />
<span style="background-color: #ffffff;">26-31 Dec 2002 and 21-25 Dec 2003. Other variables might be similarly affected.</span><br />
<br />
<span style="background-color: #ffffff;">There are some artefacts in the sea ice cover data ....</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS_CoE_simulations&diff=38ACCESS CoE simulations2015-08-19T01:58:44Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><span style="font-family: Arial,Helvetica,sans-serif;">Dr Sophie Lewis of the School of Earth Sciences at the University of Melbourne has completed a series of model simulations for the </span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Coupled Model Intercomparison Project phase 5 (CMIP5) project.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">She used the ACCESS model (both version 1-0 and 1-3), a general circulation climate model developed as a collaboration between CSIRO and BoM.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">She run three ensembles for the historical experiment, simulating the global climate between 1850-2006, using historical forcing. One simulation uses ACCESS1-0 (ensemble r2i1p1), the other two use ACCESS1-3 (ensembles r2i1p1 and r3i1p1).</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">She also completed three ensembles (r1i1p1, r2i1p1 and r3i1p1) for the historicalNat experiment, </span><span style="font-family: Arial,Helvetica,sans-serif; line-height: 1.5;">simulating the global climate between 1850-2006, using only natural forcing.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;"><span style="line-height: 1.5;">And three </span>ensembles (r1i1p1, r2i1p1 and r3i1p1) for the historicalGHG experiment, </span><span style="font-family: Arial,Helvetica,sans-serif; line-height: 1.5;">simulating the global climate between 1850-2006, using historical well-mixed greenhouse gases forcing.</span><br />
<br />
===Publication and access=== <br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">The output of the simulations has been post-processed and quality checked and then published on the NCI Earth System Grid node ([http://esg2.nci.org.au | ESG NCI node]) . </span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Both the CMIP5 and the original raw output are hosted on raijin. The raw output is stored on massdata under project w97. The CMIP5 output is hosted on </span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">/g/data/p66/CMIP5/published/CMIP5/output1/CSIRO-BOM/ACCESS.... -- this includes the output of other ACCESS simulations performed by CAWCR</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">On /g/data/p66/pfu599/CMIP5/output/CSIRO-BOM/ACCESS... -- this includes also CMIP5 output in preparation for publication.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">More information on the post-processing and quality check process are listed in the [[ACCESS publishing procedure]] page.</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">These datasets are accessible to anyone that has a NCI account, to access the raw output please contact climate_help@nf.nci.org.au</span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">More information on the CAWCR simulations is available on the [https://wiki.csiro.au/display/ACCESS/Home | CAWCR ACCESS wiki].</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">All the simulations are currently published on the ESG. </span><br />
<span style="font-family: Arial,Helvetica,sans-serif;">Metadata for the same collections have been published on the Australian National Data Service (ANDS) metadata repository, [http://researchdata.ands.org.au/ | Research Data Australia] (RDA) </span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">This is a direct link to [https://researchdata.ands.org.au/dr-sophie-lewis | Dr Sophie Lewis party record], under which all the available collections are listed.</span><br />
<br />
<span style="font-family: Arial,Helvetica,sans-serif;">Due to a bug with the RDA website while all the records are found, no more than five will be shown, so here is a complete list:</span><br />
* [https://researchdata.ands.org.au/access1-0-climate-r2i1p1-ensemble/259215 | ACCESS1-0 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical experiment, r2i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-0-climate-r2i1p1-ensemble''472321 | ACCESS1-0 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical extension (historicalExt) experiment, r2i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r2i1p1-ensemble/316806 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical experiment, r2i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r3i1p1-ensemble/316835 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical experiment, r3i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r2i1p1-ensemble/477695 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical extension (historicalExt) experiment, r2i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r3i1p1-ensemble/477694 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical extension (historicalExt) experiment, r3i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r1i1p1-ensemble/472307 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical, natural forcing (historicalNat) experiment, r1i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r2i1p1-ensemble/472311 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical, natural forcing (historicalNat) experiment, r2i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r3i1p1-ensemble/472323 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical, natural forcing (historicalNat) experiment, r3i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r1i1p1-ensemble/316837 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical, well-mixed greenhouse gases (historicalGHG) experiment, r1i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r2i1p1-ensemble/316836 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical, well-mixed greenhouse gases (historicalGHG) experiment, r2i1p1 ensemble]<br />
* [https://researchdata.ands.org.au/access1-3-climate-r3i1p1-ensemble/472302 | ACCESS1-3 climate model output prepared for the Coupled Model Intercomparison Project Phase 5 (CMIP5) historical, well-mixed greenhouse gases (historicalGHG) experiment, r3i1p1 ensemble]</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=YOTC&diff=410YOTC2015-07-30T00:30:56Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>==<span style="color: #0000ff;">YOTC - Year of Tropical Convection</span>== <br />
<br />
The <span style="font-size: 13px; line-height: 1.5;">Year of Tropical Convection is a joint project of the WWRP-THORPEX and the WCRP that combines high resolution analysis and forecast data with satellite data and a </span><br />
[http://tropicalcyclone.jpl.nasa.gov/hurricane/index.jsp | Tropical Cyclone Satellite Information System].<br />
YOTC is a Year of coordinated observing, modeling and forecasting of organized tropical convection and its influences on predictability as a contribution to the United Nations Year of Planet Earth to complement the International Polar Year (IPY). This effort is intended to exploit the vast amounts of existing and emerging observations, the expanding computational resources and the development of new, high-resolution modeling frameworks, with the objective of advancing the characterization, diagnosis, modeling, parameterization and prediction of multi-scale convective/dynamic interactions, including the two-way interaction between tropical and extra-tropical weather/climate*.<br />
<br />
The CMS is managing a replica of the re-analysis data produced by ECMWF on the NCI RDSI disk storage which is mounted on raijin (raijin.nf.nci.org.au)<br />
We copied across the pressure levels analysis data at the highest available resolution 0.125X0.125 degrees, the original format is grib and<br />
the data is momentarily located on fast access disk in the following directories<br />
/g/data/rq7/YOTC/grib/oper_an_pl/<variable>/<year><br />
The data has been converted to netcdf to facilitate its use, the netcdf files are stored under<br />
/g/data/rq7/YOTC/netcdf/oper_an_pl/<variable>/<year>/<br />
{| <br />
| Variable || || File || || Converted to netcdf <br />
|-<br />
| Divergence || || D || || Y <br />
|-<br />
| Geopotential || || Z || || Y <br />
|-<br />
| Ozone mass mixing ratio || || O3 || || Y <br />
|-<br />
| Potential vorticity || || PV || || Y <br />
|-<br />
| Relative humidity || || R || || Y <br />
|-<br />
| Specific humidity || || Q || || Y <br />
|-<br />
| Temperature || || T || || Y <br />
|-<br />
| U component of wind || || U || || Y <br />
|-<br />
| V component of wind || || V || || Y <br />
|-<br />
| Vertical velocity || || W || || Y <br />
|-<br />
| Vorticity (relative) || || VO || || Y <br />
|-<br />
|}<br />
To access the data you need to register with ECMWF to accept their [http://apps.ecmwf.int/datasets/data/yotc_od/licence/ | terms and conditions of use] and then request access to rq7 by using your NCI account to access [https://my.nci.org.au/ | My NCI portal] or e-mail [[mailto:climate_help@nf.nci.org.au | NCI climate_help]] if you need any help.<br />
For more information on the original ECMWF collection: [http://www.ecmwf.int/en/forecasts/datasets/year-tropical-convection-yotc | YOTC page on ECMWF website]<br />
<br />
For more information about this project and to access the satellite data visit the <span style="font-size: 13px; line-height: 1.5;">[http://www.ucar.edu/yotc/data.html | YOTC project webpage].</span><br />
<br />
<span style="font-size: 13px; line-height: 1.5;">GES-DISC developed a web-based data analysis tool to support coordinated observing, modeling, and forecasting, focusing on organized tropical convection.</span><br />
<span style="font-size: 13px; line-height: 1.5;">For more information on the YOTC-Giovanni tool go to </span><br />
[[http://disc.sci.gsfc.nasa.gov/gesNews/new-goddard-giovanni-tools-for-supporting-year-of-tropical-convection-yotc]].<br />
<br />
====='''Conversion to netcdf'''===== <br />
<br />
To facilitate the use of this dataset, we converted the grib files to netcdf, we have recently upgraded our conversion routine, to produce compressed netcdf4 files.<br />
We use cdo for the conversion itself and nco to enhance the metadata. Details of the procedure are described below.<br />
[[media type="custom" key="27546378"]]<br />
<br />
<br />
<br />
<br />
* extract from <span style="font-size: 13px; line-height: 1.5;">http://www.ucar.edu/yotc/about.html</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=CMIP5_announcement&diff=82CMIP5 announcement2015-06-23T00:09:47Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div><span class="s1">Dear CMIP5 analysts: </span><br />
<span class="s1">There are two separate items addressed by this email: 1) the temporary unavailability of CMIP5 data on ESGF, and 2) a request to register publications referencing CMIP5 model output.</span><br />
<span class="s1">1) All ESG Federation data nodes will be offline for an estimated 3 to 4 weeks. Further information about this can be found at:</span><span class="s2"> <span class="s3">[http://esgf.llnl.gov/ | http:''esgf.llnl.gov]</span></span><br />
<span class="s1">2) We entreat you who are authors or co-authors of publications based on CMIP5 model output to add citation information to the CMIP5 List of Publications at <span class="s3">[[http://cmip.llnl.gov/cmip5/publications/allpublications]]</span> . Instructions on how to do this can be found at: <span class="s3">[[http://cmip-pcmdi.llnl.gov/cmip5/publications.html]]</span> . This should only take you a few minutes. Nearly 800 papers have already been registered there, and we hope you will add your contributions.</span><br />
<span class="s1">''Registering your paper(s)'' ''is vitally important because by documenting CMIP’s scientific impact, the modeling groups and those developing CMIP software infrastructure can continue to secure funding. We expect that those of you who have taken advantage of the CMIP5 archive will consider this an obligation.''</span><br />
<span class="s1">If you and your co-authors have only ''cited'' the CMIP5 project, but not analyzed any of the CMIP5 output, no action is required.</span><br />
<span class="s1">Upon registering your publications, please be sure to provide the additional critical information requested, including which ''models, experiments, and variables'' you have used in your study. The information you provide will help us justify archiving these variables in future phases of CMIP.</span><br />
<span class="s1">Since anyone can enter citation information on your behalf, please check (by searching the alphabetical listing at <span class="s3">[[http://cmip.llnl.gov/cmip5/publications/allpublications]]</span> ) whether or not any of your already registered papers include complete and correct information (e.g., please provide a DOI if it is missing).</span><br />
<span class="s1">If you have questions or concerns regarding the list of publications please email: <span class="s3">[[mailto:taylor13@llnl.gov | taylor13@llnl.gov]]</span> .</span><br />
<span class="s1">Thanks and best regards,</span><br />
<span class="s1">Karl Taylor, on behalf of all CMIP participants</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ACCESS1.3b_AMIP_Land_Use_Change&diff=33ACCESS1.3b AMIP Land Use Change2014-08-08T04:41:35Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>[[ListOfExperiments | (Back to List of Experiments)]]<br />
<br />
==Owner== <br />
Ruth Lorenz, [[mailto:r.lorenz@unsw.edu.au | r.lorenz@unsw.edu.au]]<br />
<br />
==History== <br />
Based on Greg Roff's (BoM) ACCESS 1.3 job on Solar, but with CABLE2.0 instead of 1.8. Updated sea surface temperature and sea ice input files to cover longer time period (1950-2011). Alternative vegetation fraction files used for different experiments.<br />
Set up on Raijin by Ruth Lorenz with help from Scott Wales and Jhan Srbinovsky. Updated Sea surface temperature and sea ice files provided by Dan Copsey (Met Office),<br />
<br />
==Specific Ancillary Files== <br />
* /home/561/rzl561/input_data/LUC/ for vegetation files<br />
* /short/public/rzl561/input_data/sst_amip_1870-2012_n96<br />
* /short/public/rzl561/input_data/seaice_amip_1870-2012_n96<br />
<br />
==Use Cases== <br />
Deforestation experiments over Amazonia.<br />
<br />
==Modules of the Experiment== <br />
{| <br />
| Atmosphere || UM 7.3 <br />
|-<br />
| Land Surface || CABLE 2.0 <br />
|-<br />
| Ocean || Prescribed (AMIP) <br />
|-<br />
| Configuration || HadGEM3 (beta) <br />
|-<br />
|}<br />
<br />
==Performance== <br />
At 8x16 cores, 1 year run takes about 2h40<br />
<br />
==Job submission== <br />
uaoy (accesscollab)<br />
vacd (accessdev)<br />
<br />
==Code Repositories== <br />
[[https://access-svn.nci.org.au/trac/um/browser/branches/pkg/Rel/ACCESS-1.3_CABLE-2.0_replacement]]<br />
[[https://trac.nci.org.au/trac/cable/browser/tags/CABLE-2.0]]<br />
<br />
==Publications== <br />
<br />
==Data access== <br />
<br />
A dataset is available for the deforestation experiments over Amazonia. This dataset is described in ANDS RDA and will be soon available on a proper data server. In the meantime the data can be obtained by contacting the data owner Ruth Lorenz, [[mailto:r.lorenz@unsw.edu.au | r.lorenz@unsw.edu.au]] or the CMS data manager paola.petrelli@utas.edu.au.<br />
<span style="color: #2f3034;">Access to this dataset is free for non-commercial use, the users are free to download this dataset and share it with others as long as they credit the dataset owners, but they can’t change the dataset in any way or use it commercially.</span></div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ESGF_OpenID&diff=139ESGF OpenID2014-07-28T08:26:53Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>There are two known current issues with the ESGF OpenID: one local to the NCI node and one related to external nodes OpenIDs.<br />
You might have noticed on the front page of the NCI esg node (esg2.nci.org.au) the notice "Local OpenIDs will not work, use OpenIDs generated on other peers."<br />
<br />
**Local issues with the NCI node''' <br />
The reason for the local OpenID problem has to do with the SSL certificates on the NCI hosts (esg2 and esgnode2). This issue should eventually disappear once the latest ESGF version (1.7) will be installed.<br />
Another access issue with the NCI node is a search timeout. Any ESGF node permits searches for data on all ESGF nodes. When you start a search on an ESGF node, it polls all the other nodes to update the local metadata indexes used for searching. The NCI node is no different in that regard, except that for some reason that query process takes quite a bit of time, many minutes. <br />
The end result is that if you switch to the "Search" tab on the NCI node right after you log in, you'll get a timeout message, due to the polling. What you need to do is switch to the Search tab, let it time out, wait about ten minutes (it varies), then select the Search tab again, and the search should be able to proceed.<br />
You may note that the "Search all sites" option is on by default on the search interface. You may think that unchecking that option would limit the search to the local node only, and it does, but a design fault in the web interface prevents that option from being unchecked until the global metadata poll is complete.<br />
**Issues with external OpenIDs'''<br />
In the last few weeks the OpenIDs generated on some ESGF peers have become unworkable. An OpenID should permit access on all ESGF nodes, that access granted by group memberships controlled on the original OpenID host, but some hosts are now having problems with their OpenID group information, preventing access to the users. PCMDI OpenIDs are still working fine and the PCMDI node (pcmdi9.llnl.gov) should be your preferred choice if you need an account or also if you have already an account but your OpenID is not working.<br />
You might notice this if you try logging into the NCI node using (for example) a DKRZ OpenID. You should get a screen that asks for your group membership (usually "CMIP5 Research"), which will then results in an error screen when you use that group. Showing the groups says "You are not in any groups", but trying to add the "CMIP5 Research" group says "You are already in this group", as in fact should be. This conflicting behaviour is generated by some bug in the 'group' management at the node where you signed up (in the example DKRZ).</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=ERA_Interim_current_status&diff=136ERA Interim current status2014-01-06T06:01:31Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>==='''<span style="font-size: 13px; line-height: 1.5;">Current status of ERA-Interim grib download and conversion to netcdf on dcc</span>'''=== <br />
<br />
Last updated on 06/01/2014<br />
<br />
Data available on ECMWF data server: from 01/1979 to 10/2013<br />
<br />
[[media type="custom" key="24804672"]]<br />
<br />
**Lists of variables available for each dataset'''<br />
[[media type="custom" key="24804686"]]<br />
<br />
{| <br />
|||!! '''oper_an_sfc --> Operational Analysis at Surface''' <br />
|-<br />
|~ original grib !! !! <span style'''"display: block; text-align: center;">variable name</span> !! !! netcdf !! !! <span style'''"display: block; text-align: center;">original grib</span> !! !! <span style'''"display: block; text-align: center;">variable name</span> !! !! <span style'''"display: block; text-align: center;">netcdf</span> <br />
|-<br />
| || || Temperature of snow layer [K] || ||= || || || || Sea-ice cover [(0 - 1)] || || <br />
|-<br />
| || || Soil temperature level 4 [K] || ||= || || || || Total cloud cover [(0 - 1)] || || <br />
|-<br />
| || || Skin temperature [K] || ||= x || || || || Mean sea level pressure [Pa] || || x <br />
|-<br />
| || || Logarithm of surface roughness length for heat [] || ||= || || || || Charnock [] || || <br />
|-<br />
| || || Total column ozone [kg m'''-2]''' || ||= || || || || Snow depth [m of water equivalent] || || <br />
|-<br />
| || || Skin reservoir content [m of water] || ||= || || || || Soil temperature level 1 [K] || || <br />
|-<br />
| || || High cloud cover [(0 - 1)] || ||= || || || || Total column water vapour [kg m-2] || || x <br />
|-<br />
| || || Medium cloud cover [(0 - 1)] || ||= || || || || Total column water [kg m'''-2]''' || || x <br />
|-<br />
| || || Low cloud cover [(0 - 1)] || ||= || || || || Surface pressure [Pa] || || x <br />
|-<br />
| || || Soil temperature level 3 [K] || ||= || || || || Volumetric soil water layer 4 100-255 cm [m3 m'''-3]''' || || <br />
|-<br />
| || || Albedo [(0 - 1)] || ||= || || || || Volumetric soil water layer 3 28-100 cm [m3 m'''-3]''' || || <br />
|-<br />
| || || Surface roughness [m] || || || || || || Volumetric soil water layer 2 7-28 cm [m3 m'''-3]''' || || <br />
|-<br />
| || || Soil temperature level 2 [K] || || || || || || Volumetric soil water layer 1<br>0-7 cm [m3 m'''-3]''' || || <br />
|-<br />
| || || 2 metre dewpoint temperature [K] || || x || || || || Ice surface temperature layer 4100-255 cm [K] || || <br />
|-<br />
| || || 2 metre temperature [K] || || x || || || || Ice surface temperature layer 328-100 cm [K] || || <br />
|-<br />
| || || 10 metre V wind component [m s-1] || || x || || || || Ice surface temperature layer 17-28 cm [K] || || <br />
|-<br />
| || || 10 metre U wind component [m s'''-1]''' || || x || || || || Ice surface temperature layer 10-7 cm [K] || || <br />
|-<br />
| || || Snow density [kg m-3] || || || || || || Sea surface temperature [K] || || x <br />
|-<br />
| || || Snow albedo [(0 - 1)] || || || || || || || || <br />
|-<br />
|}</div>Paolautashttp://climate-cms.wikis.unsw.edu.au/index.php?title=Backup_strategy&diff=225Backup strategy2012-10-02T00:02:55Z<p>Paolautas: Imported from Wikispaces</p>
<hr />
<div>A few weeks ago several hard drives in NCI's data centre failed, meaning the Vayu supercomputer and DCC had to be brought offline while repairs were made. Thankfully no data was lost due to the use of redundant components, however one more failed drive would have meant the loss of the /short filesystem. Model output generally goes to this filesystem, and because of its size NCI doesn't perform backups. This means that if you aren't performing backups yourself there is a possibility of losing data, so it's important to make plans to avoid this.<br />
<br />
A large amount of a researcher's time is spent in the production and manipulation of data. We make measurements, run simulations & perform analyses that make use of a vast amount of computer storage, and create programs and scripts to help us do things in a consistent and reproducible way. Modern research is heavily invested in making use of computers, which has caused an explosion in the amount of data that researchers can make use of.<br />
<br />
No technology is perfect, and sometimes disasters can happen, from a fire destroying your computer, a hard drive failing or just deleting the wrong directory. No-one wants to lose all of their data, results can be costly or even impossible to reproduce. It's important to have some idea of how you could recover if something were to happen to your files.<br />
<br />
The first thing to think about for backups is which part of your data is essential to have, especially when you have a lot of it. You may want to preserve output files from a 10,000 hour simulation, but not care about the log files once you've verified your results. Or perhaps you have performed observations of an unusual weather system which couldn't be easily reproduced, or scripts to create all the plots used in your latest paper. Also think about who else might be making use of your files, are other people using your results as input to their own simulations? Alternately, are you making use of files in other people's directories? Don't depend on other people keeping files in the same place.<br />
<br />
The next thing to consider is how big your data is. Your home directory can easily be backed up to a portable hard drive or to Dropbox, this is not really an option when you have terabytes of model output however. NCI provides a data archive called MDSS to archive large files, most institutions also have their own archives for important data created by their researchers.<br />
<br />
You should also think about how often the data changes. Model output is unlikely to change, but programs are often improved with time. If you wanted to reproduce old results using a script you wrote some time ago, then you'd need to recover the state of the file at the time you first ran it. Revision control software like subversion and git is designed for this use case, there are a variety of hosting services available on the web that help you to manage software development.<br />
<br />
Not all backup solutions provide the same protection. Creating a copy of a file in a separate directory on your computer provides a measure of protection against accidental deletion, however it could be lost if your hard drive fails. On the other end of the scale you could store copies of your data on archival tapes in different cities, so that it could still be recoverable in the case of natural disasters. Generally the latter is costly for a large data set, the amount of protection needed should be balanced against the value of your data.<br />
<br />
Knowing all of this, what are the best options to use? The first thing to do is to protect your workstation. Check what is already being backed up by your local IT support, they may have an existing backup strategy. For instance NCI backs up user home directories on Vayu. Get an external hard drive at least as big as is in your computer, then either set up a cron job to automatically rsync your internal and external hard drives on linux or enable time machine if you use a mac. This gets you a backup of your whole workstation that you can use to recover all your installed programs and local data if a hard drive fails. You can also use a service like dropbox to automatically back up your home directory, these services limit the space you can use, however they provide a remote backup and allow you to access files from anywhere.<br />
<br />
Ask the CMS team or your institution's data services about archiving services available to store large data sets. These services have their own data management plans in place to ensure the integrity of data, you may need to provide a data management plan describing the value of the data & how long it needs to be stored for.<br />
<br />
----<br />
<br />
* How essential is the data - does it have to be backed up?<br />
** Intermediate output, run logs<br />
*** Not useful once an experiment is finished<br />
** Processing scripts<br />
*** Useful to reproduce an experiment but could be redone<br />
** Model input<br />
*** Essential to reproduce an experiment<br />
** Ease of reproducing the data<br />
*** Difficult to obtain weather readings vs. reproducable model output<br />
** Published results<br />
*** Research standards require archiving<br />
** Number of people accessing the data<br />
*** Just you, your group, people from around the world<br />
<br />
* How big is the data?<br />
** Text files - Source code, scripts, configuration files<br />
*** Sizes < 1 MB<br />
*** Not suitable for archives unless bundled into a tar file<br />
** Data files - NetCDF &c<br />
*** 10MB to 100's of GB<br />
*** Archive systems like MDSS designed for this<br />
<br />
* How much history do you need?<br />
** Unchanging once produced - e.g. model output<br />
*** Tape archives<br />
** Continually changing - source code under development<br />
*** Revision control system - subversion, git<br />
<br />
* How safe is the backup system?<br />
** What happens if...<br />
*** You delete a file<br />
**** Have backups<br />
*** The local file system fails<br />
**** Back up on a different file system<br />
*** Your storage provider goes bankrupt<br />
**** Multiple backup providers<br />
*** A fire destroys a building<br />
**** Offsite backups<br />
*** A flood/earthquake damages a city<br />
**** Offsite backups with a wide separation<br />
** Have you checked that you actually can recover?<br />
<br />
* Options<br />
** /home at NCI/University workstation<br />
*** Limited space<br />
*** Backed up - Institutions will have their own strategies<br />
** MDSS/University archives<br />
*** Designed for archiving large and/or important data sets<br />
*** Will have their own backup strategies - e.g. MDSS is duplicated to two separate buildings at ANU<br />
** USB hard drive<br />
*** Simple way to have a separate backup, cheap<br />
*** Can manage history manually or with something like time machine<br />
*** Not suitable for long-term storage - disks have limited lifetimes<br />
** Cloud storage, e.g. Dropbox, Github<br />
*** Can access from anywhere<br />
*** Offsite backup<br />
*** Can set up folders to automatically be backed up<br />
*** Is the service still going to be there in 5 years?</div>Paolautas