Difference between revisions of "Finding datasets"

(Starting a page covering how to find data focusing on NCI resources but also listing alternative sources)
 
m
Line 2: Line 2:
 
== '''<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">Finding data at NCI</span></span></span></span>''' ==
 
== '''<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">Finding data at NCI</span></span></span></span>''' ==
  
&nbsp;
+
<span style="font-size:medium;"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">There are a lot of climate data resources which are available at NCI. We are covering here how to locate the dataset and how to make sure that the data is suitable for your purpose.</span></span></span></span>
 
 
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">There are a lot of climate data resources which are available at NCI. We are covering here how to locate the dataset and how to make sure that the data is suitable for your purpose.</span></span></span></span>
 
  
 
&nbsp;
 
&nbsp;
  
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">Step 1: check the NCI data catalogue</span></span></span></span>
+
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">Step 1: check the [http://geonetwork.nci.org.au/geonetwork/srv/eng/main.home NCI data catalogue] </span></span></span></span>
  
 
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">NCI- geonetwork</span></span></span></span>
 
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">NCI- geonetwork</span></span></span></span>
Line 20: Line 18:
 
&nbsp;
 
&nbsp;
  
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">Step 2 : check the CMS wiki</span></span></span></span>
+
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">Step 2&nbsp;: check the CMS wiki</span></span></span></span>
  
 
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">The CMS wiki lists all the datasets we download and manage for our researchers. Some of these are also listed in geonetwork, mostly are not because for example we have downloaded only a small subset.</span></span></span></span>
 
<span style="font-size:medium"><span style="font-family:Calibri, sans-serif"><span style="caret-color:#000000"><span style="color:#000000">The CMS wiki lists all the datasets we download and manage for our researchers. Some of these are also listed in geonetwork, mostly are not because for example we have downloaded only a small subset.</span></span></span></span>
Line 48: Line 46:
 
*[[MACC|MACC]]- ECMWF  
 
*[[MACC|MACC]]- ECMWF  
 
*[[YOTC|YOTC]]- ECMWF re-analysis on raijin  
 
*[[YOTC|YOTC]]- ECMWF re-analysis on raijin  
*[[CABLE_Data|CABLE]]datasets collection (in collaboration with the CABLE users group)  
+
*[[CABLE_Data|CABLEdatasets]] collection (in collaboration with the CABLE users group)  
 
*[[OSTIA-SST|OSTIA-SST]]  
 
*[[OSTIA-SST|OSTIA-SST]]  
 
*[[NOAA_OISST|NOAA_OISST]]- Optimum Interpolation Sea Surface Temperature from NOAA  
 
*[[NOAA_OISST|NOAA_OISST]]- Optimum Interpolation Sea Surface Temperature from NOAA  
Line 79: Line 77:
 
*[[AWAP|AWAP - Australian Water Availability Project]]  
 
*[[AWAP|AWAP - Australian Water Availability Project]]  
 
*[http://openradar.io Australian Operational Weather Radar Archive]  
 
*[http://openradar.io Australian Operational Weather Radar Archive]  
*[http://nci.org.au/data-collections/data-collections/ NCI collections]
+
*[http://geonetwork.nci.org.au/geonetwork/srv/eng/main.home NCI geonetwork records]  
*[http://geonetwork.nci.org.au/geonetwork/srv/eng/main.home NCI geodata network records]  
 
 
*[http://dap.nci.org.au/thredds/catalog.html NCI Thredds catalog]  
 
*[http://dap.nci.org.au/thredds/catalog.html NCI Thredds catalog]  
  
Line 93: Line 90:
 
*[https://researchdata.ands.org.au/marine-heatwaves-detection-code/814983 MarineHeatWaves]- Marine heatwaves detection code  
 
*[https://researchdata.ands.org.au/marine-heatwaves-detection-code/814983 MarineHeatWaves]- Marine heatwaves detection code  
  
=== '''External data resources''' ===
+
== '''External data resources''' ==
  
 
*[[GIOVANNI|GIOVANNI]]- GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure  
 
*[[GIOVANNI|GIOVANNI]]- GES-DISC Interactive Online Visualization ANd aNalysis Infrastructure  

Revision as of 21:18, 29 April 2019

Finding data at NCI

There are a lot of climate data resources which are available at NCI. We are covering here how to locate the dataset and how to make sure that the data is suitable for your purpose.

 

Step 1: check the NCI data catalogue

NCI- geonetwork

This covers most of the bigger data collections hosted by NCI. The interface search is not ideal you have to type the exact dataset name so for example if I type MERRA I will get the mERRA data record but not MERRA2. Something else to keep in mind while using geonetwork is that the dataset descriptions are provided by the owners of the data, so there is a big variance in what is included.

Some of the collections are clear as CMIP6 contains only CMIP6 data but atmospheric re-analysis or even the ARCCSS own data collection are more heterogenous and if a child record hasn’t been provided for each dataset in the collection is hard to get an idea of what’ available.

NCI is working on an improved and more user-friendly interface and also is working with all the data managers to improve the quality of the records.

 

Step 2 : check the CMS wiki

The CMS wiki lists all the datasets we download and manage for our researchers. Some of these are also listed in geonetwork, mostly are not because for example we have downloaded only a small subset.

 

Step3: ask at cws_help@nci.org.au helpdesk

 

If you still can’t locate the datasets you were looking for or you find the dataset but the description wasn’t sufficient to determine that it covers your needs then feel free to e-mail us on the helpdesk.

As both geonetwork and the wiki are potentially incomplete or some of the records might be out of date, it is always a good idea to double check with us. We also might know aboput other data which is on raijin but not necessarily listed or make enquire on your behalf to our partners to help ypu locate a specific product.

 

Step4: request us to download the data

If you still couldn’t find what you were looking for we can help you downloading the data. When we receive a request to download data, we quickly check the storage and time required for the task. Unless these are “enourmous” we usually download the data for you and put in a shared environment where others can also access it. If the dataset require a lot of storage or time to download or ongoing maintenance we might need to check with the infrastructure committee before going ahead.

It is rare that we have to say no to someone and we don’t do it without a fairly strong reason because we prefer download and manage updates to the data and to have it shared in a central location where other can access it too. We also want to avoid that precious disk space that should be used for analysis by all your group gets bogged with data.

Datasets hosted on raijin and managed by the ARCCSS

 

Other datasets hosted on raijin

 

  • CMIP5- Coupled Model Intercomparison Project Phase 5 data on raijin
  • CMIP6- Coupled Model Intercomparison Project Phase 6 data on raijin

ARCCSS datasets and software published on Research Data Australia (RDA)

The Centre of Excellence for Climate System Science has started publishing its datasets on Research Data Australia (RDA), the Australian National Data Service (ANDS) metadata repository. The first datasets to be published were from the Climate Model Downscaling Data for Impacts Research (CliMDDIR), then the ACCESS CMIP5 simulations.

  • ACCESS- CMIP5 simulations
  • CliMDDIR- Climate Model Downscaling Data for Impacts Research
  • ARCCSS collection- ARCCSS datasets on the NCI Data Catalogue
  • C20C+ ACCESS- Atmospheric ACCESS1.3 historical all forcing model output for the Climate of the 20th Century Plus (C20C+) Detection and Attribution sub-project
  • MarineHeatWaves- Marine heatwaves detection code

External data resources