Difference between revisions of "GRIB Format Files"

 
Line 2: Line 2:
 
GRIB is a data file format for climate and weather data, commonly used by institutions such as ECMWF. You may come across GRIB format files when working with data from other Centres, such as the ERA-Interim reanalysis
 
GRIB is a data file format for climate and weather data, commonly used by institutions such as ECMWF. You may come across GRIB format files when working with data from other Centres, such as the ERA-Interim reanalysis
  
GRIB files can be worked with using the tools in the NCI `<span style="font-family:monospace">grib_api</span>` module. The [https://software.ecmwf.int/wiki/display/ECC/GRIB+tools | official documentation] gives the full list of commands available, this is just a subset.
+
GRIB files can be worked with using the tools in the NCI `<span style="font-family:monospace">grib_api</span>` module. The [https://software.ecmwf.int/wiki/display/ECC/GRIB+tools official documentation] gives the full list of commands available, this is just a subset.
  
==Inspecting GRIB files==  
+
== Inspecting GRIB files ==
  
To see the contents of a GRIB file use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_ls | grib_ls]</span>` program:
+
To see the contents of a GRIB file use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_ls grib_ls]</span>` program:
<syntaxhighlight lang=bash>
+
<syntaxhighlight lang="bash">
 
$ grib_ls /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head
 
$ grib_ls /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head
 
/g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405
 
/g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405
Line 15: Line 15:
 
1            ecmf        isobaricInhPa  1            20140401    0            an          t            grid_simple  regular_ll
 
1            ecmf        isobaricInhPa  1            20140401    0            an          t            grid_simple  regular_ll
 
</syntaxhighlight>
 
</syntaxhighlight>
Fields in a grib file are stored as 2d horizontal slices, so for a 4d field there will be slices for each vertical level and time value.
+
Fields in a grib file are stored as 2d horizontal slices, so for a 4d field there will be slices for each vertical level and time value.  
 
+
== Filtering results ==
==Filtering results==  
 
  
 
You can filter the results for most GRIB commands by using the `<span style="font-family:monospace">-w}}` flag. The filter can be any of the listed columns, so for instance to only list 'u' fields you could use `{{-w shortName=u</span>`
 
You can filter the results for most GRIB commands by using the `<span style="font-family:monospace">-w}}` flag. The filter can be any of the listed columns, so for instance to only list 'u' fields you could use `{{-w shortName=u</span>`
<syntaxhighlight lang=bash>
+
'"`UNIQ--syntaxhighlight-00000002-QINU`"'
$ grib_ls -w shortName=u /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head
 
/g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405
 
edition      centre      typeOfLevel  level        dataDate    stepRange    dataType    shortName    packingType  gridType
 
1            ecmf        isobaricInhPa  1            20140401    0            an          u            grid_simple  regular_ll
 
1            ecmf        isobaricInhPa  2            20140401    0            an          u            grid_simple  regular_ll
 
1            ecmf        isobaricInhPa  3            20140401    0            an          u            grid_simple  regular_ll
 
</syntaxhighlight>
 
  
 
There are many more keys you can filter by, you can use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_dump | grib_dump]</span>` command on a file to show the full list of keys. Some useful keys are:
 
There are many more keys you can filter by, you can use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_dump | grib_dump]</span>` command on a file to show the full list of keys. Some useful keys are:
Line 48: Line 40:
  
 
The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_copy | grib_copy]</span>` command can be used to create a new grib file, both by combining multiple input files together as well as applying filter operations:
 
The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_copy | grib_copy]</span>` command can be used to create a new grib file, both by combining multiple input files together as well as applying filter operations:
<syntaxhighlight lang=bash>
+
'"`UNIQ--syntaxhighlight-00000003-QINU`"'
grib_copy -w short_name=vo,level=500/700/850/925 /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_201404??_201404?? erai_vo_201404.grib
 
</syntaxhighlight>
 
  
 
==Converting to Netcdf==  
 
==Converting to Netcdf==  
  
 
The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_to_netcdf | grib_to_netcdf]}}` command will convert a GRIB file to NetCDF format. This command doesn't accept filters, so first make a copy of only the fields you need with `{{grib_copy</span>`.
 
The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_to_netcdf | grib_to_netcdf]}}` command will convert a GRIB file to NetCDF format. This command doesn't accept filters, so first make a copy of only the fields you need with `{{grib_copy</span>`.
<syntaxhighlight lang=bash>
+
<syntaxhighlight lang="bash">
 
$ grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib
 
$ grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib
 
$ ncdump -h erai_vo_201404.nc
 
$ ncdump -h erai_vo_201404.nc
Line 91: Line 81:
 
}
 
}
 
</syntaxhighlight>
 
</syntaxhighlight>
By default the data is compressed using a scale and offset, to disable this you can use `<span style="font-family:monospace">-D NC_FLOAT</span>`.
+
By default the data is compressed using a scale and offset, to disable this you can use `<span style="font-family:monospace">-D NC_FLOAT</span>`.  
 
+
[[Category:Data]]
[[Category: Data]]
 

Latest revision as of 00:28, 14 July 2021

GRIB is a data file format for climate and weather data, commonly used by institutions such as ECMWF. You may come across GRIB format files when working with data from other Centres, such as the ERA-Interim reanalysis

GRIB files can be worked with using the tools in the NCI `grib_api` module. The official documentation gives the full list of commands available, this is just a subset.

Inspecting GRIB files

To see the contents of a GRIB file use the `grib_ls` program:

$ grib_ls /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head
/g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405
edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType
1            ecmf         isobaricInhPa  1            20140401     0            an           pv           grid_simple  regular_ll
1            ecmf         isobaricInhPa  1            20140401     0            an           z            grid_simple  regular_ll
1            ecmf         isobaricInhPa  1            20140401     0            an           t            grid_simple  regular_ll

Fields in a grib file are stored as 2d horizontal slices, so for a 4d field there will be slices for each vertical level and time value.

Filtering results

You can filter the results for most GRIB commands by using the `-w}}` flag. The filter can be any of the listed columns, so for instance to only list 'u' fields you could use `{{-w shortName=u` ?'"`UNIQ--syntaxhighlight-00000002-QINU`"'?

There are many more keys you can filter by, you can use the `| grib_dump` command on a file to show the full list of keys. Some useful keys are:

shortName ECMWF field name, e.g. `u`
level Vertical level, e.g. `700` (Units will depend on the file)
cfName CF Standard Name, e.g. `eastward_wind`
dataDate Date, e.g. `20140402`
dataTime Time, e.g. `1800`

You can have multiple constraints in the same command, by separating values with '/}}' and keys with ',', e.g. '{{-w short_name=u/v/vo,level=500/700/850/925'.

Splitting and merging files

The `| grib_copy` command can be used to create a new grib file, both by combining multiple input files together as well as applying filter operations: ?'"`UNIQ--syntaxhighlight-00000003-QINU`"'?

Converting to Netcdf

The `| grib_to_netcdf}}` command will convert a GRIB file to NetCDF format. This command doesn't accept filters, so first make a copy of only the fields you need with `{{grib_copy`.

$ grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib
$ ncdump -h erai_vo_201404.nc
netcdf erai_vo_201404 {
dimensions:
    longitude = 480 ;
    latitude = 241 ;
    level = 4 ;
    time = 120 ;
variables:
    float longitude(longitude) ;
        longitude:units = "degrees_east" ;
        longitude:long_name = "longitude" ;
    float latitude(latitude) ;
        latitude:units = "degrees_north" ;
        latitude:long_name = "latitude" ;
    int level(level) ;
        level:units = "millibars" ;
        level:long_name = "pressure_level" ;
    int time(time) ;
        time:units = "hours since 1900-01-01 00:00:0.0" ;
        time:long_name = "time" ;
    short vo(time, level, latitude, longitude) ;
        vo:scale_factor = 2.0322676361996e-08 ;
        vo:add_offset = -9.61583905422992e-05 ;
        vo:_FillValue = -32767s ;
        vo:missing_value = -32767s ;
        vo:units = "s'''-1" ;
        vo:long_name = "Vorticity (relative)" ;
        vo:standard_name = "atmosphere_relative_vorticity" ;

// global attributes:
        :Conventions = "CF-1.0" ;
        :history = "2017-11-22 00:14:16 GMT by grib_to_netcdf-1.12.3: grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib" ;
}

By default the data is compressed using a scale and offset, to disable this you can use `-D NC_FLOAT`.