Difference between revisions of "GRIB Format Files"
P.petrelli (talk | contribs) |
|||
(One intermediate revision by one other user not shown) | |||
Line 2: | Line 2: | ||
GRIB is a data file format for climate and weather data, commonly used by institutions such as ECMWF. You may come across GRIB format files when working with data from other Centres, such as the ERA-Interim reanalysis | GRIB is a data file format for climate and weather data, commonly used by institutions such as ECMWF. You may come across GRIB format files when working with data from other Centres, such as the ERA-Interim reanalysis | ||
− | GRIB files can be worked with using the tools in the NCI `<span style="font-family:monospace">grib_api</span>` module. The [https://software.ecmwf.int/wiki/display/ECC/GRIB+tools | + | GRIB files can be worked with using the tools in the NCI `<span style="font-family:monospace">grib_api</span>` module. The [https://software.ecmwf.int/wiki/display/ECC/GRIB+tools official documentation] gives the full list of commands available, this is just a subset. |
− | ==Inspecting GRIB files== | + | == Inspecting GRIB files == |
− | To see the contents of a GRIB file use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_ls | + | To see the contents of a GRIB file use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_ls grib_ls]</span>` program: |
− | <syntaxhighlight> | + | <syntaxhighlight lang="bash"> |
$ grib_ls /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head | $ grib_ls /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head | ||
/g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | ||
Line 15: | Line 15: | ||
1 ecmf isobaricInhPa 1 20140401 0 an t grid_simple regular_ll | 1 ecmf isobaricInhPa 1 20140401 0 an t grid_simple regular_ll | ||
</syntaxhighlight> | </syntaxhighlight> | ||
− | Fields in a grib file are stored as 2d horizontal slices, so for a 4d field there will be slices for each vertical level and time value. | + | Fields in a grib file are stored as 2d horizontal slices, so for a 4d field there will be slices for each vertical level and time value. |
− | + | == Filtering results == | |
− | ==Filtering results== | ||
You can filter the results for most GRIB commands by using the `<span style="font-family:monospace">-w}}` flag. The filter can be any of the listed columns, so for instance to only list 'u' fields you could use `{{-w shortName=u</span>` | You can filter the results for most GRIB commands by using the `<span style="font-family:monospace">-w}}` flag. The filter can be any of the listed columns, so for instance to only list 'u' fields you could use `{{-w shortName=u</span>` | ||
− | + | '"`UNIQ--syntaxhighlight-00000002-QINU`"' | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
There are many more keys you can filter by, you can use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_dump | grib_dump]</span>` command on a file to show the full list of keys. Some useful keys are: | There are many more keys you can filter by, you can use the `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_dump | grib_dump]</span>` command on a file to show the full list of keys. Some useful keys are: | ||
Line 48: | Line 40: | ||
The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_copy | grib_copy]</span>` command can be used to create a new grib file, both by combining multiple input files together as well as applying filter operations: | The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_copy | grib_copy]</span>` command can be used to create a new grib file, both by combining multiple input files together as well as applying filter operations: | ||
− | + | '"`UNIQ--syntaxhighlight-00000003-QINU`"' | |
− | |||
− | |||
==Converting to Netcdf== | ==Converting to Netcdf== | ||
The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_to_netcdf | grib_to_netcdf]}}` command will convert a GRIB file to NetCDF format. This command doesn't accept filters, so first make a copy of only the fields you need with `{{grib_copy</span>`. | The `<span style="font-family:monospace">[https://software.ecmwf.int/wiki/display/ECC/grib_to_netcdf | grib_to_netcdf]}}` command will convert a GRIB file to NetCDF format. This command doesn't accept filters, so first make a copy of only the fields you need with `{{grib_copy</span>`. | ||
− | <syntaxhighlight> | + | <syntaxhighlight lang="bash"> |
$ grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib | $ grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib | ||
$ ncdump -h erai_vo_201404.nc | $ ncdump -h erai_vo_201404.nc | ||
Line 91: | Line 81: | ||
} | } | ||
</syntaxhighlight> | </syntaxhighlight> | ||
− | By default the data is compressed using a scale and offset, to disable this you can use `<span style="font-family:monospace">-D NC_FLOAT</span>`. | + | By default the data is compressed using a scale and offset, to disable this you can use `<span style="font-family:monospace">-D NC_FLOAT</span>`. |
− | + | [[Category:Data]] | |
− | [[Category: Data]] |
Latest revision as of 01:28, 14 July 2021
GRIB is a data file format for climate and weather data, commonly used by institutions such as ECMWF. You may come across GRIB format files when working with data from other Centres, such as the ERA-Interim reanalysis
GRIB files can be worked with using the tools in the NCI `grib_api` module. The official documentation gives the full list of commands available, this is just a subset.
Contents
Inspecting GRIB files
To see the contents of a GRIB file use the `grib_ls` program:
$ grib_ls /g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405 | head
/g/data1/ub4/erai/grib/oper_an_pl/fullres/2014/ei_oper_an_pl_075x075_90N0E90S35925E_20140401_20140405
edition centre typeOfLevel level dataDate stepRange dataType shortName packingType gridType
1 ecmf isobaricInhPa 1 20140401 0 an pv grid_simple regular_ll
1 ecmf isobaricInhPa 1 20140401 0 an z grid_simple regular_ll
1 ecmf isobaricInhPa 1 20140401 0 an t grid_simple regular_ll
Fields in a grib file are stored as 2d horizontal slices, so for a 4d field there will be slices for each vertical level and time value.
Filtering results
You can filter the results for most GRIB commands by using the `-w}}` flag. The filter can be any of the listed columns, so for instance to only list 'u' fields you could use `{{-w shortName=u` ?'"`UNIQ--syntaxhighlight-00000002-QINU`"'?
There are many more keys you can filter by, you can use the `| grib_dump` command on a file to show the full list of keys. Some useful keys are:
shortName | ECMWF field name, e.g. `u` |
level | Vertical level, e.g. `700` (Units will depend on the file) |
cfName | CF Standard Name, e.g. `eastward_wind` |
dataDate | Date, e.g. `20140402` |
dataTime | Time, e.g. `1800` |
You can have multiple constraints in the same command, by separating values with '/}}' and keys with ',', e.g. '{{-w short_name=u/v/vo,level=500/700/850/925'.
Splitting and merging files
The `| grib_copy` command can be used to create a new grib file, both by combining multiple input files together as well as applying filter operations: ?'"`UNIQ--syntaxhighlight-00000003-QINU`"'?
Converting to Netcdf
The `| grib_to_netcdf}}` command will convert a GRIB file to NetCDF format. This command doesn't accept filters, so first make a copy of only the fields you need with `{{grib_copy`.
$ grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib
$ ncdump -h erai_vo_201404.nc
netcdf erai_vo_201404 {
dimensions:
longitude = 480 ;
latitude = 241 ;
level = 4 ;
time = 120 ;
variables:
float longitude(longitude) ;
longitude:units = "degrees_east" ;
longitude:long_name = "longitude" ;
float latitude(latitude) ;
latitude:units = "degrees_north" ;
latitude:long_name = "latitude" ;
int level(level) ;
level:units = "millibars" ;
level:long_name = "pressure_level" ;
int time(time) ;
time:units = "hours since 1900-01-01 00:00:0.0" ;
time:long_name = "time" ;
short vo(time, level, latitude, longitude) ;
vo:scale_factor = 2.0322676361996e-08 ;
vo:add_offset = -9.61583905422992e-05 ;
vo:_FillValue = -32767s ;
vo:missing_value = -32767s ;
vo:units = "s'''-1" ;
vo:long_name = "Vorticity (relative)" ;
vo:standard_name = "atmosphere_relative_vorticity" ;
// global attributes:
:Conventions = "CF-1.0" ;
:history = "2017-11-22 00:14:16 GMT by grib_to_netcdf-1.12.3: grib_to_netcdf -o erai_vo_201404.nc erai_vo_201404.grib" ;
}
By default the data is compressed using a scale and offset, to disable this you can use `-D NC_FLOAT`.