noritada / grib-rs Goto Github PK
View Code? Open in Web Editor NEWGRIB format parser for Rust
License: Apache License 2.0
GRIB format parser for Rust
License: Apache License 2.0
When I try to decode some submessages, the program panics with a message 'attempt to multiply with overflow'.
dv.take(10)
to dv.take(10000)
on line 23Parse GRIB file normally.
The encoded values are incrementally increasing, and the program panicks with the following error:
0
,Grid: Latitude/longitude
Number of points: 1038240
Product: Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
Parameter Category: Mass
Parameter: Pressure reduced to MSL
Generating Proceess: Forecast
Forecast Time: 0
Forecast Time Unit: Hour
1st Fixed Surface Type: Mean sea level
1st Scale Factor: 0
1st Scaled Value: 0
2nd Fixed Surface Type: code '255' is not implemented
2nd Scale Factor: 0
2nd Scaled Value: 0
Data Representation: Grid point data - complex packing and spatial differencing
Number of represented values: 1038240
Indicator { discipline: 0, total_length: 990935 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 3, 1, 2, 0, 81, 0, 0, 0, 1, 0, 0, 0, 0, 101, 0, 0, 0, 0, 0, 255, 0, 0, 0, 0, 0] }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
size hint: (1038240, Some(1038240))
101752.59
101752.59
106462.2
:
212740210.0
214157800.0
215580060.0
thread 'main' panicked at 'attempt to multiply with overflow', /Users/noritada/gitwc/private/grib-rs/src/decoders/complex.rs:207:29
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
No response
I encountered an error while trying to parse a GRIB2 file using the grib
crate in Rust. The code fails with the error NotSupported("template 40")
.
gfs.t06z.sfluxgrbf000.grib2
.NotSupported("template 40")
.use std::fs::File;
use std::io::BufReader;
use grib::Grib;
fn main() {
let fname = "gfs.t06z.sfluxgrbf000.grib2";
// Open the input file in a normal way.
let f = File::open(fname).unwrap();
let f = BufReader::new(f);
// Read with the reader.
let grib2 = grib::from_reader(f).unwrap();
// Find the target submessage.
let (_index, submessage) = grib2
.iter()
.find(|(index, _)| *index == (1, 0))
.ok_or("no such index").unwrap();
let latlons = submessage.latlons().unwrap();
}
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotSupported("template 40")', src/main.rs:XX:YY
The error suggests that the specific GRIB2 template (template 40) is not supported by the grib
crate. It would be helpful if support for this template could be added or if there is a workaround to handle this template.
The code should parse the GRIB2 file without encountering the NotSupported("template 40")
error and should successfully retrieve the latitude and longitude information.
When parse grib2 file with 36x5011x7501 shape, the data encode with JPEG 2000, the performance is very slow. with one submessage 5011x7501 spent 10s, for parse all 36 submessage is almost need 6 min on my labtop.
Is there some optimization? Change openjpeg-sys to other jpeg 2000 library?
No response
When parsing GRIB data in the 5.3/7.3 format, sometimes there is an out-of-bounds error when calling .dispatch()
. This may be related to #30, but I am unsure.
Using the same data as #29 (NOAA GFS data), I iterated through the submessages and attempted to parse and print a few values of each, along with the definition information about the messages.
// read grib file
let fname = "./test_files/test_grib_file";
let f = std::fs::File::open(fname).unwrap();
let rd = std::io::BufReader::new(f);
let gr = grib::from_reader(rd).unwrap();
// set up decoder
for lyr in gr.iter() {
let (_, fm) = lyr;
println!(
"{}\n{:?}\n{:?}\n{:?}\n{:?}",
fm.describe(),
fm.indicator(),
fm.grid_def(),
fm.prod_def(),
fm.grid_def()
);
let dc = Grib2SubmessageDecoder::from(fm).unwrap();
match dc.dispatch() { // WILL PANIC ON SOME MESSAGES
Ok(dv) => {
println!("size hint: {:?}", dv.size_hint());
dv.take(10).for_each(|v| {
println!("{:?}", v);
});
}
Err(e) => {
eprintln!("\n\n{:?}\n\n", e);
}
};
}
Parse GRIB file normally.
Here is the backtrace along with the console output about the GRIB layer definitions. The source of the error is at src/decoders/complex.rs:36:14
.
Grid: Latitude/longitude
Number of points: 1038240
Product: Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
Parameter Category: Moisture
Parameter: Relative humidity
Generating Proceess: Forecast
Forecast Time: 12
Forecast Time Unit: Hour
1st Fixed Surface Type: Isobaric surface
1st Scale Factor: 0
1st Scaled Value: 10
2nd Fixed Surface Type: code '255' is not implemented
2nd Scale Factor: 0
2nd Scaled Value: 0
Data Representation: Grid point data - complex packing and spatial differencing
Number of represented values: 1038240
Indicator { discipline: 0, total_length: 983303 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 1, 1, 2, 0, 96, 0, 0, 0, 1, 0, 0, 0, 12, 100, 0, 0, 0, 0, 10, 255, 0, 0, 0, 0, 0] }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
thread 'read_grib::read_grib_tests::test_read_grib_file' panicked at 'range end index 4 out of range for slice of length 3', /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/complex.rs:36:14
stack backtrace:
0: rust_begin_unwind
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
2: core::slice::index::slice_end_index_len_fail_rt
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:76:5
3: core::slice::index::slice_end_index_len_fail
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:69:9
4: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:408:13
5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:18:9
6: grib::decoders::complex::decode
at /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/complex.rs:36:14
7: grib::decoders::common::Grib2SubmessageDecoder::dispatch
at /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/common.rs:121:67
8: crate::read_grib::read_grib_tests::test_read_grib_file::test_impl
at ./src/read_grib.rs:300:19
9: crate::read_grib::read_grib_tests::test_read_grib_file
at ./src/read_grib.rs:272:5
10: crate::read_grib::read_grib_tests::test_read_grib_file::{{closure}}
at ./src/read_grib.rs:272:5
11: core::ops::function::FnOnce::call_once
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
12: core::ops::function::FnOnce::call_once
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
Some of the other layers were unable to be parsed and returned a DecodeError(ComplexPackingDecodeError(NotSupported))
, but I'm guessing that those errors are related to #30.
Here is a link to my Google Drive folder containing the GRIB file that I am currently testing with. link
It would be nice to add support for GRIB2 Template 5.41, Portable Network Graphics (PNG) format.
One of data using Template 5.41 is available in MRMS product page.
I have already downloaded a data file and started to work on implementation.
CCSDS recommended lossless compression (Template 5.42) is used in ECMWF real-time forecast data, which is available as open data.
Example URL:
s3://ecmwf-forecasts/20240101/00z/0p4-beta/oper/20240101000000-0h-oper-fc.grib2
WIP.
Crate version published on crates.io is well behind the latest version of the crate in Github repo. Because of that, many latest features are missing (e.g. partial support of GFS files reading).
Release a new version of the crate (should be probably 0.4.x due to upgraded Rust edition). This will also be a great occasion to update outdated dependencies.
The impact of this release will probably be minimal as the crate has (unfortunately) not too much users. But can be advantegous by adding new features to the crate, showing on crates.io site that the crate is actively maintained and in result bringing new users.
The build has been failing on all of ubuntu-latest
, macOS-latest
, and windows-latest
on GitHub Actions since run on 2021-12-17 21:00 UTC.
cargo build
Successful build is expected.
Build fails with error messages like this (on ubuntu-latest):
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" "/home/runner/work/grib-rs/grib-rs/target/debug/deps/gribber-80f0190eb821eb8f.1222uwsbdpw5s8e2.rcgu.o" (snip) "-Wl,--end-group" "/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-2a0b2a4f96acb821.rlib" "-Wl,-Bdynamic" "-lc" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/runner/work/grib-rs/grib-rs/target/debug/deps/gribber-80f0190eb821eb8f" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs"
= note: /usr/bin/ld: /home/runner/work/grib-rs/grib-rs/target/debug/deps/libopenjpeg_sys-70c5f6d04fad6c88.rlib(t1.o): in function `opj_t1_clbl_decode_processor':
/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/openjpeg-sys-1.0.4/vendor/src/lib/openjp2/t1.c:1690: undefined reference to `opj_t1_ht_decode_cblk'
collect2: error: ld returned 1 exit status
= help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
= note: use the `-l` flag to specify native libraries to link
= note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
Successful builds before 2021-12-16 21:00 UTC used openjpeg-sys 1.0.3, while builds after 2021-12-17 21:00 UTC uses 1.0.4, and this change may have
triggered the issue.
I'm not sure if this is a broken data file or a bug.
The DWD ICON "Total precipitation accumulated since model start" first output file crashes with a LengthMismatch
error, but I think it contains an all-zero data series.
Get the grib file from reprod.zip (GitHub doesn't support grib files directly, hence it's packed into a ZIP). Then run:
$ cargo run -- decode icon_global_icosahedral_single-level_2021112018_000_TOT_PREC.grib2 0
You'll need #14 to prevent a panic.
The data is supposed to be the "Total precipitation accumulated since model start". This is the first output file (at model start) from the DWD ICON model, so I THINK that it's probably an all-zero data series.
It crashes with DecodeError(SimplePackingDecodeError(LengthMismatch))
.
Data taken from https://opendata.dwd.de/weather/nwp/icon/grib/18/tot_prec/. See https://www.dwd.de/SharedDocs/downloads/DE/modelldokumentationen/nwv/icon/icon_dbbeschr_aktuell.pdf?view=nasPublication&nn=495490 for a model description.
Thanks for the motivation to replace wgrib2.
I tried to install the repo with cargo. But unfortunately a build error occurs. It seems that some data is missing.
cargo build
``´
The error also occurs with `cargo test` and `cargo run` .
## Error message
(base) [daniel@localhost:~/dev/grib-rs]$ cargo build
Compiling grib-build v0.1.0 (/home/daniel/dev/grib-rs/gen)
Compiling grib v0.2.0 (/home/daniel/dev/grib-rs)
error: failed to run custom build command for grib v0.2.0 (/home/daniel/dev/grib-rs)
Caused by:
process didn't exit successfully: /home/daniel/dev/grib-rs/target/debug/build/grib-1c47537ab65b4df2/build-script-build
(exit code: 101)
--- stderr
thread 'main' panicked at 'called Result::unwrap()
on an Err
value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', gen/src/cct11.rs:10:31
note: run with RUST_BACKTRACE=1
environment variable to display a backtrace
## Additional information
The report is related to #1 .
Thank you very much,
Daniel
I think it would be useful to add the example script presenting how to use this library to read a GRIB file in own project.
As the library code is rather complex, it is not straightforward how to use the library in own project. Moreover, even small libraries include short presentation how to use them. I believe that adding the example script would lower a level of technical knowledge necessary for using this library. Therefore the library could also be used by atmospheric scientists, who might not have as extensive technical knowledge as programmers.
The example script could be included as .rs
file in examples
folder or as a code snippet in README
.
A for
loop:
let grib2 = Grib2::<SeekableGrib2Reader<BufReader<File>>>::read_with_seekable(f)?;
for submessage in grib2.iter() {}
returns only the first message from the .grib
file, instead of all messages that this file contains, even though all messages can be read with the crate.
The folder contains files from the ERA5 dataset (converted with ecCodes library), where _p#
files are a submessages from the era5.grib
copied into separate files:
user@user-Computer:~/grib-test$ ls -l
total 26388
-rw-rw-r-- 1 user user 13498296 lis 9 17:01 era5.grib
-rw-rw-r-- 1 user user 2076676 lis 9 17:09 era5_p1.grib
-rw-rw-r-- 1 user user 2076676 lis 9 17:09 era5_p2.grib
-rw-rw-r-- 1 user user 2076676 lis 9 17:09 era5_p3.grib
-rw-rw-r-- 1 user user 2076676 lis 9 17:09 era5_p4.grib
-rw-rw-r-- 1 user user 3114916 lis 9 17:10 era5_p5.grib
-rw-rw-r-- 1 user user 2076676 lis 9 17:10 era5_p6.grib
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5.grib
era5.grib
edition shortName gridType packingType level typeOfLevel
2 10u regular_ll grid_simple 10 heightAboveGround
2 10v regular_ll grid_simple 10 heightAboveGround
2 2d regular_ll grid_simple 2 heightAboveGround
2 2t regular_ll grid_simple 2 heightAboveGround
2 z regular_ll grid_simple 0 surface
2 sp regular_ll grid_simple 0 surface
6 of 6 messages in era5.grib
6 of 6 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p1.grib
era5_p1.grib
edition shortName gridType packingType level typeOfLevel
2 10u regular_ll grid_simple 10 heightAboveGround
1 of 1 messages in era5_p1.grib
1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p2.grib
era5_p2.grib
edition shortName gridType packingType level typeOfLevel
2 10v regular_ll grid_simple 10 heightAboveGround
1 of 1 messages in era5_p2.grib
1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p3.grib
era5_p3.grib
edition shortName gridType packingType level typeOfLevel
2 2d regular_ll grid_simple 2 heightAboveGround
1 of 1 messages in era5_p3.grib
1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p4.grib
era5_p4.grib
edition shortName gridType packingType level typeOfLevel
2 2t regular_ll grid_simple 2 heightAboveGround
1 of 1 messages in era5_p4.grib
1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p5.grib
era5_p5.grib
edition shortName gridType packingType level typeOfLevel
2 z regular_ll grid_simple 0 surface
1 of 1 messages in era5_p5.grib
1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p6.grib
era5_p6.grib
edition shortName gridType packingType level typeOfLevel
2 sp regular_ll grid_simple 0 surface
1 of 1 messages in era5_p6.grib
1 of 1 total messages in 1 files
Running this code (modified list_surfaces.rs
example) in that folder produces below output:
Code:
use grib::codetables::{CodeTable4_2, Lookup};
use grib::context::Grib2;
use grib::reader::SeekableGrib2Reader;
use std::error::Error;
use std::fs::File;
use std::io::BufReader;
use std::path::Path;
fn main() -> Result<(), Box<dyn Error>> {
println!("One file:");
let path = Path::new("era5.grib");
list_surfaces(path)?;
println!("\nSplit files:");
list_surfaces(Path::new("era5_p1.grib"))?;
list_surfaces(Path::new("era5_p2.grib"))?;
list_surfaces(Path::new("era5_p3.grib"))?;
list_surfaces(Path::new("era5_p4.grib"))?;
list_surfaces(Path::new("era5_p5.grib"))?;
list_surfaces(Path::new("era5_p6.grib"))?;
Ok(())
}
fn list_surfaces(path: &Path) -> Result<(), Box<dyn Error>> {
// Open the input file in a normal way.
let f = File::open(&path)?;
let f = BufReader::new(f);
let grib2 = Grib2::<SeekableGrib2Reader<BufReader<File>>>::read_with_seekable(f)?;
for submessage in grib2.iter() {
let discipline = submessage.indicator().discipline;
let category = submessage.prod_def().parameter_category().unwrap();
let parameter = submessage.prod_def().parameter_number().unwrap();
let parameter = CodeTable4_2::new(discipline, category).lookup(usize::from(parameter));
let forecast_time = submessage.prod_def().forecast_time().unwrap();
let (first, _second) = submessage.prod_def().fixed_surfaces().unwrap();
let elevation_level = first.value();
println!(
"{:<31} {:>14} {:>17}",
parameter.to_string(),
forecast_time.to_string(),
elevation_level
);
}
Ok(())
}
Output:
One file:
u-component of wind 0 [h] 10
Split files:
u-component of wind 0 [h] 10
v-component of wind 0 [h] 10
Dewpoint temperature 0 [h] 2
Temperature 0 [h] 2
Geopotential 0 [h] NaN
Pressure 0 [h] NaN
Clearly, the iterator does not return all messages in the file.
The SubMessageIterator
should iterate over all messages present in the .grib
file instead of returning only the first one.
SubMessageIterator
.next()
method returns only first message from the file and then returns None
.
No response
Iterator::size_hint()
s implemented in this library return the same length after consuming items.
As per description in the API reference, the implementation of size_hint()
should return the bounds on the remaining length of the iterator.
size_hint()
next()
When consuming items, the returned length of size_hint()
decreases.
Even after consuming items, the returned length of size_hint()
does not change.
No response
Copying and pasting list_surfaces.rs
example into main.rs
of a new binary crate and adding grib = "0.3.0"
as a dependency into Cargo.toml
results in build errors.
cargo new
list_surfaces.rs
from examples into main.rs
grib = "0.3.0"
as a dependency into Cargo.toml
No build errors. Example should just successfully compile and run.
Build errors:
error[E0277]: the trait bound `GribError: std::error::Error` is not satisfied
--> src/main.rs:31:85
|
31 | let grib2 = Grib2::<SeekableGrib2Reader<BufReader<File>>>::read_with_seekable(f)?;
| ^ the trait `std::error::Error` is not implemented for `GribError`
|
= note: required because of the requirements on the impl of `From<GribError>` for `Box<dyn std::error::Error>`
= note: required because of the requirements on the impl of `FromResidual<Result<Infallible, GribError>>` for `Result<(), Box<dyn std::error::Error>>`
note: required by `from_residual`
--> /home/quba/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/try_trait.rs:339:5
|
339 | fn from_residual(residual: R) -> Self;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0599]: no method named `iter` found for struct `Grib2` in the current scope
--> src/main.rs:34:29
|
34 | for submessage in grib2.iter() {
| ^^^^ method not found in `Grib2<SeekableGrib2Reader<BufReader<File>>>`
Some errors have detailed explanations: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
The examples build correctly when the latest crate version from the repo is linked (`git clone --recurse-submodules -j8 https://github.com/noritada/grib-rs.git and link the crate locally). So the example is working just okay, and the issue is with the outdated version of the crate published on crates.io
Thus, the solution is simply updating the crate on crates.io (see issue #9).
An assertion fails when calling the .dispatch()
function on a Grib2SubmessageDecoder
that is attempting to read a GRIB2 message of the 5.3/7.3 format. It fails with the error
...
panicked at 'assertion failed: `(left == right)`
left: `[-45, 45]`,
right: [6942, 6942]`', .../src/decoders/complex.rs:94:5
...
// open file
let f = std::fs::File::open("test_grib_file").unwrap();
let rd = std::io::BufReader::new(f);
let gr = grib::from_reader(rd).unwrap();
// set up decoder
let (_, first_submessage) = gr.iter().next().unwrap();
let dc = Grib2SubmessageDecoder::from(first_submessage).unwrap();
let dv = dc.dispatch().unwrap(); // SHOULD PANIC HERE
Read the grid values from the file.
Assertion panic.
No response
When parsing data at a constant Isobaric level that is encoded with Complex Packing, there are some messages that have a "Missing" 2nd Scale Factor
that cannot be parsed resulting in incorrect values being produced.
Attempt to parse a GRIB file with a supposedly missing 2nd Scale Factor
. I have posted an example GRIB file that displays this here: https://drive.google.com/drive/folders/1Zxs21uamCSF9LVUWHYO-dE-A9rzb91S_?usp=sharing. The error_grib_file
contains the troublesome data and the correct_test_file
contains GRIB data that can be properly parsed.
Note that I am only parsing messages that are at Isobaric layers (Code Table 4-5 100), not those that are altitude layers. The .idx
index file shows which messages are of which type. The specific message that I am testing on is number 295 in the index file.
The following code explains how I was comparing the data from the test file to a different file that should have fairly similar data. Note that this was copy-pasted then adapted, so it will likely require some changes to function properly.
#[test]
fn test_grib_format() {
// read through every message in a grib file and parse the first few values from each
// message
let p = "error_grib_file";
let f = std::fs::File::open(&p).unwrap();
let rd = std::io::BufReader::new(f);
let f1 = grib::from_reader(rd).unwrap();
let p = "correct_test_file";
let f = std::fs::File::open(&p).unwrap();
let rd = std::io::BufReader::new(f);
let f2 = grib::from_reader(rd).unwrap();
// set up decoder
let f1 = f1
.iter()
.map(|lyr| {
let ((i, j), fm) = lyr;
let strn = format!(
"\n\nIncorrect\n\n{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
i,
j,
fm.describe(),
fm.indicator(),
fm.grid_def(),
fm.prod_def(),
fm.prod_def().forecast_time().unwrap().to_string(),
fm.repr_def(),
);
let dc = Grib2SubmessageDecoder::from(fm).unwrap();
let dv = dc.dispatch().unwrap();
let val = dv.take(10).collect::<Vec<_>>();
(strn, val)
})
.collect::<Vec<_>>();
// set up decoder
let f2 = f2
.iter()
.map(|lyr| {
let ((i, j), fm) = lyr;
let strn = format!(
"\n\nCorrect\n\n{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
i,
j,
fm.describe(),
fm.indicator(),
fm.grid_def(),
fm.prod_def(),
fm.prod_def().forecast_time().unwrap().to_string(),
fm.repr_def(),
);
let dc = Grib2SubmessageDecoder::from(fm).unwrap();
let dv = dc.dispatch().unwrap();
let val = dv.take(10).collect::<Vec<_>>();
(strn, val)
})
.collect::<Vec<_>>();
/*
Code to pair up messages together for printing purposes
*/
f2.into_iter().zip(f1.into_iter()).for_each(|(f, l)| {
println!("{}", f.0);
println!("{:?}", f.1);
println!("{}", l.0);
println!("{:?}", l.1);
});
// ensure the test fails
panic!()
}
Parse the values properly.
Produce values that are wildly inaccurate and nowhere near the values produced by the file that is parsed correctly. Here is example output from the aforementioned message that I tested with:
Correct
Grid: Latitude/longitude
Number of points: 1038240
Product: Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
Parameter Category: Momentum
Parameter: u-component of wind
Generating Proceess: Forecast
Forecast Time: 0
Forecast Time Unit: Hour
1st Fixed Surface Type: Isobaric surface
1st Scale Factor: 0
1st Scaled Value: 20000
2nd Fixed Surface Type: code '255' is not implemented
2nd Scale Factor: 0
2nd Scaled Value: 0
Data Representation: Grid point data - complex packing and spatial differencing
Number of represented values: 1038240
Indicator { discipline: 0, total_length: 581327 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 2, 2, 2, 0, 81, 0, 0, 0, 1, 0, 0, 0, 0, 100, 0, 0, 0, 78, 32, 255, 0, 0, 0, 0, 0] }
"0 [h]"
ReprDefinition { payload: [0, 15, 215, 160, 0, 3, 195, 167, 184, 23, 0, 0, 0, 1, 9, 0, 1, 0, 98, 88, 209, 154, 255, 255, 255, 255, 0, 0, 105, 56, 0, 4, 0, 0, 0, 1, 1, 0, 0, 0, 71, 7, 2, 2] }
[-24.74382, -24.64382, -24.64382, -24.543821, -24.44382, -24.44382, -24.34382, -24.24382, -24.24382, -24.14382]
Incorrect
Grid: Latitude/longitude
Number of points: 1038240
Product: Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
Parameter Category: Temperature
Parameter: Temperature
Generating Proceess: Forecast
Forecast Time: 0
Forecast Time Unit: Hour
1st Fixed Surface Type: Isobaric surface
1st Scale Factor: 0
1st Scaled Value: 30
2nd Fixed Surface Type: code '255' is not implemented
2nd Scale Factor: Missing
2nd Scaled Value: 255
Data Representation: Grid point data - complex packing and spatial differencing
Number of represented values: 1038240
Indicator { discipline: 0, total_length: 675339 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 133, 93, 74, 128, 0, 0, 0, 0, 48, 5, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 64] }
ProdDefinition { payload: [0, 0, 0, 0, 0, 0, 2, 255, 16, 0, 0, 0, 1, 0, 0, 0, 0, 100, 0, 0, 0, 0, 30, 255, 255, 0, 0, 0, 255] }
"0 [h]"
ReprDefinition { payload: [0, 15, 215, 160, 0, 3, 70, 174, 52, 0, 0, 0, 0, 2, 8, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 89, 0, 4, 0, 0, 0, 1, 1, 0, 0, 0, 64, 6, 2, 2] }
[252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28]
Additionally, the Parameter Category
and Parameter
are seemingly incorrectly parsed as they should both be Momentum
, u-component of wind
and the incorrect file produces Temperature
, Temperature
.
When iterating through submessages, the size indicated by submessage.indicator()
is constant whereas iterating through the sections shows different sizes for each submessage.
fn test_read_grib_file() {
let f = std::fs::File::open(GRIB_TEST_FILE).unwrap();
let rd = std::io::BufReader::new(f);
let gr = grib::from_reader(rd).unwrap();
// NOTE the submessage sizes indicated by the Section0 sections
for s in gr.sections() {
println!("{:?}", s);
}
// set up decoder
for lyr in gr.iter() {
let ((i, j), fm) = lyr;
println!(
"{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
i,
j,
fm.describe(),
// NOTE the submessage sizes indicated by indicator()
fm.indicator(),
fm.grid_def(),
fm.prod_def(),
fm.grid_def(),
fm.repr_def()
);
}
}
submessage.indicator()
shows the proper size of the message.
A constant size is shown for all submessages.
I used the GRIB file found here.
Lat/lon computation of grid points fails for testdata/CMC_glb_TMP_ISBL_1_latlon.24x.24_2021051800_P000.grib2
.
It is because longitude range [180000000, 179760000] has a final value smaller than the initial value, although it is expected to increase from the scanning mode 0b01000000
.
Test code using a grid definition extracted from that data is like this:
#[test]
fn cmc_gdps_lat_lon_grid() {
let grid = LatLonGridDefinition::new(
1500,
751,
-90000000,
180000000,
90000000,
179760000,
ScanningMode(0b01000000),
);
assert!(grid.latlons().is_ok())
}
Test passes.
Test fails.
No response
When attempting to read the next message based on starting reading a GRIB file from an offset, the code panics instead of parsing the values even though it can correctly read the section information.
Here is the test I ran with the test file found here. Attempting to use other offsets that are found in the GRIB index file (second value, delimited by :
) will produce similar results, sometimes on the first message instead of the second. Starting from an offset of 0 allows the reader to read the file normally with no errors encountered.
#[test]
fn test_manual_offset_read_grib() {
// offset to the second submessage
let offset = 983_303;
let mut f = File::open(GRIB_TEST_FILE).unwrap();
f.seek(SeekFrom::Start(offset)).unwrap(); // TOGGLE COMMENTING THIS LINE TO GET ERROR
let buf = BufReader::new(f);
let gr = grib::from_reader(buf).unwrap();
for (i, (_, lyr)) in gr.submessages().enumerate() {
println!("{}", lyr.describe());
let d = Grib2SubmessageDecoder::from(lyr).unwrap();
let v = d.dispatch().unwrap(); // SHOULD RETURN AN ERROR ON THE SECOND SUBMESSAGE
for i in v.take(5) {
println!("{:?}", i);
}
println!("index: {}", i);
}
}
The message offset-to and each following message are properly parsed.
When testing on the test file that I mentioned above, the decoder can properly handle all the submessages in the GRIB file. However, when attempting to start from an offset in the file greater than 0, the decoder returns a ComplexDecodingError
for the second message.
No response
The GDAS data referred to in #29 uses Template 5.3/7.3, but it sometimes cannot be decoded with current decoder:
% ./target/debug/gribber decode nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20230111/12/atmos/gdas.t12z.pgrb2.0p25.f000 0.0
error: ComplexPackingDecodeError(
NotSupported,
)
Octet 49 (numberOfOctetsExtraDescriptors
in ecCodes) in some submessages of this data is 3, although this crate currently supports only 2. It is the reason of this error. That octet in other submessages is 2.
There is a segmentation fault when reading JPEG2000 streams due to unsafe code. This is due to the image
struct being dropped at the end of the function, so the pointer given to from_raw_parts
is no longer valid.
grib-rs/src/decoders/jpeg2000/mod.rs
Lines 113 to 119 in 1b3d559
I found this when trying to read data from the ECMWF provided files (namely the U/V components of wind)
Download some of the U/V component files from ECMWF
Run gribber decode <file.grib2.bin> 0
I get a nice array of values
Segfault on Linux
A trivial way would be to create a vec so that the values are owned:
let vec = unsafe {
std::slice::from_raw_parts(comp_gray.data, (width * height) as usize)
.iter()
.map(|x| *x as i32)
.collect::<Vec<_>>()
};
Ok(vec.into_iter())
Or the SimplePackingDecodeIterator
could be inlined into the function, ensuring the slice memory is alive until the return
Currently, the crate (particularly list_surfaces.rs
example) shows only MSLP surface in NOAA GFS file, while the file actually contains much more data (geopotential, temperature, wind etc. on both isobaric and height levels).
I believe GFS data is probably one of the most commonly used in the world. Thus, it may make this crate useful for more users. Also, it contains many fields that are present also in data from other NWP models. So support for GFS data will bring a support for multiple other datasets.
Data is available at this URL:
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.<date>/<run>/atmos/gfs.t<run>z.pgrb2.0p25.f<forecast_time>
There is also additional data in pgrb2b
files, but they contain mostly data at additional levels, so supporting data from pgrb2
files will bring most of the full support of GFS data.
I am more than happy to help with achieveing the goal set by this issue, but I am not not yet familiar enough with this crate to know where to start. I wanted to create this issue to keep track of functionality I need.
I believe, it will be easier to resolve this issue by splitting it into several more specific issues (directly relating to the parts of code that need work). If you would like to create such issues, then I also will be interested in solving them.
If the length is zero inside the next() method in FixedValueIterator we get a subtract with overflow.
This happened to me when processing grib data from the GFS
Not crashing on valid grib data.
Crashing on valid grib data.
The bug can be fixed by changing the next method in decoder/stream.rs to not subtract the length unless there is some value.
Going from this:
impl<T> Iterator for FixedValueIterator<T>
where
T: Copy,
{
type Item = T;
fn next(&mut self) -> Option<Self::Item> {
let val = if self.length > 0 {
Some(self.val)
} else {
None
};
self.length -= 1; // <--- Crash here
val
}
// -- snip -- //
}
To this:
impl<T> Iterator for FixedValueIterator<T>
where
T: Copy,
{
type Item = T;
fn next(&mut self) -> Option<Self::Item> {
let val = if self.length > 0 {
self.length -= 1;
Some(self.val)
} else {
None
};
val
}
// -- snip -- //
}
I'll be sending in a PR for this shortly.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.