Giter Club home page Giter Club logo

libecbufr's People

Contributors

alexandreleroux avatar tomkralidis avatar vsouvan avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

tomkralidis alexandreleroux beauregardc nourou6 ncar fmidev darmamaus

libecbufr's Issues

bitwidth limited to 31bits

it appears that libecbufr also has a 31 bits limit for anything other than strings type (CCITT IA5).
Tests done with descriptor like 24013 where bitwidth=32 shows values are not encoded or decoded properly
when values reached the limit of 31 bits.

Imported from Launchpad using lp2gh.

build failing on Debian testing

In a nutshell, the package build is failing due to a libtool problem very new systems.

make[1]: Leaving directory /home/bob/tmp/HEAD/libecbufr/debpackage/libecbufr-0.8.2'  debian/rules build make[1]: Entering directory /home/bob/tmp/HEAD/libecbufr/debpackage/libecbufr-0.8.2'
dh_testdir
dh_testroot
libtoolize --force
libtoolize: You should add the contents of the following files to aclocal.m4': libtoolize: /usr/share/aclocal/libtool.m4'
libtoolize: /usr/share/aclocal/ltoptions.m4' libtoolize: /usr/share/aclocal/ltversion.m4'
libtoolize: /usr/share/aclocal/ltsugar.m4' libtoolize: /usr/share/aclocal/lt~obsolete.m4'
libtoolize: Consider adding AC_CONFIG_MACRO_DIR([m4])' to configure.in and libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree. libtoolize: Consider adding -I m4' to ACLOCAL_AMFLAGS in Makefile.am.
libtoolize: AC_PROG_RANLIB' is rendered obsolete by LT_INIT'
rm -f configure
autoreconf --force --install
libtoolize: Consider adding AC_CONFIG_MACRO_DIR([m4])' to configure.in and libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree. libtoolize: Consider adding -I m4' to ACLOCAL_AMFLAGS in Makefile.am.
libtoolize: AC_PROG_RANLIB' is rendered obsolete by LT_INIT'
configure.in: installing ./missing' API/Sources/Makefile.am: installing ./depcomp'
configure.in:72: installing ./config.guess' configure.in:72: installing ./config.sub'
Makefile.am: installing ./INSTALL' configure.in:72: required file ./ltmain.sh' not found
autoreconf: automake failed with exit status: 1
make[1]: *** [build] Error 1
make[1]: Leaving directory `/home/bob/tmp/HEAD/libecbufr/debpackage/libecbufr-0.8.2'
dpkg-buildpackage: error: debian/rules build gave error exit status 2
make: *** [pkg] Error 2

libtoolize is supposed to install ltmain.sh, but isn't.

Imported from Launchpad using lp2gh.

decoded value of a missing non integer descriptor is incorrect

The decoded value (as shown in printout) of a descriptor which has the Scaling > 0 (non integer) does not show as missing
when the value is missing but a very high value, which seems to be over the range of maximum representable value
by the number of bits of the descriptor.

Imported from Launchpad using lp2gh.

Table B distributed with libecbufr was misrepresented as version 14

The version of Tables B and D currently distributed with libecbufr contains WMO descriptors up to and including version 13 of Tables. But it is misrepresented in the Table's changelog as version 14. This will be corrected shortly with a new upload to the 0.8.2 and 0.8.3 branches. An update will be posted here when that is done.

It is very important to note that the bit width of descriptors 0-14-028, 0-14-029 and 0-14-030 is not the same between Table B version 14 and Table B version 13. These descriptors are 16 bits in version 13, and 20 bits in version 14. Therefore, if you come across BUFR synop data containing these descriptors, even as "MISSING", proper decoding of these messages imperatively requires using the appropriate version of Table B. Otherwise the decoder will use improper offsets when reading the values of all subsequent descriptors and spout bad values.

Imported from Launchpad using lp2gh.

Incorrect error message when BUFR_TABLES environment variable is undefined

Currently, if BUFR_TABLES is undefined, the libecbufr utility bufr_encoder will stop with an error message stating that the template is not defined properly. This is because bufr_encoder cannot find the tables and therefore is unable to read the template. The template is probably just fine. So the message is erroneous and could be considered a bug. The root cause of the issue is that the BUFR_TABLES variable is undefined - this is what needs to be reported to the user in order to take proper action.

Imported from Launchpad using lp2gh.

decoded value of missing numeric with scaling not as missing but as upper limit value of descriptor

In decoding of a BUFR message, when the value of a descriptor is defined with scaling (float) and has the value of 'MISSING',
the decoder is reporting a value corresponding to the value obtained when all bits is on, which is the maximum possible
value of a descriptor. Which is normally a value out of range for this descriptor.

Imported from Launchpad using lp2gh.

Makefile generation does not work on some platforms

Attempting to compile libECBUFR on Ubuntu 9.04 (Jaunty) from the downloadable 0.8.0 tarball, I had the following problems, which prevented me from generating makefiles, let alone compiling.

Some Makefile.am files use the $(wildcard *.ext) construct. There is apparently a forward-compatibility issue with that. Also, this is only compatible with GNU make. One example error message:

API/Headers/Makefile.am:7: wildcard *.h: non-POSIX variable name
API/Headers/Makefile.am:7: (probably a GNU make extension)

This, or similar warnings, are generated in the following places:
API/Headers/
API/Headers/private/
Test/BUFR/
Test/Dump/

Imported from Launchpad using lp2gh.

error handling Section 1 with extra baggage

Found this with the G-AIRMETs at:
ftp://tgftp.nws.noaa.gov/SL.us008001/DC.avspt/DS.sgairmet/PT.bin_DF.buf/

The section 1 length in these messages is 22 octets. A normal (edition 3) section 1 is 17 bytes padded out to 18. The 17 bytes is tracked in s1.header_len, and the 18 byte length in s1.len. The s1.data_len ends up being 5 in this case.

It triggers a failure in the section 4 size calculations, which causes section 4 to have a 1 byte undersize because it uses s1.len + s1.data_len... well, s1.len = s1.header_len+1 and this never gets recalculated when the baggage is handled. 18+5 != 22...

Imported from Launchpad using lp2gh.

bufr_descriptor_set_svalue() doesn't work on "fresh" descriptors

Sample code:

BufrDescriptor* d = bufr_create_descriptor( tables, desc );
bufr_descriptor_set_svalue(d, value);
fprintf(stderr,"set string to '%s' ('%s')\n", bufr_descriptor_get_svalue(d, &l),value);

The output ends up being an empty string rather than what we're assigning.

the root of the problem is that bufr_create_descriptor() does a lookup in the table and sets d->encoding.type, but that's all it sets in the encoding section. Well, bufr_descriptor_set_svalue() comes along and falls down to:

bufr_value_set_string( cb->value, sval, cb->encoding.nbits/8 );

which, obviously, is going to fail miserably since cb->encoding.nbits==0.

Proper fix is to ensure that bufr_create_descriptor() fully defines the encoding based on the table entry.

Imported from Launchpad using lp2gh.

bufr_encoder generate bad BUFR if input text file not fully defined

When the input text datafile used for encoding is not well defined, truncated or not fully defined with all lines of data as required by the template file used. The output BUFR message will contains errors, with missing or shifted bits.
Specially when delayed replication(s) are present.

Imported from Launchpad using lp2gh.

memory leaks in bufr_seek_msg_start()

Any attempt to read an invalid BUFR message will cause the function to
return without freeing the allocated memory first.

Here is the Bug report from Dan Shea:

Vanh or Chris, I was performing some tests on my libncawos. If I pass a char

  • BUFR to be read by bufr_memread_message that does not have "BUFR" in it, I
    get a memory leak of 65 bytes. The str below is allocated 65 bytes, but may
    not be freed if a return -1 is issued under it.

static int bufr_seek_msg_start( bufr_read_callback readcb, void *cd, char
**tagstr )
{
unsigned char c;
int notfound=1;
char *str;
int i, tagsize;

tagsize = 64;
str = (char *)malloc( (tagsize+1) * sizeof(char) );
i = 0;

    if( bufr_read_octet( readcb, cd, &c ) != 1 ) return -1;

if ( c != '\004' )
append_char_to_string( &str, &tagsize, &i, c );
while ( notfound )
{
while ( (c != 'B') )
{
if( bufr_read_octet( readcb, cd, &c ) != 1 ) return -
1;

What valgrind is reporting.

==21330== 65 bytes in 1 blocks are definitely lost in loss record 1 of 1
==21330== at 0x401C38B: malloc (vg_replace_malloc.c:149)
==21330== by 0x4033375: bufr_seek_msg_start
(in /home/shead/ncawosBufrDecoder/lib/libecbufr.so.0.7.3)
==21330== by 0x403377A: bufr_callback_read_message
(in /home/shead/ncawosBufrDecoder/lib/libecbufr.so.0.7.3)
==21330== by 0x4034B18: bufr_memread_message
(in /home/shead/ncawosBufrDecoder/lib/libecbufr.so.0.7.3)
==21330== by 0x804AC9B: getMessage (ParseBufr.c:172)
==21330== by 0x804D90C: bufrDecode (BufrDecoder.c:59)
==21330== by 0x804A60F: main (testDecoder.c:73)

Imported from Launchpad using lp2gh.

Table D 302044 has a typo

The Table D included in libecbufr distribution has an error for 302044.

It is currently:

302044 004024 002004 013003

Instead, this should have been:

302044 004024 002004 013033

Imported from Launchpad using lp2gh.

wrong data type for "%o" passed to sscanf

in function the bufr_oct2str(), a variable's pointer passed to sscanf is declared as (unsigned char) but should have been (unsigned int) instead. This could potentially corrupt memory, altering output or crash the application.
This function is mostly used when encoding a new BUFR message from a text file containing octal characters in header string.

Imported from Launchpad using lp2gh.

Pathological compression crashes decoder

A user has come across a BUFR message where compression was executed incorrectly such that NBINC > NBITS. That is, the effective bitwidth for a descriptor with compression "on" is greater than the default bitwidth for the descriptor (as specified in Table B). Besides obviously defeating the purpose of compression, this crashes the decoder.

Imported from Launchpad using lp2gh.

decoder should fail on missing descriptors

Tests/BUFR/is_winide_BLDU.bufr, among other things, uses local descriptor 0-01-204 which has no entry in the current default tables. So the test fails, generating a gibberish decode. The decoder should fail as soon as it encounters a descriptor for which it doesn't know the data bit width.

Imported from Launchpad using lp2gh.

ambiguity between binary and integer for flagtable's value datafile loading

When loading a datafile, the string representing a value for a flagtable can either be in binary form or integer.
There is a problem if the integer form is used, and the integer value only have 1 and 0 in it.
Which will be considered as a binary and read as a binary which is wrong.
Example: 1101 is loaded as 13 instead.

The workaround is avoid using integer as value for FLAGTABLE in datafile. And only use values in binary form.

Imported from Launchpad using lp2gh.

dump file may be unreadable by bufr_encoder when LC_ALL is defined

When internationalisation is in use, i.e. defining LC_ALL to other language than english. The dump file will
contains printed values that are also internationalized, i.e. all fractions are formatted differently.
For example, in english 1.234 but in french, it would be 1,234, etc...
Currently, the character "," is used as a delimiter in the parser, with some modifications, it is possible to read back
printed string if LC_ALL is defined to use the same language.
However that will still break the encoder if dump file were not created in the same language as the encoder,
and this will make things really complicated.

Imported from Launchpad using lp2gh.

need Debian packaging

We've removed the Debian packaging script (make_debpkg) for various reasons. We need a way to easily build a proper .deb using as many of the standard Debian tools as possible.

We should also be looking at .rpm support.

Imported from Launchpad using lp2gh.

trailing zeroes removed from printed values

The removal of trailing zeroes from printed values are not always desired. As it is done in currently.
In older version, all trailing zeroes are kept, which differs from current behavior.
The removal of trailing zeroes in printed values should be made optional.

Imported from Launchpad using lp2gh.

Need error checking on BUFR Master Table number

Background

In a BUFR message, Octet 4 of Section 1 contains the BUFR Master Table number. This number is an overarching identifier for the content of Tables B and D, and NOT a version number. Currently, the BUFR regulations permit two legal values for this: zero and ten. (For Meteorology and Oceanography, respectively.) Libecbufr only handles the Meteorology Tables. The Oceanography Tables find little operational use, at least for the time being, and are not implemented by CMC.

Bug Description

Therefore, for libecbufr to proceed with decoding when the Master Table number is a non-zero value is not correct behaviour.

Proposed Fix

The software should check Octet 4 and proceed as follows:

Value = 0 ; proceed with decoding

Value = 10; issue message "Oceanography Tables are not supported by libecbufr" or in French "Les tables océanographiques ne sont pas prises en charge par libecbufr". Stop processing.

Any other value: Output the value; issue message "This Master Table number is not supported by WMO BUFR regulations. Check Octet 4 in Section 1." or in French "Ce numéro de table maîtresse n'est pas pris en charge par les règles d'encodage BUFR de l'OMM. Vérifiez l'octet 4 de la section 1." Stop processing.

Imported from Launchpad using lp2gh.

examples need updating for current API

Examples are somewhat older and not necessarily usable against the current API or there are now more efficient ways to do certain things (i.e. "missing" check). Some examples are listed but missing entirely.

Imported from Launchpad using lp2gh.

negative values can only be represented by floating point

In a nutshell, the "missing" value for INT32 and INT64 types is -1 (see bufr_missing_int()). This means that while someone could create a BufrValue with an INT32 type (cough bufr_set_key_qualifier_int32() cough), they'll be silently burned if they ever want assign a legitimate -1 to that value.

It would be far better if BUFR conventions were followed and "extreme" datatype-appropriate values like INT_MIN or INT_MAX were used to represent "missing" rather than a single integral "magic" number like -1 which conflicts with legitimate BUFR values.

Imported from Launchpad using lp2gh.

The decoding of some BUFR files generates warnings but it seems that LIBECBUFR doesn't return the flag BUFR_FLAG_INVALID in these cases.

Hi,

The decoding of some files (see the example provided) generates warnings but it seems that LIBECBUFR doesn't return the flag BUFR_FLAG_INVALID in these cases.

Warning example:
Warning: NBINC=3 is bigger than (2 bits)

Thanks,
Isabelle

Imported from Launchpad using lp2gh.

bogus BUFR string encoding

In a nutshell... bufr_value_set_string(bv,buf,len) assigns a string field of arbitrary length to a BufrValue. Internally, it ensures that the string is padded out to len with blanks (i.e. a C string of len-3 will be padded to len bytes).

However the encoder operates under the assumption that len is the same as the descriptor datawidth. If the user cough naively believes that the len==strlen(buf), then encoder dumps some random chunk of memory into the output.

Besides being a potential buffer overflow, this breaks things like BUFR compression (see the "differs check" in bufr_put_ccitt_compressed) and generally leads to BUFR binary messages where identical inputs/code leads to differing outputs.

It also means more management overhead for API users... BufrValue objects are frequently used standalone, with the various set/get functions implicitly converting types as needed and generally hiding the BUFR type details. Requiring the caller to "know" the datawidth means they have to keep tables around everywhere.

Simplest fix is to introduce a

bufr_put_padstring(BUFR_Message *bufr, const char *str, int len, int enclen)

function which implicitly blank pads output strings when the len < enclen.

Imported from Launchpad using lp2gh.

Table C operator 207YYY not working properly

Decoding BUFR messages containing Table C operator 207YYY is not working, resulting decoded message not good.
There is an error in implemented code for applying the change to reference value.

Imported from Launchpad using lp2gh.

broken behaviour with incorrect table datawidth

I was seeing:

Warning: bufr_getbits( 20 ), out of bounds!

and some very odd (and unstable) values in the decode stream. Furthermore, the 5th of 5 datasubsets was being dropped.

In a nutshell, if the table B datawidth for a particular descriptor during decoding is larger than during encoding, you start to run out of bits. Before you run out of bits, though, you start getting really, really weird values in your decode (i.e. a year of 1082, month 47, etc).

r163 should fix this (i.e. translate out-of-data situations as invalid decodes).

Imported from Launchpad using lp2gh.

inconsistency handling associated fields

Not sure if it's compiler dependent or architecture... we've been having long-term problems with the test case BUFR/AMDAR+2xUS-v15.bufr and it's use of associated fields. For some platforms, we get different output. For example, on my 32-bit oneiric system, the decoder debug output contains:

Code: 001111 AFD: (0 bits) 01000001 01000001 01000001 STR: [AAA] (24 bits)
Code: 001112 AFD: (0 bits) 01000010 01000010 01000010 STR: [BBB] (24 bits)
Code: 204002 Code: 031021 AFD: (0 bits) 001000 IVAL=8 (0 bits)
Code: 004001 AFD: (2 bits) 00 AFD: 0x0 (0 bits) 011111011001 IVAL=2009 (12 bi
ts)
Code: 004002 AFD: (2 bits) 00 AFD: 0x0 (0 bits) 0011 IVAL=3 (4 bits)
Code: 004003 AFD: (2 bits) 00 AFD: 0x0 (0 bits) 010101 IVAL=21 (6 bits)
Code: 004004 AFD: (2 bits) 00 AFD: 0x0 (0 bits) 01010 IVAL=10 (5 bits)

While the 64-bit oneiric system gives:

Code: 001111 AFD: (0 bits) 01000001 01000001 01000001 STR: [AAA] (24 bits)
Code: 001112 AFD: (0 bits) 01000010 01000010 01000010 STR: [BBB] (24 bits)
Code: 204002 Code: 031021 AFD: (0 bits) 001000 IVAL=8 (6 bits)
Code: 004001 AFD: (2 bits) 00 AFD: 0x0 (2 bits) 011111011001 IVAL=2009 (12 bi
ts)
Code: 004002 AFD: (2 bits) 00 AFD: 0x0 (2 bits) 0011 IVAL=3 (4 bits)
Code: 004003 AFD: (2 bits) 00 AFD: 0x0 (2 bits) 010101 IVAL=21 (6 bits)
Code: 004004 AFD: (2 bits) 00 AFD: 0x0 (2 bits) 01010 IVAL=10 (5 bits)

It starts right off with differences in interpretation of the 204002 and eventually leads to different output values.

Imported from Launchpad using lp2gh.

unexpected numerical lossiness during decoding

Encoder:

double kelvin = 273.15;
double drybulb_c = 0.0; /* Celsius /
double drybulb_k = drybulb_c + kelvin; / Kelvin */
int desc = 12101;
double scale = 2;
double reference = 0.
int BUFR_int = drybulb_k * pow(10,scale) - reference;

In the BUFR message, this is being coded as 27315. Which, I think, we can all agree is a sufficiently accurate representation of 0.0 C irrespective of any intermediate IEEE 754 magic.

In other words, the encoder looks good.

Decoder:

int BUFR_int = 27315;
double drybulb_k = (BUFR_int + reference) / pow(10,scale);
double drybulb_c = drybulb_k - kelvin;

Now, here's the thing... irrespective of the IEEE 754 encoding of the kelvin constant, we've supposedly kept all our transformations in the same 64-bit IEEE 754 form except where we convert to and from the integral BUFR encoding. And the value we're encoding, 273.15, is fully representable in the BUFR definition we're using. So, if the BUFR encoding/decoding sequence is stable, it should be perfectly safe to say:

assert( drybulb_c == 0.0 );

Sound reasonable?

Well, that's not what's happening. What's happening is that the double representation of the decoded kelvin_k is 273.149994 (which, even though 64-bit can't exactly represent 273.15, is less than half the precision it should be able to get), which gives me a drybulb_c of -0.000006.

So, point being is something in the decoding process appears to be lossy at the API level. What goes in is not what's coming out.

Now, there may be a perfectly logical reason; it could be something as trivial as the (BUFR_int + reference) / pow(10,scale) calculation being done in a 32-bit floating point space.

Imported from Launchpad using lp2gh.

data and descriptor mismatch while encoding BUFR from text data

When a BUFR template contains a delayed replication of a descriptors sequence, and the same descriptor leading that sequence immediately follow. And If there is zero replication, the encoder's text datafile parser will skip the line of data for this descriptor causing a alignment error between datafile and descriptors sequence.

Imported from Launchpad using lp2gh.

decoder broken for all BUFR message edition 4 when section 1 length is 23

The decoder failing on all valid BUFR messages edition 4 where section 1 length is exactly 23 bytes, which correspond to specification.
The decoder seems to be expecting a padding bytes which is not always there. As a result, failed to decode several valid message.
which tested as OK using the BUFR checker.

Imported from Launchpad using lp2gh.

cannot store a value which equal to its range limit

the encoder is rejecting a value that is equal to its descriptor's valid range lower or upper limit value.
For example descriptor 13011, this should take any value between [-0.1, 1638.1]
But the value -0.1 cannot be stored and a default missing value is used instead.

Debug show that value of -0.1 was rejected because it is out of range.

Imported from Launchpad using lp2gh.

Table C operator 203YYY not working

This operator is not implemented correctly. decoded values following this operator seems incorrect. And decoded messages
appears truncated with invalid values.

Imported from Launchpad using lp2gh.

bufr_decoder and bufr_encoder omit orig sub centre

For BUFR edition 4. The decoded text dump output of the bufr_decoder program does not contains the value of origine sub centre.
Also, the bufr_encoder does not read the value of origine sub centre in anyway. Thus, the value is lost if a BUFR message
is decoded and re-encoded. And the original and newly created BUFR file differs.

Imported from Launchpad using lp2gh.

error encoding of compressed datasetsubsets where only 1 has a value

In a dataset that contains several subsets, the encoding is incorrect when only 1 of the subsets has a value for a descriptor, and all the other subsets has a missing value for this same descriptor. The resulting BUFR message is encode as all subsets has that same value even those that should be set as missing instead

Imported from Launchpad using lp2gh.

can't decode message due to message length mismatch

There is a error while reading section 4 because total message length is not calculated properly, extra characters of section 1 were not taken into account. This abort decoding of any message of BUFR edition 4 where section 1 is > 22 octets.

Imported from Launchpad using lp2gh.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.