liteserver / binn Goto Github PK

ASAN:DEADLYSIGNAL
=================================================================
==19372==ERROR: AddressSanitizer: SEGV on unknown address 0x7ffdef02fd7f (pc 0x5567a79c235b bp 0x7ffd966db680 sp 0x7ffd966db660 T0)
    #0 0x5567a79c235a in AdvanceDataPos /home/mannol/Documents/Programming/B/src/binn.c:456
    #1 0x5567a79c5443 in binn_is_valid /home/mannol/Documents/Programming/B/src/binn.c:1237
    #2 0x5567a79c1f03 in binn_load /home/mannol/Documents/Programming/B/src/binn.c:340
    #3 0x5567a79ade2c in message_from_binn_raw /home/mannol/Documents/Programming/B/src/message.c:800
    #4 0x5567a7936b53 in test_message_fuzz /home/mannol/Documents/Programming/B/src/tests.c:877
    #5 0x5567a79275f5 in run_test /home/mannol/Documents/Programming/B/src/tests.c:50
    #6 0x5567a79bd57d in list_for_each /home/mannol/Documents/Programming/B/src/list.c:62
    #7 0x5567a792851f in run_tests /home/mannol/Documents/Programming/B/src/tests.c:3202
    #8 0x5567a794a1b8 in main /home/mannol/Documents/Programming/B/src/tests.c:3289
    #9 0x7f909e67d3f0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x203f0)
    #10 0x5567a7927389 in _start (/home/mannol/Documents/Programming/B/build/D/tests+0x2c389)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/mannol/Documents/Programming/B/src/binn.c:456 in AdvanceDataPos
==19372==ABORTING

EDIT:
Another one in a similar place:

ASAN:DEADLYSIGNAL
=================================================================
==19481==ERROR: AddressSanitizer: SEGV on unknown address 0x7ffed27d8404 (pc 0x55c48d1c240b bp 0x7ffea42b34a0 sp 0x7ffea42b3320 T0)
    #0 0x55c48d1c240a in binn_is_valid /home/mannol/Documents/Programming/B/src/binn.c:1223
    #1 0x55c48d1bef03 in binn_load /home/mannol/Documents/Programming/B/src/binn.c:340
    #2 0x55c48d1aae51 in message_from_binn_raw /home/mannol/Documents/Programming/B/src/message.c:800
    #3 0x55c48d133b53 in test_message_fuzz /home/mannol/Documents/Programming/B/src/tests.c:877
    #4 0x55c48d1245f5 in run_test /home/mannol/Documents/Programming/B/src/tests.c:50
    #5 0x55c48d1ba57d in list_for_each /home/mannol/Documents/Programming/B/src/list.c:62
    #6 0x55c48d12551f in run_tests /home/mannol/Documents/Programming/B/src/tests.c:3202
    #7 0x55c48d1471b8 in main /home/mannol/Documents/Programming/B/src/tests.c:3289
    #8 0x7fba7fa233f0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x203f0)
    #9 0x55c48d124389 in _start (/home/mannol/Documents/Programming/B/build/D/tests+0x2c389)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/mannol/Documents/Programming/B/src/binn.c:1223 in binn_is_valid
==19481==ABORTING

Steps to reproduce (FUZZ):

while (1)
{
    uint8_t data[15000];
    // fill data[] with random bytes
    binn obj[1];
    binn_load((void*)data, obj); // << wait until it crashes
}

Hi,

Currently specification doesn't include version and/or published date. May I ask if you could do so?

Just for example, one format CBOR has details about published date. It doesn't have version, but (I think) the document name RFC 7049 can be used like a version name.

In terms of using version, may I ask if you could consider adopting semantic versioning?

Thank you,

Hey,

I did some test builds with various compilers for various architectures.

On mips 32 be (gcc), the compiler halts with "invalid operands to binary % (have ‘u16 * {aka short unsigned int *}’ and ‘int’)".

I think what you meant to do at line 76, 100, 126 in binn.c is a modulo operation on a pointer address rather than a pointer.

Like:
if (psource % 2 == 0)
if ((uintptr_t)psource % 2 == 0)

For your awareness, Clang 18 throws the following warnings:

binn/binn.c:3343:54: warning: implicit conversion from 'int' to 'float' changes value from
      2147483647 to 2147483648 [-Wimplicit-const-int-float-conversion]
 3343 |     if (value->vfloat < INT32_MIN || value->vfloat > INT32_MAX) return FALSE;
      |                                                    ~ ^~~~~~~~~
E:/MinGW/include/stdint.h:83:19: note: expanded from macro 'INT32_MAX'
   83 | #define INT32_MAX 2147483647
      |                   ^~~~~~~~~~
binn/binn.c:3380:54: warning: implicit conversion from 'long long' to 'float' changes value from
      9223372036854775807 to 9223372036854775808 [-Wimplicit-const-int-float-conversion]
 3380 |     if (value->vfloat < INT64_MIN || value->vfloat > INT64_MAX) return FALSE;
      |                                                    ~ ^~~~~~~~~
E:/MinGW/include/stdint.h:84:19: note: expanded from macro 'INT64_MAX'
   84 | #define INT64_MAX 9223372036854775807LL
      |                   ^~~~~~~~~~~~~~~~~~~~~
Source/binn/binn.c:3384:56: warning: implicit conversion from 'long long' to 'double' changes value
      from 9223372036854775807 to 9223372036854775808 [-Wimplicit-const-int-float-conversion]
 3384 |     if (value->vdouble < INT64_MIN || value->vdouble > INT64_MAX) return FALSE;
      |                                                      ~ ^~~~~~~~~
E:/MinGW/include/stdint.h:84:19: note: expanded from macro 'INT64_MAX'
   84 | #define INT64_MAX 9223372036854775807LL
      |                   ^~~~~~~~~~~~~~~~~~~~~

It is possible to add a function to dump the value of a binn in json like format?

The possible prototype could be:

const char *binn_to_string(void *data, size_t data_len);

I think it will be very important for debug purpose.

Hello. I've created a small example to understand how binn works, but I'm not sure how to properly serialize arrays using the library. How should I serialize an array of uint8_t or uint16_t?

I'm currently using binn_object_set_object but this doesn't look correct, in fact I get nothing on the other side but null. I'm also serializing a size_t as a uint64, I don't know if there is a better way to do with in binn.

The example I created is below. I basically serialize some data, save it to a buffer, and then use that buffer to do the inverse operation. I'm hoping to use binn to transfer data using sockets, but I figured the buffer would be enough for testing.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <inttypes.h>
#include <string.h>

#include "binn.h"

// to test
struct my_struct {
	uint32_t num;
	size_t size;
	uint8_t buf[];
};

static uint8_t *host_to_buffer()
{
	binn *obj;
	uint8_t *buf;
	uint16_t a[] = {0x0001, 0x5502, 0x4409, 0x33A1, 0x22FC};
	size_t size = sizeof(a);
	
	obj = binn_object();
	
	binn_object_set_uint32(obj, "uint", 12345);
	binn_object_set_str(obj, "str", "hello world");
	binn_object_set_uint64(obj, "size", size);
	binn_object_set_object(obj, "array", a);  // what is the correct call?
	
	buf = malloc(binn_size(obj));
	memcpy(buf, binn_ptr(obj), binn_size(obj));
	
	binn_free(obj);
	
	return buf;
}

static void buffer_to_host(uint8_t *buf)
{
	binn *obj = binn_open(buf);
	
	uint32_t num = binn_object_uint32(obj, "uint");
	char *str = binn_object_str(obj, "str");
	size_t size = binn_object_uint64(obj, "size");
	uint16_t *a = binn_object_object(obj, "array");
	
	// print deseralization results
	printf("num=%"PRIu32", str=%s\n", num, str);
	if (a == NULL) {
		fprintf(stderr, "array is null\n");
	} else {
		for (int i = 0; i < size/sizeof(uint16_t); i++)
			printf("%04"PRIx16" ", a[i]);
		printf("\n");
	}
	
	
	binn_free(obj);
	free(buf);
}

int main()
{
	// save some data to a buffer using binn; then deserialize the data from buf
	uint8_t *buf = host_to_buffer();
	buffer_to_host(buf);
	return 0;
}

All key names should be "const char*" instead of "char*"

There variables are "maybe uninitialized" in these contexts. Nice lib btw!

binn.c: In function ‘copy_int_value’:
binn.c:2123:21: warning: ‘vint64’ may be used uninitialized in this function [-Wmaybe-uninitialized]
*(int *)pdest = (int) vint64;
^~~~~~~~~~~~
binn.c:2139:30: warning: ‘vuint64’ may be used uninitialized in this function [-Wmaybe-uninitialized]
*(unsigned int *)pdest = (unsigned int) vuint64;
^~~~~~~~~~~~~~~~~~~~~~
binn.c: In function ‘AddValue’:
binn.c:717:6: warning: ‘vint’ may be used uninitialized in this function [-Wmaybe-uninitialized]
if (vint >= INT32_MIN) {
^
binn.c:669:10: note: ‘vint’ was declared here
int64 vint;

I have managed to send a simple struct over TCP. Now, I am facing a new challenge. I need to send multiple layers of struct over TCP. Is that possible in C using your library? Multiple layers means struct A contains another struct B. Inside struct B, it contains struct C and so on.

Hi,

After reading source code, I found the magic number 0x1F22B11F. However specification doesn't mention it. Would you please update specification?

Thank you,

I am looking for ways to use binn library in C# but could not get it to work by editing the header file like below (adding dll export). Any direction you can point me to?

__declspec(dllexport) ALWAYS_INLINE BOOL Binn_object_set_str(binn *obj, char *key, char *str) {
return binn_object_set(obj, key, BINN_STRING, str, 0);
}

Hello,

Below is a valid map:

e1:0f:01:00:00:00:01:c0:00:00:00:03:41:41:41

It contains a key 1, which points to a blob of AAA (41:41:41). The blob size uses 4-bytes: 00:00:00:03.

However when blob size uses 1 byte 03, binn_map_get_blob() fails. According to specification, size can be either 1 or 4 bytes.

May I ask if you could check?

Thank you,

Hi, I was just curious if there are any performance benchmark numbers for binn?

For example, how do common operations compare to flatbuffers (flatcc) or maybe json libraries or any other serialization libraries. Like on the flatbuffers google page there is this: https://google.github.io/flatbuffers/flatbuffers_benchmarks.html

Currently I'm just playing around with storing structs in a serialized format in a key/value database. Not super worried about performance and really like the simplicity of binn. But it would be kind of cool to see how it stacks up vs larger libraries and such.

Thanks for the nice library!

I've seen a lot of changes over the last few months and many positive - increased attention to FIPS compliance which is very important to my intended efforts w/ AES256 standards compliance/interoperability with FPGA/PL implementations of AES256 standard and other implementations - very important to adhere to standards.

However the use of aes.hpp does not follow a standard support of C++ and C usages.

Currently aes.hpp contains:
#ifndef AES_HPP
#define AES_HPP

#ifndef __cplusplus
#error Do not include the hpp header in a c project!
#endif //__cplusplus

extern "C" {
#include "aes.h"
}

#endif //AES_HPP

and no issue with using aes.hpp to imply C++ as boost does where .hpp means C++ and this is fine. However this is used for C++ code that includes C++ keywords like class, private, etc... basically class definitions that if used in a .c module would look out of place - its .hpp so it goes with .cpp

However aes.hpp is not correct usage.

aes.h needs to have function prototypes only - not preprocessor symbols or structs - these are common in C and C++ but only function prototypes to use the __cplusplus reserved pre-processor symbol for C++ but not recognized in C.

Furthermore using prepended and post underscores is not necessary and does not comply w/ the standard as the use of _ with preprocessor symbols is reserved by the compiler - eg. TIME but also other symbols

#ifndef AES_H
#define AES_H

needs only be AES_H

aes.h to be compliant with C/C++ interoperability standards (ANSI C and C99) would be

#ifndef AES_H
#define AES_H
...
#ifdef __cplusplus
extern "C" {
#endif

/* function prototypes only ... /
void AES_ECB_encrypt(struct AES_ctx ctx, const uint8_t* buf);
...

#ifdef __cplusplus
}
#endif

Now this is usable by C and C++ transparently - aes.hpp does not need to be used at all unless you do want to create a C++ wrapper in aes.hpp and that would make sense for those C++ folks that want a class interface w/ private data encapsulating context etc...

its significantly better to support transparent usage than issue an #error from the preprocessor when a C module includes a .hpp file ...
if the .hpp includes a class keyword it will be flagged as an error in any case.

One could also an error in the .hpp as you've done if a C module includes it ... however it shouldn't be necessary for a C++ program to use a .hpp file if they only want the C functions to be extern "C" -- compilers do this with all the standard C library functions like strcpy, memcpy etc... where a C++ program can include stdlib.h without having to use or any other non-C++ include - C includes must use the above method to be usable transparently by C and C++ programs.

aes.h as it currently standards is not compliant to standard C and C++ usage.

This code produces hex output: e10f02000000022001000000032001

uint8_t out[1000];
binn* obj = binn_map();
binn_map_set_int16(obj, 2, 1);
binn_map_set_int16(obj, 3, 1);
memcpy(out, binn_ptr(obj), binn_size(obj)); //  << the said output

However, this code produces the following hex output: 306000009dc8e10f02000000022001

uint8_t out[1000];
binn obj[1];
binn_create(obj, BINN_MAP, 1000, out);
binn_map_set_int16(obj, 2, 1);
binn_map_set_int16(obj, 3, 1);
// now the out[] contains the said output

The consequence:

// in the first case
binn_load((void*)out); // << works
binn_load((void*)(out + 6)); // << fails

// in the second case
binn_load((void*)out); // << fails
binn_load((void*)(out + 6)); // << works

From the outputs I see the data is arranged in different order:
1: e10f02000000022001000000032001
2: 306000009dc8e10f02000000022001

Is that normal? If so, how could one know when the data starts without calling binn_load() twice?

Hi,

I am upgrading one side of my PC to use C# instead of C while the other side must stay in C. I can send from C side no problem. The problem is at the C# receiving side. I am unable to create a binn object to use binn functions. I did try to use C# method to read incoming stream like below.

byte[] dataBuf = new byte[received];
Array.Copy(_buffer, dataBuf, received);
string text = Encoding.ASCII.GetString(dataBuf);
Console.WriteLine(text);

I am unable to de-serialize correctly. I tried to use the binn-1.0.dll in C# but I got stuck needing to create a binn object in C#. Could #anyone enlighten me?

If I do the following:

uint8_t out[1000]
binn obj[1];
binn_create(obj, BINN_MAP, 1000, out);
binn_map_set_blob(obj, 1, "data1", 5);
binn_map_set_blob(obj, 2, "data1", 5); // << from this point its impossible to load out[]

EDIT:
It works when using binn_map() instead of binn_create()

Hello developers,

I've cloned from this commit: 7b62ace

Then I included both binn.c and binn.h files into my Qt project. Unfortunately only one single function is preventing Qt from compiling my project. It's at line 262:

.../binn.h:262: error: expected ‘,’ or ‘...’ before ‘(’ token
 void   APIENTRY binn_set_alloc_functions(void* (*new_malloc)(size_t), void* (*new_realloc)(void*,size_t), void (*new_free)(void*));
                                                             ^

I don't need that function, so commenting it out will compile successfully.

I have a note that I'm new to both C and C++.

May I ask if you could give me some hint to solve this issue?

Thank you,

Hello binn developers ;)

I am really liking the work you have done on this project. I have been using your binn for logging purposes. I now would like to use it to keep track of, update, and record system variables used in my project. The issue I have hit upon is that I would like to update items that already exist in a binn object. Replace a binn value within a object with a new value, or free the value within the object that has the same key I am using and drop the new one into the object. Is this possible with the current state of binn - it didn't look like it is from looking at the source but I could be wrong? Has anyone else developed an addition of this nature to binn that I could use? Or should I look into doing this myself?

Thank you,
Matt

Hello,

In specification, blob size can be 1 or 4-byte block. However, as far as I understand your code, blob size always uses 4-byte block. (I know very basic C).

If that is true, I think your implementation should be updated, to support 1-byte block.

May I ask if you could check?

Thank you,

Hi,

can you write what is the correct syntax for reading a binn object from a socket?

what i am trying is:
sending:
binn * obj = binn_object();

binn_object_set_int32(obj, (char*)"dst_enclave", 306768318);

int socket = clSockFd;

if (send(socket, binn_ptr(obj), binn_size(obj), 0) != binn_size(obj)) {
	printf("SENDER -- failed to send msg1 to dest\n");
	abort();
}
binn_free(obj);

reading:

		binn * obj = binn_object();

		if (recv(servSockFd, binn_ptr(obj), binn_size(obj), 0) < 0) {
			printf("LISTENER -- failed to read msg1 from src\n");
			abort();
		}

		int test = binn_object_int32(obj, (char*)"dst_enclave");

but i am getting 0 in test.

Ciao there,

I'm new to this project and I'd like to ask whether or not it is possible to have nested objects.
For example:

{
"message":{
"type":"QUERY",
"timestamp":1234569900,
"source":{
"id":1,
"name":"NN",
"actor":"AGT"
}
}
}

Something like: binn_object_append_object("source", obj_source, obj_message)
Could you provide a code example?

Thanks.

Compiling binn on native IBM compiler.

"binn.c", line 36.9: 1506-236 (W) Macro name _BIG_ENDIAN has been redefined.
"binn.c", line 37.10: 1506-296 (S) #include file <machine/endian.h> not found.

In the case of the native compiler, the symbols in question are defined as:

#define __BIG_ENDIAN__ 1
#define _BIG_ENDIAN 1

Also, note that symbols with a leading underscore are considered reserved and should not be defined. Might want to make your endian detection code platform independent.

"Each name that begins with an underscore is reserved to the implementation for use as a name in the global namespace."

Hey there,

We use binn to help pass OOB performance statistics from the TCP_INFO struct in our implementation of ssh. It's proven to be very useful and we could have done it effectively without binn. Thank you for that. The only thing I was wondering if we could get some more API documentation (parameters, return values, error codes, etc) so we can have more robust exception handling in our code. I understand that documentation is a huge pain so if you don't have time I get it. I just wanted to pass along the request.

Again, thanks for writing this!

Chris

Out:
Contador: 1

Estoy intentando hacer un contador pero no funciona...
I'm trying to do a counter but it doesn't work ...

Can't call inline functions that declared in binn.h. (calling from crystal binding)
Why?

Hi here is the section of codes to test on my end.
binn *obj;
// create a new object
obj = binn_object();

//Assign stuff to this object
binn_object_set_int32(obj, "id", 123);
binn_object_set_str(obj, "name", "John");
binn_object_set_double(obj, "total", 2.55);
send(s , binn_ptr(obj) , binn_size(obj), 0);

Then I get an error saying "IntelliSense: argument of type "void *" is incompatible with parameter of type "const char *"

The libraries I use are:
#include<stdio.h>
#include<winsock2.h>
#include "binn.h"
#include <tchar.h>
#pragma comment(lib,"ws2_32.lib")

The current WinSock2.h library defines send() to be:
send(
__in SOCKET s,
__in_bcount(len) const char FAR * buf,
__in int len,
__in int flags
);

How did you overwrite the send() to accept object type?

Thanks,
Hubert

I've seen there was an issue about array serialization and it was "solved" by using lists instead. But it is quite inefficient in both space and speed, especially on embedded devices (while this library seems to be the best for embedded). To give an example, I send data (array of 100000 uint16) over network split to packets of about 512 bytes each. If I use lists, I need about 600 packets, while I can fit into 400 if I use blobs and interpret them on the other side. This is a 50% difference.
In my opinion this library will benefit a lot by adding support of lists of fixed type (and fixed size of each item). Do you think it will be possible? It may be even a small variation to list interpretation where values will be stored one after another and not storing values of different size or type than first item.
Currently I use objects with two keys: t and d. t stores type of each element while d stores blob of an array. And I'm almost fine with it (it doesn't handle endianness but I don't need that). But native support would be a lot better.

Is there an opportunity to store and then call a function from binn structures?

Hello Binn's authors,

First of all, I wan to thank you very much. Your project has helped me a lot over the years.

I have an implementation in Rust: https://docs.rs/binn-ir

But now, for personal reason, I'm unable to maintain it anymore. I would like to donate the project to you. That means you can re-license it as you like. That project is published under GNU Lesser General Public License (version 3 or later). I know you've published your specification under a different license. I've picked LGPL3 because of personal taste.

I'm proposing:

• Merging that project's source code into this project, under an orphaned branch.
• Transferring ownership of that project ( https://crates.io/crates/binn-ir ) to someone here. Please simply signing up at https://crates.io/ and give me your user ID.
• Personally, I've named it binn-ir because I didn't invent Binn format. Currently the name binn is still available. Maybe you can rename it if you want.

Please let me know what you think.

Thank you,

Hi, having troubles about figuring out how to use binn without wasting too much memory since i'm into a resource-limited environment (retrocomputing, DOS), and i want to know if opening and reading data also makes additional allocations, since i'm a bit confused how the inner works in binn.

this is my code for example:

    int mapsize = 0;
    char* mapdata = vfs_load(mapfile, &mapsize);
    if(mapdata == NULL)
        return -1;

    binn* maproot = binn_open(mapdata);
    binn_release(maproot);
    free(mapdata);

binn_open just allocates just the binn structure and nothing else and fetches the data directly from the buffer without copying?, and if is for string (i assume i have to copy them with memcpy, before i free the data buffer) and objects too?
i use mostly binn for reading map data, not writing.

Hi

Firstly thanks for a great module, however I have one problem I need to encode the first 4 bytes with an uint32_t. I have tried setting the first field of object to 0 and then try resetting to correct size after packing however second function call fails. Here is some code of what I am doing.

binn *obj;

obj = binn_object();

binn_object_set_uint32(obj, "length", 0);

// add more fields to object

int size = binn_size(obj);
binn_object_set_uint32(obj, "length", size);

Thanks
Simon

Hi,

i am trying to create dll file to use in c# project. i am getting following error:

cc -Wall -c src/binn.c
process_begin: CreateProcess(NULL, cc -Wall -c src/binn.c, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [binn.o] Error 2

i followed the step given in readme

git clone https://github.com/liteserver/binn cd binn make

i have GNU Make 3.81 installed on my system, Windows 10.

Also i tried to compile the code in src/win32 using visual studio 2019, it created binn-1.0.dll, but i am unable to add this file in c# project. i am getting following error

A reference to the "binn-1.0.dll" could not be added.Please make sure that the file is accessible and that it is a valid assembly or COM component.

it will be helpful if anybody can share steps for adding dll, or if i can add this dll for c# project or not ? Also as mention in readme

Both will create the file binn-3.0.dll

but when i build the project the output file is binn-1.0.dll, is this old repo am i doing something wrong ?
if any further info is required from my end please inform

I am having trouble understanding how to use binn_object_foreach. It would be great if someone could provide an example. I would like to be able to iterate through keys in the serialized data.

Thanks.