pjk / libcbor Goto Github PK

E.g. https://readthedocs.org/projects/libcbor/builds/8455505/

E.g. http://libcbor.readthedocs.io/en/latest/api/item_types.html

Something must have changed in RTD setup

Currently annoying to do due to old binutils, which results in linker issues (see e.g. Project-OSRM/osrm-backend#3216)

/usr/bin/gcc-6 -fPIC  -std=c99 -g -O0 -Wall -pedantic -g -ggdb -DDEBUG=true -fsanitize=address -fsanitize=bounds -fsanitize=alignment  -shared -Wl,-soname,libcbor.so.0.5.0 -o libcbor.so.0.5.0 CMakeFiles/cbor_shared.dir/cbor.c.o CMakeFiles/cbor_shared.dir/cbor/streaming.c.o CMakeFiles/cbor_shared.dir/cbor/internal/encoders.c.o CMakeFiles/cbor_shared.dir/cbor/internal/builder_callbacks.c.o CMakeFiles/cbor_shared.dir/cbor/internal/loaders.c.o CMakeFiles/cbor_shared.dir/cbor/internal/memory_utils.c.o CMakeFiles/cbor_shared.dir/cbor/internal/stack.c.o CMakeFiles/cbor_shared.dir/cbor/internal/unicode.c.o CMakeFiles/cbor_shared.dir/cbor/encoding.c.o CMakeFiles/cbor_shared.dir/cbor/serialization.c.o CMakeFiles/cbor_shared.dir/cbor/arrays.c.o CMakeFiles/cbor_shared.dir/cbor/common.c.o CMakeFiles/cbor_shared.dir/cbor/floats_ctrls.c.o CMakeFiles/cbor_shared.dir/cbor/bytestrings.c.o CMakeFiles/cbor_shared.dir/cbor/callbacks.c.o CMakeFiles/cbor_shared.dir/cbor/strings.c.o CMakeFiles/cbor_shared.dir/cbor/maps.c.o CMakeFiles/cbor_shared.dir/cbor/tags.c.o CMakeFiles/cbor_shared.dir/cbor/ints.c.o CMakeFiles/cbor_shared.dir/allocators.c.o 
/usr/bin/x86_64-linux-gnu-ld: unrecognized option '--push-state'

Is it fine for libcbor to be buildable with MSVC (at least with VS 2013)?

It is possible, but the cost is losing the C99 compliance:

1. Replace C99 restrict keyword with MSVC-specific __restrict.
2. The same is with inline - replace it with __inline.

These two replacements can be made via CMakeLists:

if(MSVC)
    set(CBOR_RESTRICT "__restrict")
    set(CBOR_INLINE "__inline")
else()
    set(CBOR_RESTRICT "restrict")
    set(CBOR_INLINE "inline")
endif()

3. #include <stdlib.h> instead of #include <unistd.h>.
4. Add const parameter qualifier to cbor_new_definite_map/array implementations (typo?).

The RFC indicates how to do canonical CBOR serialization. Is this something that libcbor can support? It shouldn't be hard to have it sort the keys by byte order in maps to get this supported

Maybe it's worth it to 'specialize' for cbor_load?

After a fresh install:

ls -h /usr/local/Cellar/libcbor/0.5.0/lib 
libcbor.0.0.0.dylib libcbor.0.dylib     libcbor.a           libcbor.dylib       pkgconfig

The README claims this is C99.
The docs site claims it is C11.
Please align...

To allow easy translation from other formats.

There are build functions for bools, but no reading functions. I expected something like cbor_get_bool but failed to find any.

All cbor_new_* and cbor_build_* functions can segfault in case memory allocation fails

(reported by an external user)

Cmake conf is done in a way that installs .so (or .so.X) files in /usr/lib regardless of target architecture (i686/x86_64). It should be placed in /usr/lib OR /usr/lib64 respectively.

Thanks

Hi, first of all, thanks for this great library.

It looks that both cbor_decref and cbor_move decrement number of references, however, when refcount reaches zero, cbor_decref will destroy an object. It seems that it is possible to make refcount equal to zero by using cbor_move, so I was wondering how a cbor_item_t with zero reference count could be used apart from moving it's content into cbor map or cbor array?

Thanks!

The file findings_000000 (hex dump below) causes the readline example to crash with a segfault due to a NULL pointer in common.c cbor_typeof carried over from main.c

00000000  e3 64 68 61 6c 66 f9 00  00 66 73 69 6e 67 6c 65  |.dhalf...fsingle|
00000010  fa 7f 7f ff ff 6d 73 69  6d 70 6c 65 20 76 61 6c  |.....msimple val|
00000020  75 65 73 83 f5 f4 f6                              |ues....|
00000027

backtrace

#0  0x000000000040670b in cbor_typeof (item=0x0) at libcbor/src/cbor/common.c:61
#1  0x000000000040370d in _cbor_nested_describe (item=0x0, out=0x7ffff7dd4400 <_IO_2_1_stdout_>, indent=0x0) at libcbor/src/cbor.c:260
#2  0x0000000000403e06 in cbor_describe (item=0x0, out=0x7ffff7dd4400 <_IO_2_1_stdout_>) at libcbor/src/cbor.c:388
#3  0x0000000000402fca in main (argc=0x2, argv=0x7fffffffe5c8) at libcbor/examples/readfile.c:39
#4  0x00007ffff7a36f45 in __libc_start_main (main=0x402ee5 <main>, argc=0x2, argv=0x7fffffffe5c8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffe5b8) at libc-start.c:287
#5  0x0000000000402e09 in _start ()

From here: http://tools.ietf.org/html/rfc7049#appendix-A
There is this example:
0.00006103515625 -> 0xf90400
It turns out the left hand side is 2^-14.
Using libcbor 0.3.1 on XCode simulator (little-endian), cbor_serialize_alloc fails after cbor_build_float2 for the provided number.
Actually it appears to get into an endless loop until aborted by an illegal malloc request.
Here's a function that demonstrates the issue:

• (void) testTwoToMinus14
  {
  cbor_item_t* pItem = cbor_build_float2(0.00006103515625);
  unsigned char* buffer;
  size_t buffer_size, length = cbor_serialize_alloc(pItem, &buffer, &buffer_size);
  XCTAssert(length);
  }

Attached is an image showing my test program and the results.

More constrained clients benefit from knowing the sizes upfront.

Jacob Teplitsky has contributed the following snippet. It should be quite easy to integrate it.

#include <string.h>
#include "platform.h"
#include "cbor.h"
#include "cbor/internal/builder_callbacks.h"
#include "cbor/internal/loaders.h"
#include "cJSON.h"
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>

#ifndef CBOR_LOAD_CALLBACK_T_DEFINED
typedef struct cbor_decoder_result(*cbor_load_callback_t)(void *, size_t,  const struct cbor_callbacks *, void *);
#endif


cbor_item_t *cjson_cbor_load(void *source,
                             size_t source_size,
                             struct cbor_load_result *result,
                             cbor_load_callback_t *cbor_load_callback)
{
	/* Context stack */
	static struct cbor_callbacks callbacks = {
		.uint8 = &cbor_builder_uint8_callback,
		.uint16 = &cbor_builder_uint16_callback,
		.uint32 = &cbor_builder_uint32_callback,
		.uint64 = &cbor_builder_uint64_callback,

		.negint8 = &cbor_builder_negint8_callback,
		.negint16 = &cbor_builder_negint16_callback,
		.negint32 = &cbor_builder_negint32_callback,
		.negint64 = &cbor_builder_negint64_callback,

		.byte_string = &cbor_builder_byte_string_callback,
		.byte_string_start = &cbor_builder_byte_string_start_callback,

		.string = &cbor_builder_string_callback,
		.string_start = &cbor_builder_string_start_callback,

		.array_start = &cbor_builder_array_start_callback,
		.indef_array_start = &cbor_builder_indef_array_start_callback,

		.map_start = &cbor_builder_map_start_callback,
		.indef_map_start = &cbor_builder_indef_map_start_callback,

		.tag = &cbor_builder_tag_callback,

		.null = &cbor_builder_null_callback,
		.undefined = &cbor_builder_undefined_callback,
		.boolean = &cbor_builder_boolean_callback,
		.float2 = &cbor_builder_float2_callback,
		.float4 = &cbor_builder_float4_callback,
		.float8 = &cbor_builder_float8_callback,
		.indef_break = &cbor_builder_indef_break_callback
	};

	if (source_size == 0) {
		result->error.code = CBOR_ERR_NODATA;
		return NULL;
	}
	struct _cbor_stack stack = _cbor_stack_init();

	/* Target for callbacks */
	struct _cbor_decoder_context context = (struct _cbor_decoder_context) {
		.stack = &stack,
		.creation_failed = false,
		.syntax_error = false
	};
	struct cbor_decoder_result decode_result;
	*result = (struct cbor_load_result) {.read = 0, .error = {.code = CBOR_ERR_NONE}};

	do {
		if (source_size > result->read) { /* Check for overflows */
			decode_result = (*cbor_load_callback)(
				source, 24,
				&callbacks,
				&context);
		} else {
			result->error = (struct cbor_error) {
				.code = CBOR_ERR_NOTENOUGHDATA,
				.position = result->read
			};
			goto error;
		}

		switch (decode_result.status) {
		case CBOR_DECODER_FINISHED:
			/* Everything OK */
		{
			result->read += decode_result.read;
			break;
		}
		case CBOR_DECODER_NEDATA:
			/* Data length doesn't match MTB expectation */
		{
			result->error.code = CBOR_ERR_NOTENOUGHDATA;
			goto error;
		}
		case CBOR_DECODER_EBUFFER:
			/* Fallthrough */
		case CBOR_DECODER_ERROR:
			/* Reserved/malformated item */
		{
			result->error.code = CBOR_ERR_MALFORMATED;
			goto error;
		}
		}

		if (context.creation_failed) {
			/* Most likely unsuccessful allocation - our callback has failed */
			result->error.code = CBOR_ERR_MEMERROR;
			goto error;
		} else if (context.syntax_error) {
			result->error.code = CBOR_ERR_SYNTAXERROR;
			goto error;
		}
	} while (stack.size > 0);

	/* Move the result before free */
	cbor_item_t *result_item = context.root;
	return result_item;

	error:
	result->error.position = result->read;
	/*debug_print("Failed with decoder error %d at %d\n",
     * result->error.code, result->error.position); */
	/* cbor_describe(stack.top->item, stdout); */
	/* Free the stack */
	while (stack.size > 0) {
		cbor_decref(&stack.top->item);
		_cbor_stack_pop(&stack);
	}
	return NULL;
}

struct cbor_decoder_result cjson_cbor_stream_decode(void *source_void, size_t source_size,
                                                    const struct cbor_callbacks *callbacks,
                                                    void *context)
{
    cJSON *source = source_void;
	/* If we have no data, we cannot read even the MTB */
	if (source_size < 1) {
		return (struct cbor_decoder_result) {0, CBOR_DECODER_EBUFFER};
	}

	/* If we have a byte, assume it's the MTB */
	struct cbor_decoder_result result = {1, CBOR_DECODER_FINISHED};

    const char *name = source->string;
    if (name) {
        callbacks->string(context, (unsigned char *)name, strlen(name));
    }
	switch (source->type) {
      case cJSON_False:
        {
		callbacks->boolean(context, false);
		return result;
        }
      case cJSON_True:
        {
            callbacks->boolean(context, false);
            return result;
        }
        /* case cJSON_NULL: */
      case cJSON_Number:
        {
			callbacks->uint32(context, source->valueint);
            return result;
        }
      case cJSON_String:
        {
            int                 res;
            struct sockaddr_in  sw_addr;
            size_t              len = 0;
            const char         *str;

            str = source->valuestring;
            res = inet_pton(AF_INET, str, &sw_addr.sin_addr);
            if (res > 0) {
                len = 4;
            } else {
                res = inet_pton(AF_INET6, str, &sw_addr.sin_addr);
                if (res > 0) {
                    len = 16;
                }
            }
            if (len == 0) {
                callbacks->string(context, (unsigned char *)source->valuestring, strlen(source->valuestring));
            } else {
                callbacks->byte_string(context, (unsigned char *)&sw_addr.sin_addr, len);
            }
            return result;
        }
      case cJSON_Array:
        {
            callbacks->array_start(context, cJSON_GetArraySize(source));
            cJSON *item;
            cJSON_Foreach_Array_Item_Begin(source, item)
            {
                cjson_cbor_stream_decode(item , 2, callbacks, context);

            }
            cJSON_Foreach_Array_Item_End(source, item)
            return result;
        }
      case cJSON_Object:
        {
			callbacks->map_start(context, cJSON_GetArraySize(source));
            cJSON *item;
            cJSON_Foreach_Object_Item_Begin(source, item)
            {
                cjson_cbor_stream_decode(item , 2, callbacks, context);

            }
            cJSON_Foreach_Object_Item_End(source, item)
            return result;
        }
      default: /* Never happens - this shuts up the compiler */
        {
            return result;
        }
	}
}

cbor_item_t *
cjson_to_cbor(cJSON *json)
{
    struct cbor_load_result result;
    cbor_load_callback_t call = &cjson_cbor_stream_decode;
    return (cjson_cbor_load((void *)json, 10, &result, &call));
}

static cbor_item_t * _cbor_copy_int(cbor_item_t * item, bool negative)
{
	cbor_item_t * res;
	switch (cbor_int_get_width(item)) {
	case CBOR_INT_8: res = cbor_build_uint8(cbor_get_uint8(item)); break;
	case CBOR_INT_16: res = cbor_build_uint16(cbor_get_uint16(item)); break;
	case CBOR_INT_32: res = cbor_build_uint32(cbor_get_uint32(item)); break;
	case CBOR_INT_64: res = cbor_build_uint64(cbor_get_uint64(item)); break;
	default: return NULL;
	}

	if (negative)
		cbor_mark_negint(res);

	return res;
}

static cbor_item_t * _cbor_copy_float_ctrl(cbor_item_t * item)
{
	switch (cbor_float_get_width(item)) {
	case CBOR_FLOAT_0:
		return cbor_build_ctrl(cbor_ctrl_value(item));
	case CBOR_FLOAT_16:
		return cbor_build_float2(cbor_float_get_float2(item));
	case CBOR_FLOAT_32:
		return cbor_build_float4(cbor_float_get_float4(item));
	case CBOR_FLOAT_64:
		return cbor_build_float8(cbor_float_get_float8(item));
	}

	return NULL;
}

cbor_item_t *cbor_copy_definite(cbor_item_t * item)
{
    if (item == NULL) {
        return (NULL);
    }
	switch (cbor_typeof(item)) {
	case CBOR_TYPE_UINT:
		return _cbor_copy_int(item, false);
	case CBOR_TYPE_NEGINT:
		return _cbor_copy_int(item, true);
	case CBOR_TYPE_BYTESTRING:
		if (cbor_bytestring_is_definite(item)) {
			return cbor_build_bytestring(cbor_bytestring_handle(item), cbor_bytestring_length(item));
		} else {
            size_t i;
			cbor_item_t * res = cbor_new_indefinite_bytestring();
			for (i = 0; i < cbor_bytestring_chunk_count(item); i++)
				cbor_bytestring_add_chunk(
					res,
					cbor_move(
						cbor_copy_definite(cbor_bytestring_chunks_handle(item)[i])
					)
				);
			return res;
		}
	case CBOR_TYPE_STRING:
		if (cbor_string_is_definite(item)) {
			return cbor_build_stringn((const char *) cbor_string_handle(item), cbor_string_length(item));
		} else {
            size_t i;
			cbor_item_t * res = cbor_new_indefinite_string();
			for (i = 0; i < cbor_string_chunk_count(item); i++)
				cbor_string_add_chunk(
					res,
					cbor_move(
						cbor_copy_definite(cbor_string_chunks_handle(item)[i])
					)
				);
			return res;
		}
	case CBOR_TYPE_ARRAY: {
		cbor_item_t * res;
        size_t i;

        if (cbor_array_size(item) == 0) {
            res = cbor_new_null();
        } else {
            res = cbor_new_definite_array(cbor_array_size(item));
            for (i = 0; i < cbor_array_size(item); i++)
                cbor_array_push(
                    res,
                    cbor_move(cbor_copy_definite(cbor_move(cbor_array_get(item, i))))
                    );
        }
		return res;
	}
	case CBOR_TYPE_MAP: {
		cbor_item_t * res;
        size_t i;

        res = cbor_new_definite_map(cbor_map_size(item));

		struct cbor_pair * it = cbor_map_handle(item);
		for (i = 0; i < cbor_map_size(item); i++)
			cbor_map_add(res, (struct cbor_pair) {
				.key = cbor_move(cbor_copy_definite(it[i].key)),
				.value = cbor_move(cbor_copy_definite(it[i].value))
			});
		return res;
	}
	case CBOR_TYPE_TAG:
		return cbor_build_tag(
			cbor_tag_value(item),
			cbor_move(cbor_copy(cbor_tag_item(item)))
		);
	case CBOR_TYPE_FLOAT_CTRL:
		return _cbor_copy_float_ctrl(item);
	}

	return NULL;
}

The subject says it. It does not look like there's anything special in my environment, it's a pretty stock FreeBSD 11.1 amd64 installation.

After git clone, creating a temp dir and running cmake from it, make cbor is successful, but make cbor_shared fails with:

[ 95%] Building C object src/CMakeFiles/cbor_shared.dir/cbor/ints.c.o
[100%] Linking C shared library libcbor.so
/usr/bin/ld: unrecognized option '-plugin'
/usr/bin/ld: use the --help option for usage information
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Is there a way to access the underlying cbor information including type information? I know that with (cbor_item_t)->data I can access the data of the item. But I am also interested in the type information, especially for arrays.

Is there a way?

We need to set the baseline

I see you have created .deb files, but I don't see the debian/ directory in git. Is it possible to have that committed as well, so I can get this setup with my internal repository? I need to compile this for amd64 and arm64, and I'd prefer to not have to recreate the configs you already have so far if possible.

Thanks.

Doxyfile contains
PROJECT_NUMBER = 0.3.1
would please be so kind and correct it with next relase?

Thanks

Hi,

I've tried to integrate the licbor into a QT lib. I've been running in several issues, all related in C99 C++11 related features. The lib is statically build and could be integrated easily, except that the headers generate errors while building:

• restrict not supported:
  ../../lib/qt/libcbor-qt/libcbor-0.3.1/src/cbor/bytestrings.h:62:79: error: expected ',' or '...' before 'data'
  void cbor_bytestring_set_handle(cbor_item_t *item, cbor_mutable_data restrict data, size_t length);
• initializer not supported
  ../../lib/qt/libcbor-qt/libcbor-0.3.1/src/cbor/callbacks.h:224:1: sorry, unimplemented: non-trivial designated initializers not supported

Using -std=c++11 nor -std=c99 (which fails for c++ / qt anyhow) doesn't solve the issue.

What can be done to solve it?

Way too painful + causes problems such as PJK/libcbor-ruby#2

Hi,

I'm new to CBOR and I'm having some trouble. I'm trying to encode something like this, and I'm not able to successfully implement it.

{
  var_1 : 0
  var_2 : 1
  var_3 : "some_string"
  var_4 : 
  {
    var_5 : true
    var_6 : true
  } 
}

I'm not able to encode var_1, what type should I use to encode value 0?
I'm able to encode vars 2 to 3 but I can not encode vars 4 to 6. Is it possible to do that with this library?

Thank you in advance

"CMakeFiles/cbor_shared.dir/cbor.c.o: file not recognized: File format not recognized"
I get this error when linking C library libcbor.so

Can someone help me ? thank you
Im running Debian i386

Can I build this library on a 32bit machine ?

Thank you for your time

Hello.

I found a NULL pointer dereference in libcbor.

Please confirm.

Thanks.

Summary: NULL pointer dereference
OS: CentOS 7 64bit
Version: commit f4ec5a4
PoC Download: crash.zip

Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./readfile $POC

ASAN:DEADLYSIGNAL
=================================================================
==31346==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000020 (pc 0x0000004f32d7 bp 0x7ffe7ef988f0 sp 0x7ffe7ef988d0 T0)
==31346==The signal is caused by a READ memory access.
==31346==Hint: address points to the zero page.
    #0 0x4f32d6 in cbor_typeof /home/karas/libcbor/src/cbor/common.c:61:15
    #1 0x4e9683 in _cbor_nested_describe /home/karas/libcbor/src/cbor.c:260:10
    #2 0x4e9df8 in _cbor_nested_describe /home/karas/libcbor/src/cbor.c:353:4
    #3 0x4e963e in cbor_describe /home/karas/libcbor/src/cbor.c:388:2
    #4 0x4e7d5d in main /home/karas/libcbor/examples/readfile.c:77:2
    #5 0x7fdca2c113f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #6 0x41ba09 in _start (/home/karas/libcbor/examples/readfile+0x41ba09)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/karas/libcbor/src/cbor/common.c:61:15 in cbor_typeof
==31346==ABORTING

================
[Acknowledgement]
This work was supported by ICT R&D program of MSIP/IITP. [R7518-16-1001, Innovation hub for high Performance Computing]

In current master/src/cbor/encoding.c line 140, there appears to be a constant encoded value for 16-bit float NaN value as 0x00e700. I think this is a typo as RFC 7049 in Section 3.9 states that

If NaN is an allowed value, it must always be represented as 0xf97e00

Notice the first data octet difference e7 vs 7e.
The same 7e00 encoding comes up in the RFC 7049 NaN example in Appendix A.

Hello.

I found a heap-buffer-overflow bug in libcbor.

Please confirm.

Thanks.

OS: CentOS 7 64bit
PoC Download: crash_overflow.zip

Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./streaming_parser $POC

=================================================================
==28627==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000002a at pc 0x0000004a7e60 bp 0x7ffeeb255cd0 sp 0x7ffeeb255480
READ of size 5 at 0x60300000002a thread T0
    #0 0x4a7e5f in printf_common(void*, char const*, __va_list_tag*) /home/karas/src/llvm/projects/compiler-rt/lib/ASan/../sanitizer_common/sanitizer_common_interceptors_format.inc:544
    #1 0x4a8bda in __interceptor_vprintf /home/karas/src/llvm/projects/compiler-rt/lib/ASan/../sanitizer_common/sanitizer_common_interceptors.inc:1388
    #2 0x4a8c97 in printf /home/karas/src/llvm/projects/compiler-rt/lib/ASan/../sanitizer_common/sanitizer_common_interceptors.inc:1434
    #3 0x5145f7 in find_string /home/karas/ASan/libcbor/examples/streaming_parser.c:31:3
    #4 0x514d11 in cbor_stream_decode /home/karas/ASan/libcbor/src/cbor/streaming.c
    #5 0x5148c9 in main /home/karas/ASan/libcbor/examples/streaming_parser.c:56:19
    #6 0x7f589bf27c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
    #7 0x41b71b in _start (/home/karas/ASan/libcbor/examples/streaming_parser+0x41b71b)

0x60300000002a is located 0 bytes to the right of 26-byte region [0x603000000010,0x60300000002a)
allocated by thread T0 here:
    #0 0x4df0b6 in __interceptor_malloc /home/karas/src/llvm/projects/compiler-rt/lib/ASan/ASan_malloc_linux.cc:66
    #1 0x514805 in main /home/karas/ASan/libcbor/examples/streaming_parser.c:48:27
    #2 0x7f589bf27c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/karas/src/llvm/projects/compiler-rt/lib/ASan/../sanitizer_common/sanitizer_common_interceptors_format.inc:544 in printf_common(void*, char const*, __va_list_tag*)
Shadow bytes around the buggy address:
  0x0c067fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c067fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c067fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c067fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c067fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c067fff8000: fa fa 00 00 00[02]fa fa fa fa fa fa fa fa fa fa
  0x0c067fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==28627==ABORTING

=================
[Acknowledgement]
This work was supported by ICT R&D program of MSIP/IITP. [R7518-16-1001, Innovation hub for high Performance Computing]

I tried to test the example cjson2cbor.c. I used cjson.h from https://github.com/DaveGamble/cJSON
and I built the library into the project. If I run this example with the input json_example.cjson:

{
"a": [1, 2.2, -1, "žluťoučký kůň!!!"],
"b": {
"c": [true, false, null]
}
}

the corresponding output is:
?aa?@
??;vžluťoučký kůň!!!ab?ac????

but verifying with the tool cbor.me I expected something like this:

A2 # map(2)
61 # text(1)
61 # "a"
84 # array(4)
01 # unsigned(1)
FB 400199999999999A # primitive(4612136378390124954)
20 # negative(0)
76 # text(22)
C5BE6C75C5A56F75C48D6BC3BD206BC5AFC588212121 # "\xC5\xBElu\xC5\xA5ou\xC4\x8Dk\xC3\xBD k\xC5\xAF\xC5\x88!!!"
61 # text(1)
62 # "b"
A1 # map(1)
61 # text(1)
63 # "c"
83 # array(3)
F5 # primitive(21)
F4 # primitive(20)
F6 # primitive(22)

I note that the hexadecimal numbers are converted in ASCII. For example 61 = a
but A2 should be =ì instead of =?. Same error for 84 and many others numbers.

Hi,

I'm trying to parse CBOR data and get the values of each field. I receive a payload from a client and already have the entire CBOR. I tried following the examples in https://libcbor.readthedocs.io/en/v0.5.0/using.html and https://libcbor.readthedocs.io/en/v0.5.0/streaming/decoding.html, but does not work. Is there an example of data parsing, getting the data from each field of the cbor?

Thank you

cbor_serialize returns 0 if data size was insufficient.

It would be useful to also get some report of how many more bytes should be supplied to finish serialization.

Or maybe additional function (something like size_t cbor_calculate(const cbor_item *)) should be added that calculates required data size.

cbor_serialize_alloc provides something similar, but it does it in another way. What I want is to alloc only once, and exactly as many bytes as it is required to store serialized data.

Also add docs on how to use it

otool says:

	/usr/local/opt/libcbor/lib/libcbor.0.dylib (compatibility version 0.0.0, current version 0.0.0)

Recently tried wrap your lib in objective - c (for iOS/MAC OS client - server exchange purposes) and bump into couple issues conformance with RFC 7049 standard.

First: wrong byte order after encoding
RFC 7049, Appendix A

       //...
        int value = 1000000;
        cbor_item_t *cborItem = cbor_build_uint32(value);

        unsigned char *buffer;
        size_t buffer_size,
        length = cbor_serialize_alloc(cborItem, &buffer, &buffer_size);

        NSData *bufferData = [NSData dataWithBytes:(const void *)buffer length:sizeof(unsigned char)*length];
        //...

Result binary data (in ascii hex): 0x1a40420f00
Must be (according to RFC 7049): 0x1a000f4240

_All bytes after first byte in reverse order._

Second: negative values of all wides of integers encoding incorrect. For example, if I even reorder bytes in correct way - result not confirms to RFC 7049:

       //...
        short value = -1000;
        cbor_item_t *cborItem = cbor_build_uint16(value);
        cbor_mark_negint(cborItem);

        unsigned char *buffer;
        size_t buffer_size,
        length = cbor_serialize_alloc(cborItem, &buffer, &buffer_size);

        NSData *bufferData = [NSData dataWithBytes:(const void *)buffer length:sizeof(unsigned char)*length];
        //...

Result binary data (in ascii hex): 0x39fc18
Must be (according to RFC 7049): 0x3903e7

Third: floats encoding into something absolutely crazy. But any float encoded before with cbor_build_float4 can't be decoded with cbor_float_get_float4... It always returns 0.0

       //...
        case CBOR_TYPE_FLOAT_CTRL:

            switch (cbor_float_get_width(itemFromData)) {
                case CBOR_FLOAT_0:{

                    if (cbor_ctrl_is_bool(itemFromData)) {

                        resultNumber = [NSNumber numberWithBool:cbor_ctrl_value(itemFromData)];

                    }

                }
                    break;
                case CBOR_FLOAT_16:
                    resultNumber = [NSNumber numberWithFloat:cbor_float_get_float2(itemFromData)];
                    break;
                case CBOR_FLOAT_32:
                    resultNumber = [NSNumber numberWithFloat:cbor_float_get_float4(itemFromData)];
                    break;
                case CBOR_FLOAT_64:
                    resultNumber = [NSNumber numberWithDouble:cbor_float_get_float8(itemFromData)];
                    break;

            }

            break;
        //...

There is chance, that I am using your lib in some wrong way, but I quit trying understand this... Maybe you can help me?

Hello,
I create the negative integer as such:

        cbor_item_t *hkdf_alg = cbor_new_int8();
        cbor_mark_negint(hkdf_alg);
        int abs_i = abs(ECDH_SS_HKDF_256) - 1;
        cbor_set_uint8(hkdf_alg, abs_i);

Where ECDH_SS_HKDF_256 = -27.

Then when calling int8_t item = cbor_get_uint8(hkdf_alg) I get a positive value. The strange thing is that when I copy paste the output of cbor for the hkdf_alg (which I have written to a file) to cbor.me I can see that the value is -27.

Am I missing something?

I'm now integrating libcbor library into my iOS project and I had all the source files included in the project. When I try to compile the library I got many errors:

1. Use of undeclared identifier 'CBOR_BUFFER_GROWTH' in strings.c, maps.c etc.
2. Too many arguments to function call: cbor_string_set_handle(item, handle, len)
   ...

many thanks for any suggestion.

To prevent e.g. #35

When running ctest on i686 arch, there is failed test_float8 with cbor_float_get_float8(float_ctrl) == 1.0e+300, on x64_86 arch works as expected.

build.txt

build_test log attached

E.g. http://lcamtuf.coredump.cx/afl/ via #15

Lots of undefined references in building some of the examples:

.../Library/Homebrew/shims/mac/super/clang  -std=c99 -O3 -flto -Wall -pedantic -DNDEBUG -Wl,-search_paths_first -Wl,-headerpad_max_install_names -flto CMakeFiles/readfile.dir/readfile.c.o  -o readfile ../src/libcbor.a 
undef: _cbor_build_bool
undef: _cbor_build_uint8
undef: _cbor_build_string
undef: _cbor_move
undef: _cbor_new_definite_map
undef: _cbor_map_add
undef: _cbor_serialize_alloc
undef: _cbor_decref
Undefined symbols for architecture x86_64:
  "_cbor_build_bool", referenced from:
      _main in lto.o
  "_cbor_build_uint8", referenced from:
      _main in lto.o
  "_cbor_build_string", referenced from:
      _main in lto.o
  "_cbor_move", referenced from:
      _main in lto.o
  "_cbor_new_definite_map", referenced from:
      _main in lto.o
  "_cbor_map_add", referenced from:
      _main in lto.o
  "_cbor_serialize_alloc", referenced from:
      _main in lto.o
  "_cbor_decref", referenced from:
      _main in lto.o
ld: symbol(s) not found for architecture x86_64
undef: _cbor_new_definite_array
undef: _cbor_get_uint8
undef: _cbor_build_uint8
undef: _cbor_array_push
undef: _cbor_move
undef: _cbor_array_handle
undef: _cbor_array_size
undef: _cbor_describe
Undefined symbols for architecture x86_64:
  "_cbor_new_definite_array", referenced from:
      _main in lto.o
  "_cbor_get_uint8", referenced from:
      _comparUint in lto.o
  "_cbor_build_uint8", referenced from:
      _main in lto.o
  "_cbor_array_push", referenced from:
      _main in lto.o
  "_cbor_move", referenced from:
      _main in lto.o
  "_cbor_array_handle", referenced from:
      _main in lto.o
  "_cbor_array_size", referenced from:
      _main in lto.o
  "_cbor_describe", referenced from:
      _main in lto.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [examples/create_items] Error 1
make[1]: *** [examples/CMakeFiles/create_items.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
undef: _cbor_empty_callbacks
undef: _cbor_stream_decode
Undefined symbols for architecture x86_64:
  "_cbor_empty_callbacks", referenced from:
      _main in lto.o
  "_cbor_stream_decode", referenced from:
      _main in lto.o
ld: symbol(s) not found for architecture x86_64
make[2]: *** [examples/sort] Error 1
make[1]: *** [examples/CMakeFiles/sort.dir/all] Error 2
undef: _cbor_load
undef: _cbor_describe
undef: _cbor_decref
Undefined symbols for architecture x86_64:
  "_cbor_load", referenced from:
      _main in lto.o
  "_cbor_describe", referenced from:
      _main in lto.o
  "_cbor_decref", referenced from:
      _main in lto.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [examples/streaming_parser] Error 1
make[1]: *** [examples/CMakeFiles/streaming_parser.dir/all] Error 2
make[2]: *** [examples/readfile] Error 1
make[1]: *** [examples/CMakeFiles/readfile.dir/all] Error 2
[100%] Built target hello
cd .../tmp/libcbor-20190119-19049-1m752yw/libcbor-0.5.0/build/src && .../Cellar/cmake/3.13.3/bin/cmake -E cmake_symlink_library libcbor.0.0.0.dylib libcbor.0.dylib libcbor.dylib
[100%] Built target cbor_shared
make: *** [all] Error 2

File findings_000100 causes a heap-buffer-overflow in cbor_new_definite_array in arrays.c line 111

cbor_item_t ** data = _CBOR_MALLOC(sizeof(cbor_item_t *) * size);
if (data == NULL) {
    _CBOR_FREE(item);
    return NULL;
}

for (size_t i = 0; i < size; i++)
    data[i] = NULL; // <==

size is 0x8000000000000000 and i is 0x4198 on crash

findings_000100 hex dump

a2 9b 80 00 00 00 00 00  00 00 00 00 00 00 00 00

backtrace

#0  0x0000000000406578 in cbor_new_definite_array (size=0x8000000000000000) at libcbor/src/cbor/arrays.c:111
#1  0x0000000000405925 in cbor_builder_array_start_callback (context=0x7fffffffe480, size=0x8000000000000000)
    at libcbor/src/cbor/internal/builder_callbacks.c:254
#2  0x00000000004048ba in cbor_stream_decode (source=0x60e251 "\233\200", source_size=0xf, callbacks=0x60d0e0 <callbacks>, context=0x7fffffffe480)
    at libcbor/src/cbor/streaming.c:413
#3  0x00000000004030b6 in cbor_load (source=0x60e250 "\242\233\200", source_size=0x10, result=0x7fffffffe4e0) at libcbor/src/cbor.c:68
#4  0x0000000000402fb0 in main (argc=0x2, argv=0x7fffffffe5e8) at libcbor/examples/readfile.c:37
#5  0x00007ffff7a36f45 in __libc_start_main (main=0x402ee5 <main>, argc=0x2, argv=0x7fffffffe5e8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffe5d8) at libc-start.c:287
#6  0x0000000000402e09 in _start ()

ASAN

==9484== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60040000dfd0 at pc 0x40a33b bp 0x7fffa1348c20 sp 0x7fffa1348c18
WRITE of size 8 at 0x60040000dfd0 thread T0
    #0 0x40a33a (readfile_asan+0x40a33a)
    #1 0x4087bd (readfile_asan+0x4087bd)
    #2 0x405e6c (readfile_asan+0x405e6c)
    #3 0x4036be (readfile_asan+0x4036be)
    #4 0x40344e (readfile_asan+0x40344e)
    #5 0x7f70ab268f44 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21f44)
    #6 0x403208 (readfile_asan+0x403208)
0x60040000dfd1 is located 0 bytes to the right of 1-byte region [0x60040000dfd0,0x60040000dfd1)
allocated by thread T0 here:
    #0 0x7f70ab62141a (/usr/lib/x86_64-linux-gnu/libasan.so.0.0.0+0x1541a)
    #1 0x40a2dc (readfile_asan+0x40a2dc)
    #2 0x4087bd (readfile_asan+0x4087bd)
    #3 0x405e6c (readfile_asan+0x405e6c)
    #4 0x4036be (readfile_asan+0x4036be)
    #5 0x40344e (readfile_asan+0x40344e)
    #6 0x7f70ab268f44 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21f44)
Shadow bytes around the buggy address:
  0x0c00ffff9ba0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9bb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9bc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9bd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9be0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c00ffff9bf0: fa fa fa fa fa fa fa fa fa fa[01]fa fa fa 00 00
  0x0c00ffff9c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9c10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9c20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9c30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c00ffff9c40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Found with American Fuzzy Lop

b263837 disables ARM CI because of changes in the Rapsbian image we used to use for QEMU. Either fix that setup, or use Docker to fully virtualize the build (travis-ci/travis-ci#3376)

The docs do not mention the cmocka dependency in the Building & Installing Prerequisites.
Also for cmake-newbies like me it would be nice to clarify the '# Assuming you are in the directory where you want to build' comment. I did not know where I wanted to build.. Turns out cmake expects you to create a build directory, and then run:

cmake -DCMAKE_BUILD_TYPE=Release ../

Just trying to smoothen the onboarding here ;-)

PS: I also added -DCMAKE_INSTALL_PREFIX:PATH=$HOME to the cmake commandline, since I don't want to actually install each library I try out. You may want to clarify that root permissions are not really required for installation.

I found that there is used by default GCC option -std=c99 and this option causes automatic definition of __STRICT_ANSI__ macro.
Because in MinGW some typedefs are protected by using this macro then they are not visible and library cannot be compiled i.e.:
#ifndef __STRICT_ANSI__
typedef __int64 off64_t;
#endif /* __STRICT_ANSI__ */

I try to define C_FLAGS in CMake configuration but user C_FLAGS is put before predefined one and then I cannot override them.
After I changed CMAKE_C_FLAGS in the CMakeLists.txt to:
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=gnu99")
then everything build without problems.

cmake 5.5.2
mingw32-gcc 3.9.3.-1

$ mingw32-make cbor
[ 7%] Building C object src/CMakeFiles/cbor.dir/cbor.c.obj
In file included from c:\mingw\include\unistd.h:36:0,
from C:\tmp\libcbor-0.4.0\src\cbor.c:245:
c:\mingw\include\io.h:301:14: error: unknown type name 'off64_t'
__CRT_INLINE off64_t lseek64 (int, off64_t, int);
^
c:\mingw\include\io.h:301:36: error: unknown type name 'off64_t'
__CRT_INLINE off64_t lseek64 (int, off64_t, int);
^
c:\mingw\include\io.h:302:14: error: unknown type name 'off64_t'
__CRT_INLINE off64_t lseek64 (int fd, off64_t offset, int whence) {
^
c:\mingw\include\io.h:302:39: error: unknown type name 'off64_t'
__CRT_INLINE off64_t lseek64 (int fd, off64_t offset, int whence) {
^
In file included from C:\tmp\libcbor-0.4.0\src\cbor.c:245:0:
c:\mingw\include\unistd.h:65:20: error: unknown type name 'off_t'
int ftruncate(int, off_t);
^
src\CMakeFiles\cbor.dir\build.make:62: recipe for target 'src/CMakeFiles/cbor.dir/cbor.c.obj' failed
mingw32-make[3]: *** [src/CMakeFiles/cbor.dir/cbor.c.obj] Error 1
CMakeFiles\Makefile2:1037: recipe for target 'src/CMakeFiles/cbor.dir/all' failed
mingw32-make[2]: *** [src/CMakeFiles/cbor.dir/all] Error 2
CMakeFiles\Makefile2:1049: recipe for target 'src/CMakeFiles/cbor.dir/rule' failed
mingw32-make[1]: *** [src/CMakeFiles/cbor.dir/rule] Error 2
Makefile:571: recipe for target 'cbor' failed
mingw32-make: *** [cbor] Error 2