dvidelabs / flatcc Goto Github PK

google/flatbuffers#4143 (comment)

Is there any interest in support for the Meson build system?
If so, please write a comment explaining how / why, or why not.

Meson makes it possible to have dependencies on schema files and cause code to recompile partially and properly when changing a schema.

It requires Ninja and Python 3.4. Python is not exposed in build scripts and Meson has no dependencies on other Python libraries.

Meson install on Ubuntu:

apt-get update
apt-get install python3-pip
pip3 install meson

MacOS:

brew update
brew install meson

build:

git clone myproject
cd myproject
mkdir build
meson . build

I already ported it to flatcc 0.4.3 with the exception of a proper AppVeyor build for Windows CI (anyone up for fixing this?)

https://github.com/dvidelabs/flatcc/tree/meson

This branch is not maintained but could be integrated into the main project with sufficient interest.

A demo project using Meson to build a custom project, including a setup script to install Meson is found here: https://github.com/dvidelabs/flatcc-meson-sample

CMake likely would have to be supported as well, or are there no strong feelings about this?

Hello,

We hit an annoying issue using clang 3.4.1 on FreeBSD 10.3, but only on 32bits systems (it works fine with the same version of clang on 64bits systems)

Please find attached a schema and a C code sample that triggers the problem.

Steps to reproduce:

$ flatcc my_ns.fbs -cwv --outfile=./my_ns.fbs.h

$ cc -Wall -std=c99 test.c -o test -Iflatcc-0.4.1//include/ -DFLATCC_REFLECTION=0 -DFLATCC_PORTABLE flatcc-0.4.1/lib/libflatccrt.a

$ ./test
Assertion failed: ((B->frame[0].type) == flatcc_builder_table), function flatcc_builder_check_required, file <...>/flatcc-0.4.1/src/runtime/builder.c, line 1242.
Abort trap (core dumped)

If I add only "TEST NAME" or "TEST VALUE", it works as expected.
Maybe I do something wrong, but it is strange since it works fine on 64bit arch?

Regards

example.zip

First off -- this is probably a foolish error on my part.

But here goes. Given:

#include <stdio.h>
#include "qmsg.h"

#undef ns
#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(QQQ, x)

int main()
{
  flatcc_builder_t builder;
  flatcc_builder_init(&builder);

  ns(Msg_start_as_root(&builder));
    ns(Msg_payload_start_as_root(&builder));
      ns(Payload_someId_add(&builder, 456));
    ns(Msg_payload_end_as_root(&builder));
  ns(Msg_end_as_root(&builder));

  size_t bufSize;
  uint8_t *bufPtr = (uint8_t*)flatcc_builder_get_direct_buffer(&builder, &bufSize);
  ns(Msg_table_t) msgTable = ns(Msg_as_root(bufPtr));
  ns(Payload_table_t) nestedPayload = ns(Msg_payload_as_root(msgTable));

  uint32_t someId = ns(Payload_someId(nestedPayload)); // <--- Why a segfault here?
  printf("someId: %u\n", someId);

  return 0;
}

With the following schema:

namespace QQQ;

table Payload
{
  someId:uint;
}

table Msg
{
  payload:[ubyte] (nested_flatbuffer: "Payload");
}

root_type Msg;

I'm segfaulting on the uint32_t someId = ns(Payload_someId(nestedPayload)); line:

bt
#0  0x00000000004008a3 in __flatbuffers_soffset_read_from_pe (p=0x10060b9c0) at flatcc/include/flatcc/flatcc_endian.h:83
#1  0x0000000000400a41 in QQQ_Payload_someId (t=0x10060b9c0) at qmsg.h:943
#2  0x0000000000400fdf in main () at fbs_crash.c:24

Would you happen to know why? I've basically tried to follow this example:
https://github.com/dvidelabs/flatcc/blob/master/test/monster_test/monster_test.c#L1736
And as far as I can see, I have done so. Hopefully I'm missing something silly?

Also, my build script (linux):

FLATBUFFERS_DIR=../flatcc
FLATCC_EXE=../flatcc/bin/flatcc
FLATCC_INCLUDE=$FLATBUFFERS_DIR/include
FLATCC_LIB=$FLATBUFFERS_DIR/lib

$FLATCC_EXE --stdout -a qmsg.fbs > qmsg.h || exit 1
gcc -I$FLATCC_INCLUDE -g -o fbs_crash fbs_crash.c -L$FLATCC_LIB -lflatcc -lflatccrt || exit 1
gdb --args ./fbs_crash

From time to time, I see garbage in generated json strings. This issue appears to be present in both 0.4.3 (with the "JSON parser" and the "alligned_free" patches), 0.5.0 and 0.5.1-pre (commit ba9a8e6).

For testing, I'm sending the very same json string in a websocket request, that results in the very same json string for the response. Sometimes garbage is seen at the end of the json string.
When garbage is seen, the response is not sent - the write function of libwebsocket apparently don't like the garbage.
So far, I've seen the garbage strings "pU" and "�U" .

From the test cycles I've executed, it appears that garbage is NEVER seen in the first response, but then it appears randomly.

Please find attached the fbs: fbs.zip

Below is an example of an test cycle

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}�U

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}�U

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}�U

JSON request:
{"msg_type":"GetLdvReq","msg":{"dvName":"Measurement_01"}}
JSON response:
{"msg_type":"GetLdvRsp","msg":{"st":{"code":0},"dv":
{"name":"Measurement_01","type":12,"value":"4.040000"}}}

(A NICETOHAVE enhancement, nothing wrong with decimal even if sometimes annoying ;-)

As a C programmer, I often find that hex/octal and simple arithmetic/logic expressions useful.

For example:

enum FOO:uint {
A = 256+1,
B = 0xffff,
C = (0x10000 | 1),
Z = 0x3fffffff,
}

json printer like void _XXX_print_json_enum(flatcc_json_printer_t *ctx, int8_t v) does not have "static" prefix, when include in multi files, gcc will complain multiple definition of these functions.

Pick up conversation from #12 related to compilation on big endian platforms such as some IBM AIX systems.

And related, support for FLATBUFFERS_PROTOCOL_IS_BE with big endian encoded flatbuffers.

To transparently handle both situations, the term native endian is used for hosts endianness, and protocol endian (pe suffix on some operations) indicate the endianness encoded into the buffer. For official flatbuffers this is always little endian but the flatcc implementation does not hardcode little endian assumptions.

In summary, it appears that reading flatbuffers on big endian largely works, at least the monster test buffer created by googles flatc tool can be read and verified by a flatcc reader on big endian platforms, but buffer building and json handling have known issues.

Two fixes have so far been provided after release 0.3.5a:

vtables were only partially converted to protocol endian (which is bad), and hash table used to deduplicate vtables were prone to many hash collisions (which is inefficient)

structs created with an argument list would convert to protocol endian twice, which wasn't helpful. Most struct handling works without this fix though.

There are several other issues being investigated.

Hello,

Using gcc 5.4.0 in C99 mode does not work.
Here is is the compile issue:

             from /usr/home/user/git/src/test/test.c:2:

/usr/include/flatcc/portable/paligned_alloc.h:81:23: fatal error: mm_malloc.h: No such file or directory
compilation terminated.
*** Error code 1

I guess you include this file in order to make sure that posix_memalign is defined?
The problem is there is no more mm_malloc.h file shipped with gcc5.

Regards,

I'm testing the json parser in flatcc, and the parsing fails with an error that I doesn't really make any sense, taking the fbs and the json input into account.
This is the output I get from the test program (the json input followed by an error message):

{
  "data_type": "Ping",
  "data": {
    "msg": 123
  }
}
fbwsecho_ping.json:6:3: 20 unterminated string

I don't see any unterminated strings in the json input file, nor is the schema defining a string types for the Ping table in use.

Please find below the fbs and a code snippet, which basically is a extract of the json test program provided with the flatcc v0.4.3 sources.

Would someone please be so kind to tell me what I'm doing wrong?

Thanks.

pingpong.fbs:

union Data { Ping, Pong }

table Ping {
  msg:short;
}

table Pong {
  msg:string;
}

table Message {
  data:Data;
}

root_type Message;

main.c:

  char *buf = NULL;
  void *flatbuffer = NULL;
  size_t in_size, out_size;
  flatcc_json_parser_t ctx;
  flatcc_builder_t builder, *B = &builder;
  int ret = -1, flags = 0;

  flatcc_builder_init(B);
  buf = readfile(filename, FILE_SIZE_MAX, &in_size);
  if (!buf) {
    printf("%s: could not read input json file\n", filename);
    return -1;
  }
  printf("%s\n", buf);
  if (!pingpong_parse_json(B, &ctx, buf, in_size, flags)) {
    printf("%s: successfully parsed %d lines\n", filename, ctx.line);
    flatbuffer = flatcc_builder_finalize_aligned_buffer(B, &out_size);
    hexdump("parsed content", flatbuffer, out_size, stdout);
    printf("input size: %lu, output size: %lu\n",
        (unsigned long)in_size, (unsigned long)out_size);
    cdump("pingpong", flatbuffer, out_size, stdout);
  } else {
    printf("%s:%d:%d: %d %s\n",
            filename, (int)ctx.line, (int)(ctx.error_loc - ctx.line_start + 1),
            ctx.error, flatcc_json_parser_error_string(ctx.error));
  }

Considering you already have CI builds running, would it be possible for you to keep the built applications as artifacts from travis/appveyor and publish them on the github releases?

This can be done automatically as part of the build:
https://victorhurdugaci.com/github-releases-travis
https://www.appveyor.com/docs/deployment/github/

Question:
Is it guaranteed that there is full compatibility with flatbuffers?
I want to build the buffer with flatcc and read it on another machine with flatbuffers.
If compatibility is guaranteed where can I find with which version of flatbuffers each flatcc version is compatible?
Thanks

Fixed in this PR, but the reason why it works is not fully understood, and it may not cover all paths, e.g. perhaps the problem remains for structs while being fixed for tables.

#27

While writing a schema with a very large enum, I see this assertion failure

flatcc -ad --json --schema rpmtag.fbs
flatcc: /X/src/FLATBUFFERS/flatcc/src/runtime/builder.c:1396: flatcc_builder_create_offset_vector_direct: Assertion `vec[i] != 0' failed.

Removing --schema generates other files correctly.

The issue SHOULD be easy to reproduce: create an enum with something like 50 members.

My rpmtag.fbs schema is attached.
rpmtag.fbs.gz

This issue will opened and closed as needed for new releases.

The be branch will not be updated nor tagged for new releases, but branching off of it locally and merging master into it should enable native big endian.

@csstub please test for big endian for the 0.4.1 release, and prefereable also the native big endian format.

include/flatcc/portable/pendian.h includes the following constants in the bswap macros:

#define bswap32(v)                                                          \
      ((((uint32_t)(v) << 24))                                              \
          | (((uint32_t)(v) << 8) & UINT32_C(0x00FF0000U))                  \
          | (((uint32_t)(v) >> 8) & UINT32_C(0x0000FF00U))                  \
          | (((uint32_t)(v) >> 24)))
#endif

#ifndef bswap64
#define bswap64(v)                                                          \
      ((((uint64_t)(v) << 56))                                              \
          | (((uint64_t)(v) << 40) & UINT64_C(0x00FF000000000000ULL))       \
          | (((uint64_t)(v) << 24) & UINT64_C(0x0000FF0000000000ULL))       \
          | (((uint64_t)(v) << 8) & UINT64_C(0x000000FF00000000ULL))        \
          | (((uint64_t)(v) >> 8) & UINT64_C(0x00000000FF000000ULL))        \
          | (((uint64_t)(v) >> 24) & UINT64_C(0x0000000000FF0000ULL))       \
          | (((uint64_t)(v) >> 40) & UINT64_C(0x000000000000FF00ULL))       \

The problem comes in that UINT32_C and UINT64_C append U and ULL to the end of the constant passed to them, so the resulting text has U and ULL appended twice:

include/flatcc/flatcc_endian.h", line 76.3: 1506-258 (S) Hexadecimal integer constant 0x00FF000000000000ULLULL is not valid.

ULLULL is not a valid ANSI C99 specifier. GCC must let it thru, but other compilers do not.

Remove the extra constant specifiers from inside the bswap() macros and let UINT32/64_C add them naturally.

Note:
I'm doing testing of flatcc under AIX using the native IBM compiler, so I'll probably have some more suggestions/comments like this.

Thanks!

While struggling mightily with creating arrays of unions of arrays of types top-down,
I found a simple hack to reuse the previous offset for identical values.

Here is the code snippet I eventually stumbled upon adding
table STRING_ARRAY { v:[string] (required); }
with some simple logic to reuse the previous offset if/when strings are identical
.... case RPM_STRING_ARRAY_TYPE: { nsc(string_ref_t) refs[he->c]; for (size_t ix = 0; ix < he->c; ix++) { if (ix > 1 && !strcmp(he->p.argv[ix], he->p.argv[ix-1])) { fb->saved += strlen(he->p.argv[ix]) + 1; refs[ix] = refs[ix-1]; } else refs[ix] = nsc(string_create_str(B, he->p.argv[ix])); } ns(Tag_val_STRING_ARRAY_add(B, ns(STRING_ARRAY_create(B, nsc(string_vec_create(B, refs, he->c)) )) )); } break;

(aside)
RPM data contains many arrays of nearly identical items. The bytes fb->saved calculation
for my specific test case was modest but not trivial (the units below are in bytes):
headerSizeof: 7564 flatbuffer: 11730 saved: 688 json_nbuf: 25748
RPM has many sparse arrays of uint32_t that might benefit from a similar redundancy removal
(OTOH, KISS arrays of int* might preclude lazy redundancy removal, I'd have to measure)

Here's the code snippet creating uint32_t arrays (for reference), lazy redundancy removal for ints
would require changes to uint32_vec_create, and may not be feasible)
case RPM_UINT32_TYPE: ns(Tag_val_UI32_add(B, ns(UI32_create(B, nsc(uint32_vec_create(B, he->p.ui32p, he->c)) )) )); break;

The implementation is too obvious not to have been considered at some point in time ;-)

Looks like JSON parser generator sometimes generates corrupted path for parsing _type fields. I've attached a simple repro case:
test.fbs.txt
test.json.txt

After lots of experiments I found that fields with length from 1 to 3, 8 to 11, etc work correctly, while lengths 4 to 7, 12 to 15 lead to nonparseable field_type. From comments in generated parser file it seems that some symbols get eaten. I tried to debug code generator, found that field dictionary is built correctly, and then something bad happens in gen_trie.

Firstly, apologies in advance for the vagueness.

I can fill this in with more detail/debugging/bisecting/etc later if this doesn't ring a bell for you off the bat.

Somewhere in the range of d9a7559..v0.4.3 I started getting a lot more of these kinds of warnings on my buffers during verify:
buffer header too small
table header out of range or unaligned

Can you think of what might have started this of the top of your head?
I didn't change any build settings between building d9a7559 and v0.4.3.

Would you consider incorporating this tool directly into flatbuffers itself? It seems several people are interested here: google/flatbuffers#7.

In CMakelists.txt, the section that sets up the compiler flags for the final else case assumes you're using GCC.

if (CMAKE_C_COMPILER_ID MATCHES "Clang")
(snipped)
elseif (CMAKE_C_COMPILER_ID STREQUAL "GNU")
(snipped)
else()
    # Best effort
    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}" -Wall)
    set(FLATCC_PORTABLE true)
endif()

In my case, I'm using the system compiler, not GCC. I fall into the best effort case. But -Wall is a GCC flag, not a common compiler option. Shouldn't the "best effort" case be pretty generic? I removed -Wall for my testing as it fails otherwise.

It seems if the keyword in schema file is too long, It will be truncate in JSON parser and the result is the keyword can't be matched.
for example:
aaaa_bbbb_steps,
aaaa_bbbb_start_steps,
then the second one will be truncate to aaaa_bbbb_start, and result fail to match symbol.

Standard calloc() returns NULL if memory can't be allocated either due to out of memory condition or if either argument is 0.

In src/compiler/codegen_c_json_printer.c:

static int gen_json_printer_table(fb_output_t *out, fb_compound_type_t *ct)
(clipped)
    map = calloc((size_t)ct->count, sizeof(*map));
    if (!map) {
        gen_panic(out, "internal error: memory allocation failure");
        return -1;
    }

ct->count is zero, causing a null return, causing a failure of the test. To get around it quickly, I changed it to ct->count+1, but this condition should be handled.

Might want to check other uses of calloc() to make sure that zero is not being passed as num/size when its assumed that the return is non null.

Hi,

I'm not clear about the memory management when I was reading a buffer.

Suppose I have read a large size of float array (and I checked that flatcc generated header returns a const float * representing this array) from a serialized buffer file. Do I need to free this array if I don't need to use it anymore?

Hi,

I am seeing an issue where the data has a null value, eg. "field1":null and the parser generated fails with "expected string" error.

Looking at the generated parser.h, the buf = flatcc_json_parser_string_start(ctx, buf, end) returns the error as the buffer does not begin with ".

Is null value supported? If it is, how do I specify this in the schema?

Regards
Elvin

While compiling with gcc 4.7.2 together with glib-2.0 (2.40.2) I run into problems with the static assert feature.

In include/flatcc/portable/pstatic_assert.h:

...
#ifdef __has_feature

#if __has_feature(c_static_assert)
#define static_assert(pred, msg) _Static_assert(pred, msg)
#endif

#else
...

glib seems to define __has_feature but not _has_feature(c_static_assert) even though _Static_assert works for this particular version gcc. I don't know if this is as bug in glib or in flatcc, but just wanted to report it. My temporary workaround is just commenting out the #if __has_feature(c_static_assert) and corresponding #endif.

same schema file as input,
for client, I use FlatBuffers's flatc to generate javascript proto defination and sender.
for server, I use flatcc to generate proto handler.
when I verify the recv data in server use verify_as_root API,
It says: runtime: buffer header not aligned!

I noticed flatcc supports hex constant parsing:
0f3564a

This is a flatcc specific feature, correct? I couldn't find reference to '0x' ... type syntax listed beside "integer constant" in the flatbuffer schema grammar document at https://google.github.io/flatbuffers/flatbuffers_grammar.html

Having parsed a JSON string with success, I now try to extract the details from the flatbuffer. Below is the output from the app. First the JSON string is dumped, then the flatbuffer, and finally info on extracting the details

{
  "msg_type": "Ping",
  "msg": {
    "str": "123"
  }
}

successfully parsed 7 lines
parsed content:
  0000  08 00 00 00 50 49 50 4f  de ff ff ff 08 00 00 00  ....PIPO........
  0010  01 00 00 00 f0 ff ff ff  04 00 00 00 03 00 00 00  ................
  0020  31 32 33 00 06 00 08 00  04 00 08 00 09 00 08 00  123.............
  0030  04 00                                            ..
input size: 58, output size: 50
const unsigned char pingpong[] = {
    0x08, 0x00, 0x00, 0x00, 0x50, 0x49, 0x50, 0x4f,    0xde, 0xff, 0xff, 0xff, 0x08, 0x00, 0x00, 0x00,
    0x01, 0x00, 0x00, 0x00, 0xf0, 0xff, 0xff, 0xff,    0x04, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00,
    0x31, 0x32, 0x33, 0x00, 0x06, 0x00, 0x08, 0x00,    0x04, 0x00, 0x08, 0x00, 0x09, 0x00, 0x08, 0x00,
    0x04, 0x00,
};
Ping message detected
Ping.str:
  0000  ff ff ff 04 00 00 00 03  00 00 00 31 32 33 00 06  ...........123..
Ping.str=4026531840:"����"

As can be seen, I manage to detect the Ping message, and apparently I also get the string (at least I see the "123" text in the hex dump) - but the string length returned by nsc(string_len(s)) is way off (4026531840 instead of 3).

I've been reading the docs and the flatcc code with a magnifying glass, but I just can't identify what I am doing wrong. The big difference from my code and the test code, seems to be that I'm accessing a string in a table in a union.
What have I missed? (using latest flatcc master branch - december 20)

The fbs contains:

namespace eurisco.modiccp.mockup.pingpong;
attribute "priority";
union Message { Ping, Pong }
table Ping {
  str:string;
}
table Pong {
  str:string;
}
table Body {
  msg:Message;
}
root_type Body;
file_identifier "PIPO";
file_extension "msg";

And the test code is as follows:

#include <pingpong_builder.h>
#include <pingpong_reader.h>
#include <pingpong_verifier.h>
#include <pingpong_json_parser.h>
#include <pingpong_json_printer.h>
#include "flatcc/support/hexdump.h"
#include "flatcc/support/cdump.h"
#include "flatcc/support/readfile.h"

#ifdef NDEBUG
#define COMPILE_TYPE "(optimized)"
#else
#define COMPILE_TYPE "(debug)"
#endif

#define FILE_SIZE_MAX (1024 * 10)

#undef ns
#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(eurisco_modiccp_mockup_pingpong, x)
#undef nsc
#define nsc(x) FLATBUFFERS_WRAP_NAMESPACE(flatbuffers, x)

int verify_parse (void *buffer) {
  ns(Body_table_t) body = ns(Body_as_root_with_identifier)
      (buffer, ns(Body_identifier));
  ns(Ping_table_t) ping = ns(Ping_as_root_with_identifier)
      (buffer, ns(Ping_identifier));
  ns(Pong_table_t) pong = ns(Pong_as_root_with_identifier)
      (buffer, ns(Pong_identifier));
  ns(Message_union_type_t) msgType;
  nsc(string_t) pingStr;
  
  if (!(body = ns(Body_as_root(buffer)))) {
    printf("Body not available\n");
    return -1;
  }

  msgType = ns(Body_msg_type(body));
  switch (msgType) {
  case ns(Message_Ping):
    printf("Ping message detected\n");
    if (!(ping = ns(Ping_as_root(buffer)))) {
      printf("Ping not available\n");
      return -1;
    }
    if (!(pingStr = ns(Ping_str(ping)))) {
      printf("Ping.str not available\n");
      return -1;
    }  
    hexdump("Ping.str", (void*)pingStr, 16, stdout);
    printf("Ping.str=%zd:\"%s\"\n", nsc(string_len(pingStr)), (char*)pingStr);
    break;
  case ns(Message_Pong):
    printf("Pong message detected\n");
    break;
  default:
    printf("Unexpected message detected\n");
    break;
  }
  return 0;
}

void testParseJsonBuffer (const char *buf, size_t in_size) {
  void *flatbuffer = NULL;
  size_t out_size;
  flatcc_json_parser_t ctx;
  flatcc_builder_t builder, *B = &builder;
  int ret = -1, flags = 0;

  printf("%s\n", buf);
  flatcc_builder_init(B);
  if (!pingpong_parse_json(B, &ctx, buf, in_size, flags)) {
    printf("successfully parsed %d lines\n", ctx.line);
    flatbuffer = flatcc_builder_finalize_aligned_buffer(B, &out_size);

    hexdump("parsed content", flatbuffer, out_size, stdout);
    printf("input size: %lu, output size: %lu\n",
        (unsigned long)in_size, (unsigned long)out_size);

    cdump("pingpong", flatbuffer, out_size, stdout);
    verify_parse(flatbuffer);
  } else {
    printf("%d:%d: %d %s\n",
        (int)ctx.line, (int)(ctx.error_loc - ctx.line_start + 1),
        ctx.error, flatcc_json_parser_error_string(ctx.error));
  }

  if (flatbuffer) {
    flatcc_builder_aligned_free(flatbuffer);
  }
  flatcc_builder_clear(B);
}

void testParseJsonFile (const char *filename) {
  char *buf = NULL;
  size_t in_size;
  buf = readfile(filename, FILE_SIZE_MAX, &in_size);
  if (buf) {
    testParseJsonBuffer(buf, in_size);
    free((void *)buf);
  } else {
    printf("%s: could not read input json file\n", filename);
  }
}

Sometimes order of items in array is important, yet it there could be need to find some item in array by some key. In this case the only option is to perform linear search, which is still okay for small number of items (and small number of items seem to be quite frequent case for flatbuffers). Writing search functions manually can be tedious, so it could be great if flatcc generated them automatically, either giving them suffix _unsorted, or generating them in place of usual binary searches when some key is given to flatcc. Or maybe some other way (ukey attribute, instead of key?)

I want to use flatcc project in C++ application (__cplusplus defined). But I am using gcc version < 4.6. So I use -D FLATCC_PORTABLE option (for e.g. because stdalign.h does not exist).
All portables files like "pstdalign.h" uses c++ macros for exclusion, hence not executing.
#ifndef __cplusplus
#endif /* __cplusplus */

I can manually remove __cplusplus from these ps header to work. But want to check if it will cause any issues.

Thanks.

It's very convenient to store data in flatbuffers format, linearly load it into memory and just use generated accessors to traverse it. However, there are some cases when this loaded data needs to reference some other data, which can be stored elsewhere. While it's certainly possible to use strings or hashes as these references and manually dereference them when needed, this could be suboptimal when accessing one field many times, and error prone, since accessors must be written manually for each such field. It would be much better if there was a way to traverse such buffer and store pointers to external data, so that they could be used directly. Flatbuffers C++ API does have support for external references in unpack functionality, but for me it seems a bit excessive for the task. Since flatcc project already have some extensions to flatbuffers format I'll try to make a draft proposal for another one :)

1. in fb schema add support for field attribute (ref:typename), which will generate hidden field (just like union)
2. when generating binary flatbuffers write number of references needed by that buffer just after offset to the root table
3. use hidden ref fields to store offset into reference array
4. add function to reader interface to allocate array of pointers, traverse flatbuffers data and fill that array using callback in form of void * get_reference( const char * typename, const char * name )
5. generate functions to access these reference fields as pointers using table and reference array pointers

Another variation of this is to save enough space for pointers instead of offsets to reference array, this way API would be even more simple (no need for additional allocation, no need for providing additional pointer to reference array to accessor), but the drawback is that we need to make some assumption about size needed to store pointer (will 64 bit be enough for all cases?), and binaries will contain some junk data.

Any thoughts?

P.S. I'd very like to have this feature for my own pet project, so I'm quite willing to implement this in some form, as long as it have a chance to be accepted upstream :)

Should supply a more clear API doc

might be help if there were a section of security considerations in docs, like what kind of invalid protocol data may cause segment fault, buffer overflow, access violation.

Another head scratcher for me.

Given this:

#include <stdio.h>
#include "qmsg.h"

#undef ns
#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(QQQ, x)

int main()
{
  size_t nestedBufSize;
  uint8_t nestedBuffer[1024];

  // Build and store the nested buffer
  {
    flatcc_builder_t pBuilder;
    flatcc_builder_init(&pBuilder);
    ns(Payload_start_as_root(&pBuilder));
      ns(Payload_someId_add(&pBuilder, 456));
    ns(Payload_end_as_root(&pBuilder));

    nestedBufSize = flatcc_builder_get_buffer_size(&pBuilder);
    flatcc_builder_copy_buffer(&pBuilder, nestedBuffer, sizeof(nestedBuffer));
    flatcc_builder_clear(&pBuilder);
  }

  {
    printf("nested buffer: %p nested buffer size:%u\n", nestedBuffer, (uint32_t)nestedBufSize);
    ns(Payload_table_t) nestedPayload = ns(Payload_as_root(nestedBuffer));
    uint32_t someId_before_nesting = ns(Payload_someId(nestedPayload));
    printf("someId (before nesting): %u\n", someId_before_nesting); // <--- prints 456 as expected
  }

  {
    flatcc_builder_t builder;
    flatcc_builder_init(&builder);

    ns(Msg_start_as_root(&builder));
      //ns(Msg_payload_start_as_root(&builder)); <-- also tried with payload_start/end_as_root, prints "12" for someId after nesting.
      ns(Msg_payload_nest(&builder, nestedBuffer, nestedBufSize, 0)); // <--- also tried with 4 and 8 for align. Still prints 0
      //ns(Msg_payload_end_as_root(&builder));
    ns(Msg_end_as_root(&builder));

    size_t bufSize;
    uint8_t *bufPtr = (uint8_t*)flatcc_builder_get_direct_buffer(&builder, &bufSize);
    ns(Msg_table_t) msgTable = ns(Msg_as_root(bufPtr));
    ns(Payload_table_t) nestedPayload = ns(Msg_payload_as_root(msgTable));

    uint32_t someId = ns(Payload_someId(nestedPayload));
    printf("someId (after nesting): %u\n", someId); // <--- Why does this print 0 instead of 456?
  }

  return 0;
}

With this schema:

namespace QQQ;

table Payload
{
  someId:uint;
}

table Msg
{
  payload:[ubyte] (nested_flatbuffer: "Payload");
}

root_type Msg;

I'm expecting this line:

printf("someId (after nesting): %u\n", someId);

To print: "456". Any idea why it's printing 0 instead? Thank you.

Hi,
I have a project with a schema for messages where a union is used to include the contents of the message.

namespace UnionWithEmpty;
table RequestSomething {}
table Something { thing:uint; }
union MessageContents {RequestSomething, Something}
table Message {
    to:[uint] (required);
    from:uint;
    contents:MessageContents (required);
}
root_type Message;

However, when I try to create a RequestSomething message bottom up, I get a required assert. If i add a dummy field to RequestSomething, the example starts to work:

#include <stdio.h>
#include <assert.h>
#include "union_with_empty_builder.h"

#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(UnionWithEmpty, x)
#define ARRAYSIZE(x) (sizeof(x)/sizeof(x[0]))

static size_t create_works(uint8_t *buf)
{
  flatcc_builder_t builder;
  flatcc_builder_t *B = &builder;
  flatcc_builder_init(B);

  ns(Message_start_as_root(B));
  uint32_t to[] = {1,2,3};
  uint32_t from = 5;
  ns(Message_to_create(B, to, ARRAYSIZE(to)));
  ns(Message_from_add(B, from));

  ns(RequestSomething_ref_t) req = ns(RequestSomething_create(B));
  ns(Message_contents_add(B, ns(MessageContents_as_RequestSomething(req))));
  ns(Message_end_as_root(B));

  size_t size = flatcc_builder_get_buffer_size(B);
  flatcc_builder_copy_buffer(B, buf, size);

  flatcc_builder_clear(B);
  return size;
}

static size_t create_fails(uint8_t *buf)
{
  flatcc_builder_t builder;
  flatcc_builder_t *B = &builder;
  flatcc_builder_init(B);

  ns(RequestSomething_ref_t) req = ns(RequestSomething_create(B));

  ns(Message_start_as_root(B));
  uint32_t to[] = {1,2,3};
  uint32_t from = 5;
  ns(Message_to_create(B, to, ARRAYSIZE(to)));
  ns(Message_from_add(B, from));
  ns(Message_contents_add(B, ns(MessageContents_as_RequestSomething(req))));
  ns(Message_end_as_root(B));

  size_t size = flatcc_builder_get_buffer_size(B);
  flatcc_builder_copy_buffer(B, buf, size);  
  flatcc_builder_clear(B);
  return size;
}

static void decode(const uint8_t *buf)
{
  ns(Message_table_t) dec = ns(Message_as_root(buf));
  printf("type = '%s'\n", ns(MessageContents_type_name(ns(Message_contents_type(dec)))));
}

int main()
{
  uint8_t buf[1024];
  create_works(buf);
  create_fails(buf);
  decode(buf);
  return 0;
}

Thank you for making flatcc!

https://medium.com/@icex33/10-thousand-times-faster-swift-737b1accd973#.heh3ty70g

json_parser.c uses theisinf()function, which requires the math library to be linked in to any executable bound to the lib.a objects.

I don't know enough about Cmake to know where to add LDFLAG type options so that final targets that use the lib.a libraries get -lm added. Could you point me in the right direction?

My first impression came when I wrote up a simple schema file for testing, and forgot the semicolon after namespace id and the compiler crashed.

$ cat test.idl
namespace dummy
$ ./flatcc test.idl
test.idl:2:1: error: '': missing ';' expected by namespace at: test.idl:1:11: 'dummy'
*** Error in `./flatcc': corrupted double-linked list: 0x0000000000fa9490 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7beb2)[0x7f639aec6eb2]
/lib64/libc.so.6(+0x7cf36)[0x7f639aec7f36]
./flatcc(flatcc_destroy_context+0x41)[0x4065a8]
./flatcc(main+0x3ad)[0x40542d]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f639ae6cb15]
./flatcc[0x404459]
======= Memory map: ========
...

In fact the same happens if you leave the identifier out.

(gdb) bt
#0  0x00007f8af44a45f7 in raise () from /lib64/libc.so.6
#1  0x00007f8af44a5ce8 in abort () from /lib64/libc.so.6
#2  0x00007f8af44e4327 in __libc_message () from /lib64/libc.so.6
#3  0x00007f8af44eaeb2 in malloc_consolidate () from /lib64/libc.so.6
#4  0x00007f8af44ebf36 in _int_free () from /lib64/libc.so.6
#5  0x00000000004065a8 in flatcc_destroy_context (ctx=0x205c050) at /home/eddy/src/EXT/flatcc/src/compiler/flatcc.c:503
#6  0x000000000040542d in main (argc=2, argv=0x7ffe098e0938) at /home/eddy/src/EXT/flatcc/src/cli/flatcc_cli.c:436
(gdb) frame 5
#5  0x00000000004065a8 in flatcc_destroy_context (ctx=0x205c050) at /home/eddy/src/EXT/flatcc/src/compiler/flatcc.c:503
503         free(P);

I'm at revision ed90f67 (tag v0.3.5)

Attempting an enum definition like this
enum SUBTYPE:int {
XREF = -12,
BIN_ARRAY = -11,
REGION = -10,
}
leads to error messages like this

rpmtag_json_parser.h: In function ‘_RPM_SUBTYPE_parse_json_enum’:
rpmtag_json_parser.h:196:30: error: integer constant is so large that it is unsigned [-Werror]
*value = 18446744073709551605, *value_sign = 1;
^~~~~~~~~~~~~~~~~~~~

Testing the latest master.

During the "Scanning dependencies of target gen_reflection_test" phase, process goes runaway.

root 1286372 872504 93 12:45:32 pts/0 2:19 ../../../../bin/flatcc_d --schema -o /work/stupc/flatcc-master/build/Debug/test/reflection_test/generated /work/stupc/flatcc-master/test/monster_test/monster_test.fbs

Attaching with the debugger shows that its stuck in an endless loop inside flatcc_builder_create_cached_vtable().

Stack trace:

flatcc_builder_create_cached_vtable(B = 0x2ff22140, vt = 0x20072d9a, vt_size = 12, vt_hash = 1232720082), line 1082 in "builder.c"
flatcc_builder_end_table(B = 0x2ff22140), line 1243 in "builder.c"
reflection_Field_end(B = 0x2ff22140), line 30 in "reflection_builder.h"
reflection_Field_vec_push_end(B = 0x2ff22140), line 139 in "reflection_builder.h"
export_fields(B = 0x2ff22140, ct = 0x200029d8), line 167 in "codegen_schema.c"
export_objects(B = 0x2ff22140, objects = 0x20071a78, nobjects = 11, object_map = 0x20071b08), line 188 in "codegen_schema.c"
export_schema(B = 0x2ff22140, opts = 0x200011f8, S = 0x200012d8), line 281 in "codegen_schema.c"
__flatcc_fb_codegen_bfbs_alloc_buffer(opts = 0x200011f8, S = 0x200012d8, size = 0x2ff22274), line 394 in "codegen_schema.c"
__flatcc_fb_codegen_bfbs_to_file(opts = 0x200011f8, S = 0x200012d8), line 417 in "codegen_schema.c"
flatcc_generate_files(ctx = 0x20001178), line 468 in "flatcc.c"
main(argc = 5, argv = 0x2ff225b8), line 430 in "flatcc_cli.c"

If you continue, and break again, you're still inside the same loop:

flatcc_builder_create_cached_vtable(B = 0x2ff22140, vt = 0x20072d9a, vt_size = 12, vt_hash = 1232720082), line 1080 in "builder.c"
flatcc_builder_end_table(B = 0x2ff22140), line 1243 in "builder.c"
reflection_Field_end(B = 0x2ff22140), line 30 in "reflection_builder.h"
reflection_Field_vec_push_end(B = 0x2ff22140), line 139 in "reflection_builder.h"
export_fields(B = 0x2ff22140, ct = 0x200029d8), line 167 in "codegen_schema.c"
export_objects(B = 0x2ff22140, objects = 0x20071a78, nobjects = 11, object_map = 0x20071b08), line 188 in "codegen_schema.c"
export_schema(B = 0x2ff22140, opts = 0x200011f8, S = 0x200012d8), line 281 in "codegen_schema.c"
__flatcc_fb_codegen_bfbs_alloc_buffer(opts = 0x200011f8, S = 0x200012d8, size = 0x2ff22274), line 394 in "codegen_schema.c"
__flatcc_fb_codegen_bfbs_to_file(opts = 0x200011f8, S = 0x200012d8), line 417 in "codegen_schema.c"
flatcc_generate_files(ctx = 0x20001178), line 468 in "flatcc.c"
main(argc = 5, argv = 0x2ff225b8), line 430 in "flatcc_cli.c"

utility functions like buffer encrypt/compression with crc check might be very useful!

Comments like

/////////////////////////////////////////////////////////////
// Some big comment
/////////////////////////////////////////////////////////////

become

/**/////////////////////////////////////////////////////////////////////
* ///////////////////////////////////////////////////////////////////// */

in flatbuffer reader, which is clearly incorrect.

I hope to look into and fix this approximately by Wednesday.

For network buffer optimize:
I can get buffer directly by flatcc_builder_get_direct_buffer and send buffer data over socket API. But for large buffer, how can I iterate emitter buffers and send them one by one over socket, instead of call flatcc_builder_finalize_buffer to alloc and copy buffer data?

Let's say I have:

table SomeTable {
  id:uint;
  someStrings:[string];
  name:string;
  name1:string;
  name2:string;
  name3:string;
  name5:string;
  
  // Etc... Pretend this is a very large table with many, many members.
}

I have a whole bunch of these SomeTable flatbuffers saved out into separate binary files somewhere on my filesystem as root objects.

Now let's say that in my code, I want to create a new table comprised of a vector of those SomeTables flatbuffers:

table SomeTables {
  tables:[Sometable];
}

How can I build this SomeTables buffer as concisely as possible? That is, I'd very much prefer to avoid code like this (pseudocode):

ns(SomeTables_start_as_root(B));
ns(SomeTables_tables_start(B));
ns(SomeTables_tables_push_start(B));
for(thisTableFromMyFilesystem from someTablesOnMyFilesystem)
{
  ns(SomeTable_table_t) thisTable = ns(SomeTable_as_root(thisTableFromMyFilesystem));
  ns(SomeTables_tables_push_create(&b, flatbuffers_string_create_str(B, ns(SomeTable_name1(thisTable), ns(SomeTable_name2(thisTable), ...)); // <-- I have to update this line when the SomeTable schema changes.
}
ns(SomeTables_tables_push_end(B));
ns(SomeTables_tables_end(B));
etc...

In such an arrangement, I have to update the SomeTables_tables_push_create line when the SomeTable schema changes. I'd prefer a "Take this SomeTable_table_t, copy its contents into the flatbuffer, and push it onto this table vector" function.
Does such a thing exist? e.g. SomeTables_tables_push_create_from_table_t in this example?

ns(SomeTables_start_as_root(B));
ns(SomeTables_tables_start(B));
ns(SomeTables_tables_push_start(B));
for(thisFlatBufferFromMyFilesystem from someTablesOnMyFilesystem)
{
  ns(SomeTable_table_t) thisTable = ns(SomeTable_as_root(thisTableFromMyFilesystem));
  ns(SomeTables_tables_push_create_from_table_t(B, thisTable)); // With this, I don't have to worry about a long push_create() line where I pass in all the values...
}
ns(SomeTables_tables_push_end(B));
ns(SomeTables_tables_end(B));
etc...

My goal (likely naive ;-) is to attach custom bits to all table fields for additional
code generation.

Some examination of the "required" attribute seems to indicate that
flatcc generates
static const flatbuffers_voffset_t ___RPM_CHAR_required[] = {0 };

In principle, only a single bit is needed to test if a "required" field is present.
That would leave 15 bits in the flatbuffers_voffset_t that might be abused for
other purposes with (perhaps) minimal disruption to the existing code.

One possible usage would be (from monster_test.fbs) to implement "priority:1"
as a bit field (i.e. "1<<1" where "required" would be "1<<0") but that is perhaps
a bit hacky because limited to 15 bits rather than fully general attribute values.

Does my question make sense? AFAICT, attribute is parsed in a schema but unimplemented
(but I likely am -ENOCLUE ;-)

I think that could help write tests that will cover more code, improving overall quality. I've made some quick hacks to generate report with gcov so you could see what it looks like, but in order to be efficient this better be automated and run regularly. If you agree to use gcov/lcov I could provide help, since I already had experience with these tools.
flatcc-coverage.tar.gz

Commenting out a line following a C comment in a schema enum definition causes a generation failure:
flatcc -ad --json rpmtag.fbs
rpmtag.fbs:372:19: error: ' ': member identifier expected

The failure reproducer looks like this
enum TAGNO:uint {
...
FILESIGNATURELENGTH = 5091, /* i */

//  FIRSTFREE_TAG,                      /*!< internal */
}

Is it possible to supply an WinHex or Wireshark template for analysis binary data?

It seems that with latest gcc 7.1.1 and clang 4.0.1 (C++17 updates) code can't be compiled:

src/flatcc/include/flatcc/portable/pprintint.h:100:58: error: this statement may fall through [-Werror=implicit-fallthrough=]
#define __print_unaligned_copy_16(p, q) ((uint16_t)(p) = (uint16_t)(q))

adding "-Wimplicit-fallthrough=0" to CMakeLists.txt resolved problem for me.