google / libnop Goto Github PK
View Code? Open in Web Editor NEWlibnop: C++ Native Object Protocols
License: Other
libnop: C++ Native Object Protocols
License: Other
include/nop/base/map.h:142:35: error: loop variable ‘element’ of type ‘const std::pair<int, std::__cxx11::basic_string<char> >&’ binds to a temporary constructed from type ‘const value_type’ {aka ‘const std::pair<const int, std::__cxx11::basic_string<char> >’} [-Werror=range-loop-construct]
142 | for (const std::pair<Key, T>& element : value) {
| ^~~~~~~
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=984190
Is there a reason why arrays of floating point numbers are not being encoded in a binary container?
Hello,
I've looked through the docs and examples and this library fits very well with my needs but I'm not finding an important feature which is to be able to do zero copy serialization of external data. An example of this is when you have image data that needs to be read as a binary blob as part of a structure. Like so:
struct Sprite{
...other members
...
std::vector<uint8_t> img;
}
When writing such a struct I'd like to be able to directly read the image data into the serialization buffer. So it would be a API like so:
std::vector<uint8_t> buf(1024*1024);
nop::Serializer<nop::BufferWriter> encoder(buf.data(), buf.size());
int maxlen;
uint8_t* ptr = encoder.GetPtr(cost T& value, &maxlen); //Gets a pointer and maxlen only to the structure but doesn't copy anything
//...Here we can read image data directly into buf via read at ptr up to maxlen
...
int len = read(fd, ptr,maxlen);
...
encoder.Finish(len); //Finishes encoding the length
This avoids expensive memcpy for large binary chunks. Should be restricted to last member, and vector types in order to avoid holes.
Is there a way to do this or something to add to the library? Seems pretty straightforward. It can be done with: [0xBC 0x82 LENGTH(uint32_t) ...data follows...] just to keep it simple always use 32 bit unsigned for length?
Thanks, Best
Mark
https://buildd.debian.org/status/logs.php?pkg=libnop&ver=0.0~git20200728.45dfe0f-2
test/serializer_tests.cpp: In member function ‘virtual void Serializer_size_t_Test::TestBody()’:
test/serializer_tests.cpp:2970:19: error: conversion from ‘long long unsigned int’ to ‘std::size_t’ {aka ‘unsigned int’} changes value from ‘4294967296’ to ‘0’ [-Werror=overflow]
2970 | value = (1LLU << 32);
| ~~~~~~^~~~~~
test/serializer_tests.cpp:2978:13: error: conversion from ‘long long unsigned int’ to ‘std::size_t’ {aka ‘unsigned int’} changes value from ‘18446744073709551615’ to ‘4294967295’ [-Werror=overflow]
2978 | value = 0xffffffffffffffffLLU;
| ^~~~~~~~~~~~~~~~~~~~~
https://github.com/google/libnop/blob/master/test/serializer_tests.cpp#L2968
does the right thing at runtime, and the compiler might not emit any code for the if().
But the contents is checked at build time, and you are treating these warnings as errors.
namespace example {
struct Person {
std::string name;
std::uint32_t age_years;
std::uint8_t height_inches;
std::uint16_t weight_pounds;
NOP_STRUCTURE(Person, name, age_years, height_inches, weight_pounds);
};
} // namespace example
int main(int argc, char** argv) {
using Writer = nop::StreamWriter<std::stringstream>;
nop::Serializer<Writer> serializer;
serializer.Write(example::Person{ "John Doe", 37, 72, 180 });
serializer.Write(std::vector<example::Person>{
{"John Doe", 37, 72, 180}, { "Jane Doe", 36, 69, 130 }});
const std::string data = serializer.writer().stream().str();
std::cout << "Wrote " << data.size() << " bytes." << std::endl;
// [write to file]
std::ofstream saveFile("e.bin", std::ios::binary);
saveFile.write(data.c_str(), data.size());
saveFile.flush();
saveFile.close();
return 0;
}
MSVC 2022 Result:
Wrote 8 bytes.
Warnings
CLANG LLVM Result:
Wrote 50 bytes.
why this not working with MSVC 2022?!
Whenever I try to serialize a vector, it crashes reporting a vector subscript out of range.
Here's a screenshot of the crash report and the code that caused it:
Debug Assertion Failed!
Program: ...CLionProjects\test_newint4\cmake-build-debug\test_newint4.exe
File: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\vector
Line: 1371
Expression: vector subscript out of range
For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.
(Press Retry to debug the application)
//
// Created by PC on 7/25/2019.
//
#include "catch.hpp"
#include <nop/serializer.h>
#include <nop/utility/buffer_reader.h>
#include <nop/utility/buffer_writer.h>
#include "Geographics/GeoPoint.h"
SCENARIO("check serializer and deserializer", "[de-serializer]")
{
GIVEN("A Serializer and a Deserializer with a 30 byte buffer")
{
constexpr std::size_t kBufferSize = 1048;
std::uint8_t buffer[kBufferSize];
nop::Serializer<nop::BufferWriter> serializer{buffer, kBufferSize};
nop::Deserializer<nop::BufferReader> deserializer{buffer, kBufferSize};
WHEN("Arthur at ravex, 25/07/2019 in areas 15, 658, 743 and 25")
{
GeoPoint in_position {-27.5872, -48.2968};
Person in_person {"Arthur", 21};
std::vector<uint32_t> in_areas {15, 658, 743, 25};
uint32_t in_timestamp = 1564057349;
serializer.Write(in_position);
serializer.Write(in_areas);
serializer.Write(in_person);
serializer.Write(in_timestamp);
THEN("Out should be the same as input")
{
GeoPoint out_position;
Person out_person;
std::vector<uint32_t> out_areas;
uint32_t out_timestamp;
deserializer.Read(&out_position);
deserializer.Read(&out_person);
deserializer.Read(&out_areas);
deserializer.Read(&out_timestamp);
REQUIRE(in_position == out_position);
REQUIRE(in_person == out_person);
REQUIRE(in_areas == out_areas);
REQUIRE(in_timestamp == out_timestamp);
}
}
}
}
I'm running on Windows and building with MSVC 19.20.27508.1 for x64 . Please note that it works perfectly on GCC.
The examples provided use objects and the NOP_STRUCTURE is defined for the class/struct. How can I define a structure for shared pointers instead?
The static assert fails with
static_assert(sizeof(T) != sizeof(T), "Encoding<T> must be specilaized for type T. Make sure to " "include the appropriate encoder header.");
NOP_STRUCTURE defined in class T for example looks like:
NOP_STRUCTURE(T, name);
Please advise on how to serialize a shared_ptr or a vector of shared_ptrs. @eieio
I have a situation where at deserialization time I'd like a view-only object like std::span (which I would give to another object's constructor where the data will be copied out). I'd like to implement my own nop::Encoding<std::span<uint8_t>>
specialization for that. Unfortunately, it cannot be done on that level of abstractions since I would need to access, say, buffer_[index_]
of BufferReader
(https://github.com/google/libnop/blob/master/include/nop/utility/buffer_reader.h#L80).
I can instead read the prefix_byte and length at the place where I would ideally just call deserializer.Read(&myspan)
, get the offset via deserializer.reader().capacity() - deserializer.reader().remaining()
, call deserializer.reader().Skip(length_bytes)
and use the offset to index into my buffer. However, it would be nicer to encapsulate this logic.
One option would be to add a Status<void*> Ptr() { return buffer_[index_]; }
method to BufferReader
, and perhaps similar methods returning a failure status for other readers.
Hi!
I'm using visual studio 2017 v141.
Tried to use libnop to serialize a vector of a userdefined class.
The problem arised at NOP_STRUCTURE not being detected as macro, but as a member function of the userdefined class.
This is the output:
1>------ Operación Compilar iniciada: proyecto: TreeStorage, configuración: Debug x64 ------
1>Element.cpp
1>c:\sdk\nop-master\nop\base\encoding.h(784): error C4146: operador unario menos aplicado a un tipo unsigned; el resultado aún no tiene signo
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\element.h(85): warning C4003: no hay suficientes argumentos para la invocación de macro "_NOP_FIRST_ARG" de tipo función
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\element.h(85): warning C4003: no hay suficientes argumentos para la invocación de macro "_NOP_REST_ARG" de tipo función
1>Records.cpp
1>c:\sdk\nop-master\nop\base\encoding.h(784): error C4146: operador unario menos aplicado a un tipo unsigned; el resultado aún no tiene signo
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\element.h(85): warning C4003: no hay suficientes argumentos para la invocación de macro "_NOP_FIRST_ARG" de tipo función
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\element.h(85): warning C4003: no hay suficientes argumentos para la invocación de macro "_NOP_REST_ARG" de tipo función
1>c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.16.27023\include\numeric(26): warning C4267: '=': conversión de 'size_t' a '_Ty'; posible pérdida de datos
1> with
1> [
1> _Ty=unsigned int
1> ]
1>c:\sdk\nop-master\nop\base\vector.h(65): note: vea la referencia a la creación de instancias de plantilla de función '_Ty std::accumulate<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types>>,unsigned int,nop::Encoding<T,void>::Size::<lambda_f56d3b10dc1e1e7371a8b1943862477a>>(const _InIt,const _InIt,_Ty,_Fn)' que se está compilando
1> with
1> [
1> _Ty=unsigned int,
1> T=std::vector<Element,std::allocator>,
1> _InIt=std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types>>,
1> _Fn=nop::Encoding<std::vector<Element,std::allocator>,void>::Size::<lambda_f56d3b10dc1e1e7371a8b1943862477a>
1> ]
1>c:\sdk\nop-master\nop\base\vector.h(59): note: durante la compilación de la función miembro de plantilla clase de 'size_t nop::Encoding<T,void>::Size(const std::vector<Element,std::allocator<_Ty>> &)'
1> with
1> [
1> T=std::vector<Element,std::allocator>,
1> _Ty=Element
1> ]
1>c:\sdk\nop-master\nop\base\serializer.h(56): note: vea la referencia a la creación de instancias de plantilla de función 'size_t nop::Encoding<T,void>::Size(const std::vector<Element,std::allocator<_Ty>> &)' que se está compilando
1> with
1> [
1> T=std::vector<Element,std::allocator>,
1> _Ty=Element
1> ]
1>c:\sdk\nop-master\nop\base\serializer.h(178): note: vea la referencia a la creación de instancias de plantilla clase de 'nop::Encoding<T,void>' que se está compilando
1> with
1> [
1> T=std::vector<Element,std::allocator>
1> ]
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\records.cpp(80): note: vea la referencia a la creación de instancias de plantilla de función 'nop::Status nop::Deserializer::Read<std::vector<Element,std::allocator<_Ty>>>(T *)' que se está compilando
1> with
1> [
1> _Ty=Element,
1> T=std::vector<Element,std::allocator>
1> ]
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\records.cpp(80): note: vea la referencia a la creación de instancias de plantilla de función 'nop::Status nop::Deserializer::Read<std::vector<Element,std::allocator<_Ty>>>(T *)' que se está compilando
1> with
1> [
1> _Ty=Element,
1> T=std::vector<Element,std::allocator>
1> ]
1>main.cpp
1>c:\sdk\nop-master\nop\base\encoding.h(784): error C4146: operador unario menos aplicado a un tipo unsigned; el resultado aún no tiene signo
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\element.h(85): warning C4003: no hay suficientes argumentos para la invocación de macro "_NOP_FIRST_ARG" de tipo función
1>c:\users\jmichel\source\repos\vitalzero\requis\treestorage\element.h(85): warning C4003: no hay suficientes argumentos para la invocación de macro "_NOP_REST_ARG" de tipo función
I tried to use the stream in the example, and the result was B9 00. I tried many ways, all of which were the result
Only the result of Writing STL Containers to a Stream is normal
The libnop github about user defined structures: https://github.com/google/libnop/blob/master/docs/getting-started.md#user-defined-structures
"The easiest and most flexible way annotate a user-defined type is by using the macro NOP_STRUCTURE(type, ... /* members */)."
My Code with a simple std::map:
My struct:
struct EVHand
{
double added_values = 0;
uint64_t count_hands = 0;
NOP_STRUCTURE(EVHand, added_values, count_hands);
};
std::string GetTextFile(const std::string& file_path)
{
const std::ifstream myfile(file_path, std::ifstream::binary | std::ifstream::in);
if (myfile.is_open())
{
std::stringstream buffer;
buffer << myfile.rdbuf();
return buffer.str();
}
return "";
}
namespace {
// Sends fatal errors to std::cerr.
auto Die() { return nop::Die(std::cerr); }
} // anonymous namespace
Serialize Data
template <typename T>
void SerializeDatatype(const T& value) {
// Create a serializer around a std::stringstream.
using Writer = nop::StreamWriter<std::stringstream>;
nop::Serializer<Writer> serializer;
// Write some data types to the stream.
serializer.Write(value) || Die();
const std::string data = serializer.writer().stream().str();
std::cout << "Wrote " << data.size() << " bytes." << std::endl;
//Save data
std::string path = std::string("C:\\Log\\") + "test.txt";
try {
std::ofstream ofs;
ofs.open(path, std::ofstream::binary | std::ofstream::out);
ofs << data;
ofs.close();
}
catch (std::exception & e)
{
std::cout << e.what() << std::endl;
}
}
Read Serialized Data
template<class T>
T ReadSerializedDatatype()
{
//1. Read whole Text File
const std::string file = GetTextFile("C:\\Log\\test.txt");
// Create a deserializer around a std::stringstream with the serialized data.
using Reader = nop::StreamReader<std::stringstream>;
nop::Deserializer<Reader> deserializer{ file };
// Read some data types from the stream.
T map_value;
deserializer.Read(&map_value) || Die();
return map_value;
}
Main
int main(int, char**) {
std::map<double,EVHand> map;
double amount = 2.5;
auto amount_key_not_found = map.find(amount) == map.end();
if (amount_key_not_found)
{
map.insert(std::make_pair(amount, EVHand()));
}
double amount_won = 4;
EVHand test;
test.added_values = amount_won + amount;
test.count_hands = map.at(amount).count_hands + 1;
map[amount] = test;
std::cout << "\n";
std::cout << "Added values: " << map.at(amount).added_values;
std::cout << "\n";
std::cout << "counted hands: " << map.at(amount).count_hands;
std::cout << "\n";
SerializeDatatype(map);
map = ReadSerializedDatatype< std::map<double, EVHand>>();
for (auto const& [key, val] : map)
{
std::cout << "Key: " <<key << "\n";
std::cout << "Counted hands: " << val.count_hands << "\n";
std::cout << "Value: " << val.added_values << "\n";
}
return 0;
}
Results:
Added values: 6.5
counted hands: 1
Wrote 13 bytes.
Key: 2.5
Counted hands: 0
Value: 0
Changing the std::map value to INT produces correct serialization.
So there has to be something wrong with how I NOP_STRUCT my struct?
Im not sure if questions like that are allowed here but I saw the Help wanted label so I guess its ok?
Hi, I am a newbie in C++ and everything...
Could I use this library to serialize my class with some complex data like :
struct BoneNode
{
std::string name;
glm::mat4 transMatrix;
std::vector<BoneNode> children;
int meshBoneId = -1;
int nonMeshBoneId = -1;
};
and
std::unordered_map<std::string, size_t> boneExMap
?
The loading speed of FBX is driving me crazy and forcing me to learn some new words like "binary files" and "serialize"...
And sorry for my poor English...
Hi there, did you make a performance test on it and other serializers like messagepack or protobuffers?
Hello!
Have you support user-defined structures with inheritance?
I have the following data structures where only Base class stores data and Derived classes are only interfaces around the Base class.
class Base
{
protected:
int a; //example of field
public:
virtual void foo(); //interface to override
NOP_STRUCTURE(Base, a);
};
class Derived : public Base
{
/*no data fields at all*/
public:
virtual void foo() override; // new interface
};
So, how can I serialize an instance of Derived class?
nop\base\array.h(192,18): error C2665: “nop::StreamWriterstd::stringstream::Write”:
template
static constexpr Status WritePayload(EncodingByte /prefix/,
const Type& value,
Writer* writer) {
auto status = Encoding::Write(Length * sizeof(T), writer);
if (!status)
return status;
return writer->Write(value.begin(), value.end());
}
simple_protocol.cpp
// Construct a serializer to output to a std::stringstream.
Serializer<StreamWriter<std::stringstream>> serializer;
// Write a message to the stream using the first overload of WriteMessage.
WriteMessage(&serializer, 1, 2, 3, 4) || Die();
In file included from include/nop/traits/is_fungible.h:28:0,
from include/nop/protocol.h:21,
from examples/shared.cpp:24:
include/nop/base/logical_buffer.h: In instantiation of ‘static constexpr nop::Status nop::Encoding<nop::LogicalBuffer<BufferType, SizeType, IsUnbounded>, typename std::enable_if<(! nop::IsIntegral<typename nop::ArrayTraits::ElementType>::value), void>::type>::ReadPayload(nop::EncodingByte, nop::Encoding<nop::LogicalBuffer<BufferType, SizeType, IsUnbounded>, typename std::enable_if<(! nop::IsIntegral<typename nop::ArrayTraits::ElementType>::value), void>::type>::Type*, Reader*) [with Reader = nop::BufferReader; BufferType = {anonymous}::Triangle [1]; SizeType = long unsigned int; bool IsUnbounded = true; nop::Encoding<nop::LogicalBuffer<BufferType, SizeType, IsUnbounded>, typename std::enable_if<(! nop::IsIntegral<typename nop::ArrayTraits::ElementType>::value), void>::type>::Type = nop::LogicalBuffer<{anonymous}::Triangle [1], long unsigned int, true, void>]’:
include/nop/types/detail/member_pointer.h:146:39: required from ‘static constexpr nop::Status nop::MemberPointer<First Class::, FirstPointer, Second Class::, SecondPointer, typename std::enable_if<nop::IsLogicalBufferPair<First, Second>::value, void>::type>::ReadPayload(nop::EncodingByte, Class*, Reader*, MemberList) [with Reader = nop::BufferReader; MemberList = nop::MemberList<nop::MemberPointer<{anonymous}::Triangle ({anonymous}::CPolyhedron::)[1], &{anonymous}::CPolyhedron::triangles, long unsigned int {anonymous}::CPolyhedron::, &{anonymous}::CPolyhedron::size, void> >; Class = {anonymous}::CPolyhedron; First = {anonymous}::Triangle [1]; Second = long unsigned int; First Class::* FirstPointer = &{anonymous}::CPolyhedron::triangles; Second Class::* SecondPointer = &{anonymous}::CPolyhedron::size]’
include/nop/base/value.h:64:32: required from ‘static constexpr nop::Status nop::Encoding<T, typename std::enable_if<nop::IsValueWrapper::value, void>::type>::ReadPayload(nop::EncodingByte, T*, Reader*) [with Reader = nop::BufferReader; T = {anonymous}::CPolyhedron]’
include/nop/base/encoding.h:132:38: required from ‘static constexpr nop::Status nop::EncodingIO::Read(T*, Reader*) [with Reader = nop::BufferReader; T = {anonymous}::CPolyhedron]’
include/nop/base/serializer.h:178:29: required from ‘constexpr nop::Status nop::Deserializer::Read(T*) [with T = {anonymous}::CPolyhedron; Reader = nop::BufferReader]’
include/nop/protocol.h:40:36: required from ‘static constexpr nop::Status nop::Protocol::Read(Deserializer*, T*) [with Deserializer = nop::Deserializernop::BufferReader; T = {anonymous}::CPolyhedron; Enable = void; ProtocolType = {anonymous}::Polyhedron]’
examples/shared.cpp:119:68: required from here
include/nop/base/logical_buffer.h:135:14: error: uninitialized variable ‘size’ in ‘constexpr’ function
SizeType size;
g++: 6.4.0
os:3.10.0-693.el7.x86_64
Are there any benchmark about the performance ?
I'm trying to build and run the test suite on my company's toolchain and I'm seeing an error in the Optional.Basic test. I managed to narrow it down to a small repro. I am using GCC 9.2.1. First, I modify the Makefile as follows:
-HOST_CFLAGS := -g -O2 -Wall -Werror -Wextra -Iinclude
+HOST_CFLAGS := -fstack-protector -g -O2 -Wall -Werror -Wextra -Iinclude
Then I build with make out/test
. And when I run out/test
I get:
[----------] 7 tests from Optional
[ RUN ] Optional.Basic
test/optional_tests.cpp:205: Failure
Value of: std::equal(expected.begin(), expected.end(), value.get().begin())
Actual: false
Expected: true
[ FAILED ] Optional.Basic (0 ms)
Which comes from here:
libnop/test/optional_tests.cpp
Lines 200 to 206 in a7e824f
EXPECT_TRUE
the entire suite passes. If I print the items of value.get()
I see that they are 1 and 20 rather than 10 and 20.
I am not sure whether this is a compiler bug or an issue in libnop, as I don't really know what -fstack-protector
does. Could someone with more expertise look into this? Perhaps running those tests with ASAN could shed some light on the issue... Thanks!
Are there any plans or reason why std::unordered_set
is not supported, but std::unordered_map
is?
Are there workarounds to support such types?
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.