only-cliches / noproto Goto Github PK

I'd love to see the bincode crate included in the benchmarks and comparisons.

Hey 👋 Another cool project of yours 👍

The README mentions that the library is WASM-ready. But is there any example of how to use it from Javascript?

Awesome work here! Reviewing the documentation, I don't see an immediately obvious option for being able to specify that a field may be one of a set of types. In Rust this would typically look like:

enum Variants {
    ObjA(ObjA),
    ObjB(ObjB),
}

I see that enums can be used for a known collection of ordered strings, and I see that there is also the any type. Would the idea be to use a combination of those items within a struct?

It looks like that RPC is done differently in 0.10, but is not quite implemented yet. Is that correct?

Could you summarize the principle differences between 0.10 and 0.9.6? It is difficult to see in the code, because of the move of the old source tree.

Thank you!

Hello!
I found weird bug. I tried to wrap NP_Buffer in struct like this:

use no_proto::{NP_Factory, buffer::NP_Buffer, error::NP_Error, memory::NP_Memory_Owned};

struct Test {
    buffer: NP_Buffer<NP_Memory_Owned>
}

impl Test {
    fn new(field: i64) -> Result<Self, NP_Error> {
        let factory = NP_Factory::new(r#"
            struct({fields: {
                field: i64()
            }})
        "#)?;
        let mut test = Self {
            buffer: factory.new_buffer(None)
        };
        test.set_field(field)?;
        Ok(test)
    }

    fn get_field(&self) -> Result<Option<i64>, NP_Error> {
        self.buffer.get::<i64>(&["field"])
    }

    fn set_field(&mut self, field: i64) -> Result<bool, NP_Error> {
        self.buffer.set(&["field"], field)
    }
}

fn main() -> Result<(), NP_Error> {
    let test = Test::new(0)?;
    assert_eq!(test.get_field()?, Some(0));
    Ok(())
}

On Windows 10 this code:

• sometimes works
• sometimes test.get_field() returns Ok(None)
• sometimes crashes with
  error: process didn't exit successfully: 'target\debug\np_test.exe' (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

On Ubuntu (WLS) this code always crashes with
Segmentation fault (core dumped)

Without struct everything work fine.

16 bit addresses are more compact but limit the address space to 64KB, switching to 32 bit addresses will allow buffers of up to 4GB in size.

It would be nice to add Veriform to the comparisons list.

When calling NP_RPC_Factory::new_bytes() with an empty byte slice, it causes a thread panic.

I am new to Rust, so excuse my question: I would have expected the function to fail with an NP_Error.

I am loading the rpc spec in bytes from a remote endpoint, so I wanted to check for validity.

On line https://github.com/only-cliches/NoProto/blob/master/bench/src/run_bench_flatbuffers.rs#L40, using the FlatBufferBuilder::reset method should greatly improve your FlatBuffers write performance. To do this, you'll need to make sure to-reuse a FlatBufferBuilder instead of creating a new one each time through the loop.

Hi,
I try to use NoProto and Sled as a simple DB and figured it would make sense to parse the Schema only once.
However, its all across an async code base and the inserts and updates would come only after some awaits happened.
This is seems to be a roadblock with NoProto, as NP_Factory is not Send.

I basically

1. start a Server
2. create a NP_Factory
3. wait for IO to happen
4. want to store that IO

Compiling my simple PoC gives me:

error: future cannot be sent between threads safely
   --> src/main.rs:82:38
    |
82  |     let server = Server::bind(&addr).serve(make_service);
    |                                      ^^^^^ future returned by `proxy` is not `Send`
    |
    = help: within `no_proto::NP_Factory`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<no_proto::schema::NP_Schema>`
note: future is not `Send` as this value is used across an await

I guess an Arc instead of an Rc (in NP_Factory and NP_SchemaKinds) would solve it.
Or am I doing it wrong?

Would be nice to use no_proto in places were trying to be as minimal as possible (like web assembly)

The benchmarks right now compare NoProto to formats that are fairly different - either the data carries along a schema (json), or the schema is precompiled (protobuf).

I think flexbuffers and avro are probably much better comparisons, as they're both formats where the schemas can be loaded dynamically. I think it would be very interesting to see comparisons to those two.

In the readme, in comparisons against each of Avro, Protobuf, JSON, and Flatbuffers, each comparison lists "Safely handle untrusted data" as a benefit of Noproto. I can't speak for Avro or FlatBuffers, but Protobuf is regularly used to handle untrusted data, and JSON is obviously used by basically everyone to handle untrusted data. So, these bullet points seem incorrect.

Later on, the readme says this:

When to use Flatbuffers / Bincode / CapN Proto

If you can safely compile all your data types into your application, all the buffers/data is trusted, and you don't intend to mutate buffers after they're created, Bincode/Flatbuffers/CapNProto is a better choice for you.

I can't speak for Faltbuffers or Bincode, but Cap'n Proto is explicitly designed for and heavily used in sandboxing scenarios involving unstruted data. In particular, Cap'n Proto was originally built to serve as the protocol that Sandstorm.io uses to communicate with untrusted apps running in secure sandboxes. It is believed to be secure for such use cases.

The statement also seems to imply that Cap'n Proto can only operate on schemas known at compile time. This is incorrect: Cap'n Proto features a mechanism for loading schemas dynamically and using them to operate on types with a reflection-like API. Incidentally, Protobuf features a very similar API. Practical experience has shown that these features are almost never used: people almost always know their schemas at compile time, and the benefits of compile-time type checking and optimizations are large. But dynamic schema loading can be useful in some niche scenarios.

Finally, regarding "mutation": Cap'n Proto actually supports mutation, with caveats. Fixed-width fields can be mutated in-place, but variable-width data will require new allocations which are always added to the end of the message. Removing an object will leave a zero'd "hole" in the data, which can only be reclaimed by making a full-tree copy. In principle, a Cap'n Proto implementation could use a complex memory allocation algorithm that is able to reuse holes in-place for later allocations of same or smaller size, but at present no implementation bothers to attempt this. This problem is generally fundamental to zero-copy formats, due to the need to allocate contiguous memory for each message. If NoProto has solved it somehow, it would be great to see some explanation of how; I couldn't immediately find it in the docs.

You cannot build with any version of no_proto, the library does not work. It throws more than 71 serious bugs in version 9.6.0.