serde-rs / bytes Goto Github PK

Cow is handled in this way in serde. But serde-bytes deserializes Cow by borrowing. As a result, this compiles:

#[derive(Serialize, Deserialize)]
struct A<'a> {
    a: Cow<'a, [u8]>,
}

But this does not compile:

#[derive(Serialize, Deserialize)]
struct A<'a> {
    #[serde(with = "serde_bytes")]
    a: Cow<'a, [u8]>,
}

The error message generated by compiler:

error: lifetime may not live long enough
 --> src/main.rs:4:21
  |
4 | #[derive(Serialize, Deserialize)]
  |                     ^^^^^^^^^^^
  |                     |
  |                     lifetime `'de` defined here
  |                     associated function was supposed to return data with lifetime `'a` but it is returning data with lifetime `'de`
5 | struct A<'a> {
  |          -- lifetime `'a` defined here
  |
  = help: consider adding the following bound: `'de: 'a`
  = note: this error originates in the derive macro `Deserialize` (in Nightly builds, run with -Z macro-backtrace for more info)

Isn't it more reasonable to handle Cow in a way similar to serde so that deserializing Cow does not borrow?

This doesn't look critical, but it might be nice to fix the tests to do the right thing here, if for no purpose than to allow testing this configuration to ensure the crate works properly.

    Updating crates.io index
  Downloaded bincode v1.2.1
   Compiling proc-macro2 v1.0.8
   Compiling serde v1.0.104
   Compiling unicode-xid v0.2.0
   Compiling byteorder v1.3.4
   Compiling syn v1.0.14
   Compiling quote v1.0.2
   Compiling serde_derive v1.0.104
   Compiling bincode v1.2.1
   Compiling serde_test v1.0.104
   Compiling serde_bytes v0.11.3 (/home/kent/.cpanm/work/1582251421.24569/serde_bytes-0.11.3)
error[E0432]: unresolved import `serde_bytes::ByteBuf`
 --> tests/test_partialeq.rs:1:26
  |
1 | use serde_bytes::{Bytes, ByteBuf};
  |                          ^^^^^^^ no `ByteBuf` in the root

error[E0432]: unresolved import `serde_bytes::ByteBuf`
 --> tests/test_derive.rs:1:19
  |
1 | use serde_bytes::{ByteBuf, Bytes};
  |                   ^^^^^^^ no `ByteBuf` in the root

error[E0432]: unresolved import `serde_bytes::ByteBuf`
 --> tests/test_serde.rs:1:19
  |
1 | use serde_bytes::{ByteBuf, Bytes};
  |                   ^^^^^^^ no `ByteBuf` in the root

error[E0425]: cannot find function `deserialize` in crate `serde_bytes`
 --> tests/test_derive.rs:5:21
  |
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
  |                     ^^^^^^^^^^^ help: a function with a similar name exists: `serialize`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: could not compile `serde_bytes`.
warning: build failed, waiting for other jobs to finish...
error: could not compile `serde_bytes`.
warning: build failed, waiting for other jobs to finish...
error[E0277]: the trait bound `serde_bytes::bytes::Bytes: std::clone::Clone` is not satisfied
  --> tests/test_derive.rs:23:5
   |
23 |     cow_bytes: Cow<'a, Bytes>,
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `serde_bytes::bytes::Bytes`
   |
   = note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
   = note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
   = note: only the last field of a struct may have a dynamically sized type

error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
  --> tests/test_derive.rs:23:5
   |
23 |     cow_bytes: Cow<'a, Bytes>,
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
   |
   = help: within `serde_bytes::bytes::Bytes`, the trait `std::marker::Sized` is not implemented for `[u8]`
   = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
   = note: required because it appears within the type `serde_bytes::bytes::Bytes`
   = note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
   = note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
   = note: only the last field of a struct may have a dynamically sized type

error[E0277]: the trait bound `serde_bytes::bytes::Bytes: std::clone::Clone` is not satisfied
 --> tests/test_derive.rs:5:21
  |
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
  |                     ^^^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `serde_bytes::bytes::Bytes`
  |
  = note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
  = note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
  = note: only the last field of a struct may have a dynamically sized type

error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
 --> tests/test_derive.rs:5:21
  |
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
  |                     ^^^^^^^^^^^ doesn't have a size known at compile-time
  |
  = help: within `serde_bytes::bytes::Bytes`, the trait `std::marker::Sized` is not implemented for `[u8]`
  = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
  = note: required because it appears within the type `serde_bytes::bytes::Bytes`
  = note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
  = note: required because it appears within the type `std::borrow::Cow<'a, serde_bytes::bytes::Bytes>`
  = note: only the last field of a struct may have a dynamically sized type

error[E0277]: the trait bound `serde_bytes::bytes::Bytes: std::clone::Clone` is not satisfied
 --> tests/test_derive.rs:5:10
  |
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
  |          ^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `serde_bytes::bytes::Bytes`
  |
  = note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
  = note: required by `std::borrow::Cow`

error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
 --> tests/test_derive.rs:5:10
  |
5 | #[derive(Serialize, Deserialize, PartialEq, Debug)]
  |          ^^^^^^^^^ doesn't have a size known at compile-time
  |
  = help: within `serde_bytes::bytes::Bytes`, the trait `std::marker::Sized` is not implemented for `[u8]`
  = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait>
  = note: required because it appears within the type `serde_bytes::bytes::Bytes`
  = note: required because of the requirements on the impl of `std::borrow::ToOwned` for `serde_bytes::bytes::Bytes`
  = note: required by `std::borrow::Cow`

error: aborting due to 8 previous errors

Some errors have detailed explanations: E0277, E0425, E0432.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `serde_bytes`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Some of the conversions are hard to discover from rustdoc.

let v: Vec<u8> = bytebuf.into();

This would be more discoverable as:

let v = bytebuf.into_vec();

Hi!

I stumbled around some confusing behaviour with regards to no_std builds:

use serde::{Deserialize, Serialize};

#[derive(Deserialize, Serialize)]
struct Testing<'a> {
    #[serde(with = "serde_bytes")]
    testing: &'a [u8],
}

Compiles if the std or alloc feature is enabled, but doesn't compile if it isn't.
I do not understand why the deserialize function is behind such a feature flag.

I understand that serde_bytes only changes the serialization of &[u8] since serde's implementation is already optimized, so you can get around the issue with

#[derive(Deserialize, Serialize)]
struct Testing<'a> {
    #[serde(serialize_with = "serde_bytes::serialize")]
    testing: &'a [u8],
}

But:

1. That is counter intuitive, since the doc's main example uses #[serde(with = "serde_bytes")]
2. It is more verbose
3. I do not understand why this should be gated behind the requirement of an allocator.
4. The error confusing since rustc can't understand the problem (though recent version do correctly point out the feature flag).

Can this feature-gate be removed?

Thanks

I'm relatively new to Rust so bear with me. I'm working on a project that provides a Python API to Rust code via Pyo3. In my project I'm trying to enable fast serialization and deserialization using bincode. I read this crate can mitigate the performance issues I'm seeing (bincode takes 3 times longer than serialization/deserialization with serde_json).

It's not clear to me how to take advantage of this crate when I don't use a struct field to hold the data.

For reference, this is what the unedited, actual serialization and deserialization code snippet looks like, with the Pyo3 API intact:

pub fn to_bincode(&self) -> PyResult<Vec<u8>> {
    Ok(serialize(&self).unwrap())
}
#[classmethod]
pub fn from_bincode(_cls: &PyType, encoded: Vec<u8>) -> PyResult<Self> {
    Ok(deserialize(&encoded).unwrap())
}

With the API removed, I suppose would be the analogous code:

pub fn to_bincode(&self) -> Vec<u8> {
    serialize(&self).unwrap()
}
pub fn from_bincode(encoded: Vec<u8>) -> Self {
    deserialize(&encoded).unwrap()
}

See also: PyO3/pyo3#100

How can I utilize this crate in this case?

Hi,
i have a message to send in the form of:

type PubKey = Vec<u8>;

#[derive(Serialize, Deserialize, Debug)]
#[serde(tag="kind")]
pub enum Envelope<T> {
    Greeting {
        #[serde(with="serde_bytes")]
        id: PubKey,
        #[serde(with="serde_bytes")]
        shared: PubKey,
        thin: bool
    },
    AllKnown { 
        all_known: Vec<PubKey>
    },
}

As you can see, PubKey is only a Vec<u8>, what is the most idiomatic way to pack this. I do not want to put the Vec<u8> in a struct to use serde_bytes.
Is there something i am missing?
It would be really neat to annotated all_known just with #[serde(with="serde_bytes")].
It seems a pretty common problem to be to have an array of byte arrays.

serde supports Arc with the 'rc' feature. Is it possible to have support for Arc<Vec< u8>>?

For example, I hava a strcut defined as follow:

pub struct VariableLength(u32);

And the field using a variable length encoding scheme, how can I implement Serializer/Deserializer for it?

VaribaleLength <---> Vec<u8>/&[u8]

Thanks a lot.

While this crate helps serialize bytes in binary formats efficiently, human-readable formats will still have them as vectors of integers. Is efficient serialization in human-readable formats something potentially fitting for this crate, or should it be implemented elsewhere?

That may include:

• support for hex/base64 encoding
• multibase support
• selection of the specific representation for the encoding (e.g. prefix or no prefix, or lowercase or uppercase for hex)

The README has this line:

Rust support for specialization is being tracked in rust-lang/rust#31844. Once it lands in the stable compiler Serde will be deprecating these wrapper types in favor of optimizing &[u8] and Vec<u8> out of the box.

Serialization in Go has taken a similar tack, always serialising []uint8 as []bytes, but this has significant knock-on effects which can be quite painful.

As an example, serializing []int8 (&[i8]) to JSON results in an array of numbers, while []uint8 (&[u8]) results in a base64 blob (credit @edef1c for the example).

Taking this choice away from the user, by having specialization rather than explicit wrapper types, may make it difficult to impossible to comply with schemas that originate somewhere other than the Rust project using Serde - in Go, the solution is an expensive promotion to 16-bit integers, which costs a copy.

f8c4812
f810459
#3

They could coexists with ByteBuf::from and ByteBuf::into_vec if you think these functions are clearer in documentation. I never had problems with looking for From trait implementations.

Now is not possible to use it with generic functions which uses things like T: From<Vec<u8>>. There is reason why there are traits like From trait in rust.

And solution to problems like this #14 require even more extra glue code as before.

It would be much appreciated.

…or perhaps a SmallByteBuf<N> wrapper?

I'm trying to make some structures serialize into bytes in a nice way. However, these types contain both optional data and maps of bytes (Option<Vec<u8>> and BTreeMap<usize, Vec<u8>>). #[serde(with = "serde_bytes")] doesn't seem capable of dealing with these types. I've found a way to make them work by changing the types to Option<ByteBuf> and BTreeMap<usize, ByteBuf>, but it adds quite a lot of extra glue code. Is there a nicer way to achieve this?

Currently Travis will only run cargo test with default features.

drop-in replacement solution is in https://crates.io/crates/serde_bytes_ng
I hope it can be merged into this, instead of maintaining another crate.

These traits are not intended to be implemented outside of this crate.

Hi,

I'm having a hard time figuring out how exactly #[serde(with = "serde_bytes")] on a &[u8] provides any efficiency gains for deserialization. From what I can see, this causes serde_bytes::deserialize() to be called, which forwards to serde_bytes::Deserialize for &[u8], which just calls serde::Deserialize::deserialize(). How is this any different from not using serde_bytes at all? Am I missing something, or is this crate simply useless for optimizing deserialization of &[u8]?

Sorry, I am still a new Rust user. I realize it's a wrapper, but I can't seem to convert pieces to an Vec!!!

pieces: b"J\u{a0}\u{c3}\u{fb}\u{ce}q&\u{8c}N\u{fb}V\u{fe}L\u{b1}\u{f4}\"&\u{bc}YY&\u{c5}\u{f5}\u{90}\u{f8}\u{8d}\u{c5}`\u{90}]-,\u{1d}J\u{82}\u{db}9O\u{ae}\u{98}\u{c4}Sa\u{a1}\u{85}\u{8c}7\u{cf}\u{df}w\u{bb}qlH\u{fa}6\u{8f}6\u{5}\u{af}MB\u{89}\u{c7}i\u{94}\u{ee}\u{95}\u{b0}0+v\u{ca}\r\u{f2}\u{a3}Q\u{a1}\n\u{fc}\u{84}\u{ea}\u{c8}-?8>l\u{1b}\u{b9}\u{d5}\u{a0}\u{c1}\u{8b}\\\u{db}\u{c1}\u{b7})\u{af}\u{db}&_\u{87}\u{a7}\u{f6}\u{4}y\u{16}\u{c3}\u{2}\u{98}G\u{9c}\u{ae}\u{3}\u{c9}\u{dc}\u{ec}\u{cb}\u{fa}(W\u{f4}\u{fb}\u{eb}M>\u{9d}}\u{84}~K\u{94}\u{c6}\u{b4}\u{18}\u{f4}\u{fa}\u{83}"

This is my current self.info.pieces

That would be useful while writing tests, to compare a Vec<u8> or a bytestring literal to a Bytes or a ByteBuf without trying to wrap things up.

Per common convention, Bytes should implement ToOwned that converts them to ByteBuf.

In particular, this would allow easily using Cow<Bytes> in contexts where you want either borrowed or owned data.

When I try to use:

#[derive(Serialize, Deserialize, Debug)]
struct Test<'a>
{
	#[serde(with = "serde_bytes")]
	a: &'a [u8]
}

I am getting compile errors:

135 | #[derive(Serialize, Deserialize, Debug)]
    |                     ^^^^^^^^^^^ the trait `std::convert::From<std::vec::Vec<u8>>` is not implemented for `&[u8]`
    |
    = help: the following implementations were found:
              <&'a [u8] as std::convert::From<serde_bytes::Bytes<'a>>>
    = note: required by `serde_bytes::deserialize`

When I try:

#[derive(Serialize, Deserialize, Debug)]
struct Test<'a>
{
	a: serde_bytes::Bytes<'a>
}

Then:

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'de` due to conflicting requirements
   --> main.rs:135:21
    |
135 | #[derive(Serialize, Deserialize, Debug)]

It would be more efficient to support deserialisation into &[u8] to avoid copies similarly as I can deserialise into &str. Or I am missing something?

Currently Cow<'a, [u8]> uses &'a [u8] deserialization implementation which would fail when deserializing from any non-binary format.
While this is expected for &'a [u8] but Cow<'a, [u8]> can contain Vec<u8> as well and doesn't have to be a pointer into deserializer owned data.

I suggest to write a special visitor for Cow<'a, [u8]> that will create Cow::Borrowed when visit_borrowed_bytes or visit_borrowed_str is called, and Cow::Owned for visit_str, visit_string, visit_bytes or visit_byte_buf.

Hello! Thanks for this great crate! I tried to add Vec<Vec<u8>> under efficient bytes conversion but was unable to. Probably expected, would it be possible to add such support?

More detailed errors below:

...
error[E0277]: the trait bound std::vec::Vec<std::vec::Vec<u8>>: std::convert::AsRef<[u8]> is not satisfied
--> src/lib.rs:169:10
|
169 | #[derive(Serialize, Deserialize, Debug, Clone)]
| ^^^^^^^^^ the trait std::convert::AsRef<[u8]> is not implemented for std::vec::Vec<std::vec::Vec<u8>>
|
= help: the following implementations were found:
<std::vec::Vec as std::convert::AsRef<[T]>>
<std::vec::Vec as std::convert::AsRef<std::vec::Vec>>
= note: required by serde_bytes::serialize

error[E0277]: the trait bound std::vec::Vec<std::vec::Vec<u8>>: std::convert::From<std::vec::Vec<u8>> is not satisfied
--> src/lib.rs:169:21
|
169 | #[derive(Serialize, Deserialize, Debug, Clone)]
| ^^^^^^^^^^^ the trait std::convert::From<std::vec::Vec<u8>> is not implemented for std::vec::Vec<std::vec::Vec<u8>>
|
= help: the following implementations were found:
<std::vec::Vec as std::convert::From<serde_bytes::ByteBuf>>
<std::vec::Vec as std::convert::From<std::collections::BinaryHeap>>
<std::vec::Vec as std::convert::From<std::boxed::Box<[T]>>>
<std::vec::Vec as std::convert::From<&'a str>>
and 6 others
= note: required by serde_bytes::deserialize
...

Now that const_generics are stable, I wonder if it is possible to implement support for [u8; N] arrays (serde seems to support them)?