Pointer._validate is too eager and too strict about capnp-ts HOT 3 CLOSED

Any struct received on the wire -- regardless of its section sizes -- is a valid match for any defined struct schema. If a section is larger than the reader expected, the reader should ignore trailing words/pointers. If the section is shorter than expected, the reader should treat the missing words/pointers as if they were zero/null. (Note that because data values are XOR'd with their defaults on read/write, and a null pointer is treated equivalent to its default value, treating the missing data as zero/null is equivalent to treating it as the default value.)

This is necessary to support the evolution properties described here: https://capnproto.org/language.html#evolving-your-protocol

Specifically, if a struct was constructed using an older version of the protocol, it may be missing newly-added fields, making the sections shorter than expected. If it was constructed using a newer version, it may have set new fields that the reader doesn't know about, making the sections larger than expected.

Of course, for zero-copy operation, you don't want to have to resize structs that came off the wire smaller than expected. So, on the reader side, the getter methods for each field will probably need to check if the field offset is out-of-range for the underlying section, and return the default value if so.

Also note that it's important to preserve data you don't understand when copying a struct from a Reader into a Builder. An intermediate server that processes and forwards messages should not silently drop data it doesn't understand.

To that end, copying data from a Reader to a Builder should be implemented in a schema-agnostic fashion. It's possible to perform a deep copy using only the metadata available in the pointers as encoded on the wire.

This leads to a possible problem: What if you copy a struct from a Reader to a Builder, but the struct is smaller than its schema expects, and then the application attempts to access and modify that struct, modifying fields that are out-of-bounds? Since the copying is done schema-agnostic, you can't guarantee that the struct will be big enough.

To solve this, in the C++ code, when the application tries to get a struct builder, and the underlying pointer indicates the struct is smaller than expected, the code makes a new properly-sized shallow copy of the struct, and updates the pointer to point at that. (This leaves a "hole" in the message of bytes which waste space but carry no data, which is unfortunate, but at least the hole is zero'd, so it should pack well.) Note that this copy should never reduce the size of the struct, only increase, because, again, you don't want to discard trailing data you don't understand.

I guess I should integrate this into the capnp docs...

from capnp-ts.

This is a great find, and definitely one of the rough patches of the implementation (precisely because of the ambiguity in the spec).

I'm working on Pointer test coverage right now and will give this a very careful look; chances are I just need to relax the check.

I'm less concerned right now about the eagerness because premature optimization is evil, etc.. I am still keeping it in mind since you are right that it is one of the design goals to be as frugal as possible while still staying safe.

from capnp-ts.

@kentonv thanks, having this in the docs would be v helpful to implementers in other languages.

from capnp-ts.