Adding new variable length field in a repeating group about fix-simple-binary-encoding HOT 5 CLOSED

This was previously discussed in real-logic/simple-binary-encoding#109

A proposal was made to the SBE working group but it was not accepted. The gist of the argument was that adding more metadata to an SBE header would have a cost for every message while the benefit would only accrue occasionally.

Without making a change to SBE, there are strategies for dealing with some cases. A new var-length field at the root of a message does not break compatibility if you use a framing protocol to delimit messages. As you pointed out elsewhere, it is still possible to add a new var-length field to a repeating group the by deploying an updated decoder before using the new encoder.

The proposed change would have been a breaking change for version 1.0 release candidates. Nevertheless, I will keep this issue open so the working group can reconsider for a future version of SBE if users support it.

from fix-simple-binary-encoding.

@adkapur, total message length was not the motivation for this change. Rather, it is just a count of repeating groups and variable length data so new groups and variable field can be added in message schema extension. It allows a decoder to walk the blocks (message root, repeating groups, variable-length) even if it doesn't have the latest version of the schema.

This doesn't prevent you from using a framing header for total length of a message or batch of messages. For mixed encodings, SOFH is useful. Other users may wish to use a simple size prefix or delimiters.

from fix-simple-binary-encoding.

Note: this enhancement changes wire format and therefore is a breaking change with version 1.0.

from fix-simple-binary-encoding.

Isn't this all about trying to determine the total message length? If so then do we really need to add two new fields to the header (4 bytes) and group size encoding (4 bytes) to arrive at the total message length? Couldn't we just add message length to the SBE header itself?

Martin also seems to be alluding to this here in a discussion about SOFH so this will kill two birds with one stone --> real-logic/simple-binary-encoding#482

from fix-simple-binary-encoding.

It is a dangerous practice to not to check the version number of the schema and still try to use the old decoder at the same time. It may have unwanted business level consequence undetected at the technical level. For safety purpose, I prefer to always use the corresponding version of decoder.

from fix-simple-binary-encoding.