tabcorp / bison-types Goto Github PK

waiting for this issue to be resolved https://github.com/TabDigital/smart-buffer/issues/5

To avoid the run-time dependency on CoffeeScript, we should compile this to JS at build-time.
We can base it on the Makefile from hyperactive.

The current utf-8 type only supports reading known-length strings. This might be confusing for people who just want to read and write any string. It's also error prone since the naive approach of writing {str.length}{str} does not work for UTF strings, since str.length can actually be smaller than the number of required bytes.

Any ideas?

Variable length strings

We should make it very clear in the README that the current types only work for known-length strings.

We could also introduce a varchar type (or similar) that automatically encodes the length as well. It needs to handle multi-bytes strings, for example using

tmp = new Buffer(str, 'utf-8')  # figure out the actual byte count in the given encoding
@buffer.writeUint8(tmp.length)
@buffer.writeString(str)

or be more efficient by using the return value of writeString and then going back to write the length.

Because we need to reserve space for the length, we could

• always reserve 1 byte, and support strings up to 256 characters
• have a flexible varchar(len) where len is the maximum string length in bytes, so we simply reserve len/256 bytes to store the length

Encoding

Once we figure that out, the next question is: should bison-types mirror clever-buffer and expose a raw string, in addition to more sub-types like utf-8, ascii, etc...? What does that look like with fixed-length / variable-length? Maybe varchar(20, 'utf-8')?

Going back then, does it make sense to have a matching fixchar(20, 'utf-8'), which reads a utf-8 string of 20 bytes?

Reading a skip returns a value. Currently the offset of the buffer
Would be nice to not include an values that have been skipped in the resulting json object

And we could have other "ascii", "utf16", etc?