keichi / binary-parser Goto Github PK

I am using binary-parser to write a parser for a tree-like file format: it is composed of "nodes" which are followed by children. Those children themselves are nodes.

My parser definition boils down to:

var Parser = require('binary-parser').Parser;

var nodeParser = new Parser();

// (Some fields definition here)

nodeParser.array('nestedList', {
    type: nodeParser,
    readUntil: function(item, buffer) {
        // (An actual ending condition here)
        return true;
    }
});

console.log(nodeParser.getCode());

But executing this code leads to a recursion error:

pwd>node crash.js
pwd\node_modules\binary-parser\lib\context.js:87
    var params = Array.prototype.slice.call(arguments, 1);
                                       ^

RangeError: Maximum call stack size exceeded
    at Context.interpolate (pwd\node_modules\binary-parser\lib\context.js:87:40)
    at Context.pushCode (pwd\node_modules\binary-parser\lib\context.js:61:38)
    at Parser.generateArray (pwd\node_modules\binary-parser\lib\binary_parser.js:474:13)
    at Parser.generate (pwd\node_modules\binary-parser\lib\binary_parser.js:295:37)
    at Parser.generateNext (pwd\node_modules\binary-parser\lib\binary_parser.js:334:25)
    at Parser.generate (pwd\node_modules\binary-parser\lib\binary_parser.js:304:17)
    at Parser.generateArray (pwd\node_modules\binary-parser\lib\binary_parser.js:491:14)
    at Parser.generate (pwd\node_modules\binary-parser\lib\binary_parser.js:295:37)
    at Parser.generateNext (pwd\node_modules\binary-parser\lib\binary_parser.js:334:25)
    at Parser.generate (pwd\node_modules\binary-parser\lib\binary_parser.js:304:17)

Is binary-parse able to express such recursive parser definitions? If so, how should I define it?

Regards.

Hey Keichi

I was wondering if it's somehow possible to use a variable from a parent scope from inside a choice type item. If it's not implemented I would greatly appreciate a pointer on how to do so.
My use-case is parsing of a DNSRecord:

var TXTRecord = new Parser()
    .string('txt' { length: 'rdlength' }) // rdlength is undefined

var answer = new Parser()
    .nest('name', { type: fqdn_pretty })
    .uint16be('type')
    .uint16be('class')
    .uint32be('ttl')
    .uint16be('rdlength')
    .choice('record', {
        tag: 'type',
        choices: {
            0x01: ARecord,
            0x02: PTRRecord, 0x05: PTRRecord, 0x0c: PTRRecord,
            0x10: TXTRecord,
            0x21: SRVRecord,
            0x1C: AAAARecord
        },
        defaultChoice: new Parser().buffer('rdata', { length: 'rdlength' }) //rdlength is undefined
    })

I enjoy working with your module, thanks for the help in advance!

You can't use formatters with bitfields other than the last one in a sequence, since the bitfield reading code generation gets deferred until the last one, but the formatter code is inserted after each field in a sequence. Thus all the other bit field values get overwritten when the actual reading and bit shifting code is executed, because the formatters were called earlier.

EDIT:

It's been almost five years, but I thought I'd add a concrete example. 😄 I'm using [email protected] for these.

Here's a simple parser that parses uint8 values, and uses formatter for those (although these formatters just return a constant string).

new Parser()
    .uint8('a', { formatter: () => 'first' })
    .uint8('b', { formatter: () => 'second' })
    .uint8('c', { formatter: () => 'third' });

The generated code is fine. First we read the value to the vars and then we immediately replace that value with the formatted version.

vars.a = dataView.getUint8(offset, false);
offset += 1;
vars.a = (() => 'first').call(this, vars.a);

vars.b = dataView.getUint8(offset, false);
offset += 1;
vars.b = (() => 'second').call(this, vars.b);

vars.c = dataView.getUint8(offset, false);
offset += 1;
vars.c = (() => 'third').call(this, vars.b);

Here's the same thing, except we're parsing bitfields. We're still using formatters.

new Parser()
    .bit8('a', { formatter: () => 'first' })
    .bit8('b', { formatter: () => 'second' })
    .bit8('c', { formatter: () => 'third' });

Now there's a bug in the generated code:

vars.a = (() => 'first').call(this, vars.a);  // Formatter called with vars.a being undefined
vars.b = (() => 'second').call(this, vars.b); // Same with vars.b
var $tmp1 = dataView.getUint16(offset);
var $tmp2 = dataView.getUint8(offset + 2);
var $tmp0 = ($tmp1 << 8) | $tmp2;
offset += 3;
vars.a = $tmp0 >> 16 & 255; // The values are finally parsed
vars.b = $tmp0 >> 8 & 255;
vars.c = $tmp0 >> 0 & 255;
vars.c = (() => 'third').call(this, vars.c); // But only the last one is properly formatted

When parsing multiple successive bitfields, the code that populates the varibles is "deferred" to the end. However, the code that calls the formatters is not deferred. Thus the formatters get called too early, when the values don't exist yet.

The "deferring" feature here causes the issue. Or rather the lack of deferring for the formatters. The bitfield parsing code is only generated when there are no more bitfields in the succession. But the code that calls the formatters is generated every time.

It would be wonderful if this could be expanded to handle streams. Piping a stream into a parser and then having a callback when it is complete would be perfect.

When implementing an assertion, e.g. crc32 that's defined as being against the raw buffer, it would be useful to be able to access the raw buffer to implement the check.

Currently you have to do the check afterwards, which is annoying when you're parsing arrays of sub-types.

Consider the following example:

var Parser = require('binary-parser').Parser;

var packet = new Parser()
    .endianess('big')
    .uint8('cmd')
    .buffer('data',{
        readUntil:function(item) {
            return item === 2;
        }
    });

var buf = Buffer.from("AABB2C","hex");
packet.parse(buf);

That code results in the following error:

/home/bagger/src/binary-parser/node_modules/binary-parser/lib/context.js:125
      s = s.replace(match, params[index].toString());
                                         ^

TypeError: Cannot read property 'toString' of undefined
    at /home/bagger/src/binary-parser/node_modules/binary-parser/lib/context.js:125:42
    at Array.forEach (<anonymous>)
    at Context.interpolate (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/context.js:123:13)
    at Context.pushCode (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/context.js:66:36)
    at Parser.generateBuffer (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/binary_parser.js:588:9)
    at Parser.generate (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/binary_parser.js:407:33)
    at Parser.generateNext (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/binary_parser.js:452:21)
    at Parser.generate (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/binary_parser.js:416:15)
    at Parser.generateNext (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/binary_parser.js:452:21)
    at Parser.generate (/home/bagger/src/binary-parser/node_modules/binary-parser/lib/binary_parser.js:416:15)

Is there problem with function or I'm doing something wrong?

Thanks anyway.

Hi,
i have some design issues using binary-parser in combination with typescript interfaces.
My goal is to define a protocol header as an interface. This is my first approach:

/* Protocol Header*/
interface IHeader {
  command: Buffer;       // UINT
  length: Buffer;        // UINT
  sessionHandle: Buffer; // UDINT
  status: Buffer;        // UDINT
  senderContext: Buffer; // Array of 8 octets
  option: Buffer;        // UDINT
}

export class Protocol {
  private header: IHeader;
  private data: Buffer; // Array of max.65511 octets

  constructor() {
    this.header = <IHeader>{};
    this.header.command = new Buffer(2);
    this.header.length = new Buffer(2);
    this.header.sessionHandle = new Buffer(4);
    this.header.status = new Buffer(4);
    this.header.senderContext = new Buffer(8);
    this.header.option = new Buffer(4);
  }

  public setHeader() {
    // for now some default values
    this.header.command.writeUInt16LE(1, 0);
    this.header.length.writeUInt16LE(2, 0);
    this.header.sessionHandle.writeUInt32LE(3, 0);
    this.header.status.writeUInt32LE(4, 0);
    this.header.senderContext.writeUInt16LE(5, 0);
    this.header.option.writeUInt16LE(6, 0);
  }
 
  public getHeader() {
    const ret: Buffer = Buffer.concat(
      [
        this.header.command, this.header.length, this.header.sessionHandle,
        this.header.status, this.header.senderContext, this.header.option],
      24, 
    );
    return ret;
  }
}

How could I use the advantages of the library for my application?

Is it possible to .skip() bits instead of bytes? What I'm doing now is .bitX('skip') and it seems inconvenient.

Hi,

is there a way a parser can get access to the current, absolute value of offset in the input Buffer? I need it to express a readUntil condition.

On a related note: if the provided function for readUntil returns false when first called, does array return an empty array? Or is at least one array entry read before the provided function is called?

Regards.

It looks like with 3d7cb57 you switched from using a plain Function() constructor to using the Node "vm" module for evaluation. Since "vm" is not available in browser environments (such as Electron or plain browser-ified code) it will no longer work there.

There are vm polyfills for browsers like this, but not sure how those fit into your bigger plans. Is browser environment something you care about?

Thanks!

This could make code a bit cleaner: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals :-)

I have this example:

var payload = Parser.start()
        .uint8('data_type')
        .uint8('data_length')
        .array('data', {
          type: 'uint8',
          length: 'data_length',
          formatter: function(array){
            // ACCESS 'data_type' HERE!!
            return array
          }
        })

I want to be able to access data_type inside formatter of array.

I inspected this, but it wasn't clear how to get to this value.
The scope is available in `length'.

Is this possible? If yes, how?

Have you considered adding the generation of an assemble method (or whatever you want to call it) where you pass in an object and the parser converts it to a buffer?

require('binary-parser').Parser.start().bit32('a').getCode()

returns

var vars = {};
var offset = 0;
if (!Buffer.isBuffer(buffer)) {
throw new Error("argument buffer is not a Buffer object");
}
var $tmp0 = buffer.readUInt32BE(offset);
offset += 4;
vars.a = $tmp0 >> 0 & 0;
return vars;

Please note the & 0, which will always lead to the result 0.

From your example in the Readme (on the namely('alias') section)

If "self" reference is replaced in the parser it throws an error on compiling
=> Uncaught TypeError: Cannot set property 'resolved' of undefined

Example:
// the line below registers the name 'self', so we will be able to use it in // twoCells` as a reference
var parser = Parser.start().namely("self");

var stop = Parser.start().namely("stop");

var twoCells = Parser.start()
.namely("twoCells")
.nest("left", { type: "self" })
.nest("right", { type: "stop" });

parser.uint8("type").choice("data", {
tag: "type",
choices: {
0: "stop", // !!! replaced "self" with "stop"
1: "stop",
2: "twoCells"
}
});

var buffer = Buffer.from([2, /* left / 1, 1, 0, / right */ 0]);

parser.parse(buffer);`

I noticed in the API documentation that "name should consist only of alphanumeric characters and start with an alphabet." It looks like this is because the compiled code uses dot notation to access properties:

Context.prototype.generateVariable = function(name) {
    var arr = [];

    Array.prototype.push.apply(arr, this.scopes[this.scopes.length - 1]);
    if (name) {
        arr.push(name);
    }

    return arr.join('.');
};

Would there be any harm in using array bracket notation instead, so that any valid property name can be used as a name? It's not very elegant, but something like:

Context.prototype.generateVariable = function(name) {
    var arr = [];

    Array.prototype.push.apply(arr, this.scopes[this.scopes.length - 1]);
    if (name) {
        arr.push(name);
    }
    
    return arr[0] + arr.slice(1).map(function(x) { return '["' + x + '"]'; }).join('');
};

It's be amazing if we could get some TypeScript definitions. I can make them if needed.

Hi,
The Binary Parser is great and works flawless, however when I run istanbull over mocha to test my parsing I get a wierd errors.

My Parsers are inside an Object like this:

function aParser(client) {
    var self=this;
    var BinParser = require('binary-parser').Parser;
     self.eventReportParser =  new BinParser()
        .endianess('big')
        .uint32('UpdateTime')
        .uint32('TimeofFix')
        .int32('Latitude', {formatter: function(x) { return x /10000000;}})
        .int32('Longitude', {formatter: function(x) { return x /10000000;}});

 self.calampHeaderParser = new BinParser()
        .endianess('big')
        .uint8('OptionsByte')
        .uint8('ServiceType')
        .uint8('MessageType')
        .uint16('Sequence')
        .choice('data', {
            tag: 'MessageType',
            choices: {      
                2: self.eventReportParser,
                 }
        });   
}
// Usage is as follows:
aParser.prototype.parse = function (message) {
    var msg;
    try {
        msg = this.calampHeaderParser.parse(message);
    } catch (e) {
 }

This works fine, and test works fine when running mocha only.

However, when I am running istanbul

./node_modules/.bin/istanbul cover ./node_modules/.bin/_mocha -- --recursive -R xunit-file -r should

I got execptions only on parsers with function inside,
Any idea why ?

Thanks in advance

I have a 64bit integer I need to parse. How am I going to do this?

var buf = new Buffer('00000b1a2a5585c3', 'hex'); // big endian

Thanks for the great app, but one thing I am wondering. Is it possible to prevent choice from adding a key to the object? Once I get a few choices deep, referencing a value requires something akin to data.a.data.b.data.c. It would be great if there were a way to keep the data structure flatter.

I'm parsing a bitfield-encoded date, bits 15-12=month, 11-7=day, 6-0=(year-2000). The twist is the bytes are in little-endian.
Looking at the code and tests for bitfields, it looks like the algorithm is

read bytes in big-endian order
retrieve bitfields starting from the LSB and working toward the MSB

The LE encoding for 01/07/2017 (US order) is 0x1391, but reading BE (0x9113) puts the LSB of the day at the topmost bit 15, and the MSBs of the day at bits 3-0.

How can I properly parse this date using BIT directives?

I am trying to use Parser in Angular 5 (using @types/binary-parser).
In order to test the module first I use the IP header example.
when doing that I get the following error

ERROR ReferenceError: Buffer is not defined
at Parser.eval [as compiled] (eval at Parser.compile (webpack-internal:///../../../../binary-parser/lib/binary_parser.js), :3:1)
at Parser.parse (webpack-internal:///../../../../binary-parser/lib/binary_parser.js:337)
at AppComponent.parse (webpack-internal:///../../../../../src/app/app.component.ts:47)
at AppComponent.ngOnInit (webpack-internal:///../../../../../src/app/app.component.ts:21)
at checkAndUpdateDirectiveInline (webpack-internal:///../../../core/esm5/core.js:12596)
at checkAndUpdateNodeInline (webpack-internal:///../../../core/esm5/core.js:14123)
at checkAndUpdateNode (webpack-internal:///../../../core/esm5/core.js:14066)
at debugCheckAndUpdateNode (webpack-internal:///../../../core/esm5/core.js:14959)
at debugCheckDirectivesFn (webpack-internal:///../../../core/esm5/core.js:14900)
at Object.eval [as updateDirectives] (ng:///AppModule/AppComponent_Host.ngfactory.js:8)

If I call Buffer,isBuffer(buf) on my code before calling .parse() it works just fine. I removed this piece of code from binary_parser.js in function getCode and the generated code works fine.
It seems that I am missing something in the declaration of Buffer. It is included in my app.Module. so I can not put the finger on what am I missing?
This is my code as I used it:

`import { Parser } from 'binary-parser';
import { Component } from '@angular/core';
import { OnInit } from '@angular/core/src/metadata/lifecycle_hooks';
import { Buffer } from 'buffer';

@component({
selector: 'app-root',
templateUrl: './app.component.html',
styleUrls: ['./app.component.css']
})
export class AppComponent implements OnInit{

title = 'app';

ngOnInit(): void {
this.parse()
}
parse(){

let ipHeader = new Parser()
.endianess('big')
.bit4('version')
.bit4('headerLength')
.uint8('tos')
.uint16('packetLength')
.uint16('id')
.bit3('offset')
.bit13('fragOffset')
.uint8('ttl')
.uint8('protocol')
.uint16('checksum')
.array('src', {
    type: 'uint8',
    length: 4
})
.array('dst', {
    type: 'uint8',
    length: 4
});

const buf = new Buffer('450002c5939900002c06ef98adc24f6c850186d1', 'hex');
Buffer.isBuffer(buf);

// Parse buffer and show result
console.log(ipHeader.parse(buf));
}

}`

I am currently parsing a binary file and I have to make a decision based on a byte value.

If the value is 0x16, I want to start parsing an array that consumes the 0x16 at the start of every array element (which itself has some more fields) until no more 0x16 follow.

If the value is 0x15, I want to skip parsing the array completely and consume the 0x15 with the next parser function.

The 0x15 value is guaranteed to be present, but the array is optional. I want the 0x15 value to be saved to the parser with a dedicated key

Currently, I think I have to consume either 0x16 or 0x15 and make the decision based on that, and as a consequence my array-parser has to take care of the fact that I already consumed 0x16 for the object in the following array, making it hard to split up the parsing logic overall.

A read-ahead-choice that does not require to consume data would make this scenario quite easy. Do you have any recommendations on how to deal with this with the current tools aswell?

Edit: Another option might be calling the readUntil-function once before starting to parse an array for the first time instead of after the first element.

Hi!

What about AOS SPACE DATA LINK PROTOCOL ?

   const header = new Parser()
        .endianess('big')
        .bit2('version')
        .bit8('spaceCraftId')
        .bit6('vcid')
        .bit24('vcfc');

    header.parse(...);

/home/user/project/node_modules/binary-parser/lib/binary_parser.js:503
      throw new Error(
      ^

Error: Currently, bit field sequence longer than 4-bytes is not supported.
    at Parser.generateBit (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:503:13)
    at Parser.generate (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:407:33)
    at Parser.generateNext (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:452:21)
    at Parser.generate (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:416:15)
    at Parser.generateNext (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:452:21)
    at Parser.generate (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:416:15)
    at Parser.generateNext (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:452:21)
    at Parser.generate (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:416:15)
    at Parser.generateNext (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:452:21)
    at Parser.generate (/home/user/project/node_modules/binary-parser/lib/binary_parser.js:416:15)

Hi. I'was taking a look at your package and wrote the following code for parsing some binary data structure:

this.TFormula = new Parser().endianess('big').array('sp_macros', {
  type: 'uint16',
  length: 10
}).array('sp_medios', {
  type: 'uint16',
  length: 10
}).array('sp_premix', {
  type: 'uint16',
  length: 10
}).array('sp_otros', {
  type: 'uint16',
  length: 4
}).uint16('sp_premix_manual').uint16('sp_aceite').uint16('sp_excipiente').bit1('con_macros').bit1('con_medios').bit1('con_premix').bit1('con_premix_manual').bit1('con_aceite').bit1('con_otros').bit10('otras_banderas').array('id_macros', {
  type: 'uint16',
  length: 10
}).array('id_medios', {
  type: 'uint16',
  length: 10
}).array('id_premix', {
  type: 'uint16',
  length: 10
}).array('id_otros', {
  type: 'uint16',
  length: 4
}).int16('id').int16('peso_batch').int16('tiempo_mezclado').int16('cant_batchs_premix').string('nombre', {
  length: 20,
  stripNull: true
}).array('fecha', {
  type: 'uint16',
  length: 8
});

I'm getting the error Specified primitive type "uint16" is not supported.. I fixed this replacing the type with uint16be. Is this correct if I'm using endianess('big') at the begining?

Thanks!

float & double are NOT supported but listed in the documentation - they break the system when used.

Thanks for the library, really helpful!

I've noticed that parsing rest of message into a buffer, it comes back one too short.

var Parser = require('binary-parser').Parser;
var bp = new Parser().buffer('test',{ readUntil: 'eof'});
var buf=new Buffer([1,2,3]);

console.log(bp.parse(buf));

output:

{ test: <Buffer 01 02> }

I am not an expert but I think that there should not be - 1 on this line :

ctx.pushCode('{0} = buffer.slice(offset, buffer.length - 1);',

.bit2('Sensor', { formatter: function (Sensor) { console.log(Sensor) } });

console.log is empty

Is there a way to reverse the decoding process? I mean, to get the original buffer back from the parsed data?

Thanks

line 440 of binary_parser.js

Parser.prototype.generateBuffer = function(ctx) {
    if (this.options.readUntil === 'eof') {
        ctx.pushCode('{0} = buffer.slice(offset, buffer.length - 1);',

maybe need buffer.slice(offset, buffer.length);',

Greetings!
It is probably silly by me but I really couldn't find a solution for that...
I recently downloaded npm and when I tried to use 'npm install' on webstorm it first told me that 'binary-parser' is missing.
I checked online and found out that I should use 'npm install binary-parser' and so I did.
Now when I try to run 'npm install' it says that the binary-parser isn't located in the npm registery.
I couldn't find a solution for it...is there any way that you can help me?

Hello! Explain how to use variables and functions that were previously announced.
Example:
function test() {
return 1;
}
var foo = 5;
var myData = new Parser()
.string('type', {length: 1, formatter: function(arr) {
console.log(foo);
console.log(test());
return arr;
}});
var buf = new Buffer('2D035958501', 'hex');
console.log(myData.parse(buf));

RefferenceError: foo is not defined
RefferenceError: test is not defined

Background

... this is due to them taking up two indices in the Constant Pool:

All 8-byte constants take up two entries in the constant_pool table of the class file. If a CONSTANT_Long_info or CONSTANT_Double_info structure is the item in the constant_pool table at index n, then the next usable item in the pool is located at index n+2. The constant_pool index n+1 must be valid but is considered unusable.

In retrospect, making 8-byte constants take two constant pool entries was a poor choice.

• Java 6 spec
• Java 8 spec

I know this isn't a repo focused on JVM class file parsing, but thought it might spark a discussion or feature that allows for better interoperability non-standard binary formats like this.

Reproducing

$ javap -v <path to class file with at least one long or double> can verify my claims if you compare the output from the constant pool to that of the example Parser instance. The only problem with that, is there will be an error thrown before you can grab the contents from the parser.

Here's what I did to fix it and it works across Java 6/7/8:

export const ClassFileParser =
  Parser.start()
    .endianess('big')
    .uint32('magic', { assert: JVM_CLASS_FILE_MAGIC_NUMBER })
    .uint16('minor_version')
    .uint16('major_version')
    .uint16('constant_pool_count')
    .array('constant_pool', {
        type: ConstantPoolInfo,
        length: function () {
          let previousIndex = this.constant_pool.length - 1;
          let previousEntry = this.constant_pool[previousIndex];
          if (previousEntry && (previousEntry.tag === 6 || previousEntry.tag === 5)) {
            this.constant_pool[previousIndex - 1] = this.constant_pool[previousIndex];
            this.constant_pool_count -= 1;
            this.constant_pool.push(false);
          }

          return this.constant_pool_count - 1;
        }
    })
    .uint16('access_flags')
    .uint16('this_class')
    .uint16('super_class')
    .uint16('interface_count')
    .array('interfaces', {
      type: InterfaceInfo,
      length: function () {
        return this.interface_count - 1;
      }
    })
    .uint16('field_count')
    .array('fields', {
      type: ClassMemberInfo,
      length: function () {
        return this.field_count;
      }
    })
    .uint16('method_count')
    .array('methods', {
      type: ClassMemberInfo,
      length: function () {
        return this.method_count;
      }
    })
    .uint16('attribute_count')
    .array('attributes', {
      type: AttributeInfo,
      length: function () {
        return this.attribute_count;
      }
    });

While this does work, it's not very pretty having to do the following if the previous entry was a Long or Double:

• move the previous entry back one slot in the array
• subtract from the pool count
• push a value that somewhat resembles a sentinel value (if I see an entry === false I know there was an incorrect index used)

But at least it makes it work 😉

Thanks for releasing this library open source -- really made my life a lot easier and it has great performance 😄

Output from running the above code.

1102 'NameAndType'
1103 'Fieldref'
1104 'Long'
1106 'Long'
1108 'Long'
1110 'NameAndType'
1111 'Fieldref'

Here's some validation for this by grepping javap's output:

Based on the discussion in #27, skip() should be renamed to a more neutral name to imply that it can move the current pointer not only forward but also backward.

Hello,

I use this module to read a buffer from TCP socket. It's possible to get the size of parsed buffer?

Thanks!

It would be great to be able to use a parents tags in a nested parser.

For example lets say you have a parser such as:

var ParentParser = Parser.start()
.uint32('version')
.nest('Child', {type: ChildParser})

var ChildParser = Parser.start()
.choice('basedOnParentTag', {
tag: 'version',
choices: {
1: ...
}
})

Hiya Keichi,

Thanks a lot for the library! Really sweet. I've just noticed that since 1.1.3 the zero-terminated string stuff is broken.

Code:

var Parser = require('binary-parser').Parser;
var StringOptions = {length: 99, zeroTerminated:true};

// Build an IP packet header Parser
var ipHeader = new Parser()
    .skip(1, StringOptions)
    .string("command", StringOptions)
    .string("username", StringOptions);

// Prepare buffer to parse.
var buf = new Buffer('0e757365726e616d6500526f6200', 'hex');

// Parse buffer and show result
console.log(ipHeader.parse(buf));

Output on 1.1.2:

$ npm install; node test.js
[email protected] node_modules\binary-parser
{ command: 'username', username: 'Rob' }

Output on 1.1.3+

$ npm install; node test.js
[email protected] node_modules\binary-parser
{ command: 'username\u0000Rob\u0000', username: '' }

Don't have the time to look (yet) but maybe I can check it out lateron.

If i try to delete an object with this formatter, it does not work.

Something like:

new Parser()
    .int16('a')
    .int16('b')
    .int16('c')
    .tap(function(ctx) { console.log(ctx) })
    .int16('d');

to get:

{ a: ..., b: ..., c: ... }

Inspired by https://github.com/substack/node-binary#btapcb

Hi,
Would it be possible to add a way to set the array key from parsed data? A bit like:

.array('lumps', {
    type : Parser.start()
        .int32le('filepos')
        .int32le('size')
        .string('name', { length: 8, encoding : 'ascii'}),
    length : 'numlumps'
    key : 'name', <-- Would be nice
});

User defined types does not seem very straight forward. After mucking around with it a bit, it appears that I need to fork the repo and rewrite a large portion of the code, include the "SPECIAL_TYPES" to add a VLQ type.

Is this true, or am I missing something?

var Parser = require("binary-parser").Parser;


var test = Parser.start()
    .string("head", {encoding:"ascii", length: 2})
    .buffer("something", {
        encoding:"ascii",
        readUntil: function(item, buffer){
            return item === 2;
        }
    });
    
var testBuffer = Buffer.from([0,1,2,3,4]);
console.log(test.parse(testBuffer))

This gives an error of toString with error

TypeError: Cannot read property 'toString' of undefined

Also while we are at this, a request to have readUntil choice for .string()?

Hello,

As a result of #7, it was possible to define the key of an array based on a variable but based on my reading of the code key can only be a string as such a function is not supported in this case.

My goal is ultimately to provide a custom key for based on the position in the array. For example position 0 would be "ExportTable", 1 would be "ImportTable", 2 would be "ResourceTable" and so forth. I imagined I could achieve this if it was possible to define key as a function which knew how many times it was called (thus knowing the index in the array), so it could return a suitable string/name of the element. I suspect in general it would be useful having the key option of a array accept a function would also cover more uses.

My first question is my goal already achievable?

Second question would it be useful if the key option of an array could be a function (like how assert can be a function or a number)?

I found some other parsers have uint24 type, why not in binary-parser?

First let me say that this parser is great and easy to use - thanks!

However, given a defined function:

    function range (min, max) {
        return function(x) {
        return min <= x && x <= max;
        };
    }

I would like to create a parser like this to use my 'range' function.

var dataParser = new Parser()
    .endianess('little')
    .uint8('inside_humidity', {assert: range(0, 100)})
    .uint16('barometer', {assert: range(20000, 32500),
            formatter: function(x) {return x/1000;}})
    .uint16('wind_direction', {assert: range(0, 360)})
    .uint16('crc');

But this does not work -- and according to: http://stackoverflow.com/questions/37597883/javascript-using-non-inline-function the reason is because you 'stringify the function'.

Because of that, if we want to use named functions for asserts and formatters we have to use a work-around like this:

function range (min, max) {
    return new Function("x", "return " + JSON.stringify(min) +
            " <= x && x <=  " + JSON.stringify(max)+ ";");
}

Is this something you'd be interested in changing so the use of named 'assert' and 'formatter' functions will be more straightforward?

Thanks,
Kevin Ryan

Hello binary-parser Team,

thanks for binary-parser. I found it last week and it is the simplest and best binary parser i've seen.

I have a question or feature request. Is it possible to use a parsed value only for computing in a next section ob a binary block?

Parser.start()
  .string('length', {encoding: 'ascii', length: 2, formatter: str2int})
  .skip(23)
  .array('position', {
    type: 'int32be',
    length: "length"
  });

Here u can see i want to parse the length to find the length for my array. But i dont want the value 'length' in my parsed output.

Is this possible?

Hello Team,

can u help me please. How can i use moment in a formatter?

const Parser = require("binary-parser").Parser;
const moment = require('moment');

let time = function(timestamp) {
	return moment(timestamp, 'YYMMDDHHmmssSS').format('YYYY-MM-DD HH:mm:ss.SS');
};

let Telegram = new Parser()
	.string('timestamp', {encoding: 'hex', length: 7, formatter: time});

The Exception is:

evalmachine.<anonymous>:9
        return moment(timestamp, 'YYMMDDHHmmssSS').format('YYYY-MM-DD HH:mm:ss.SS');
        ^

ReferenceError: moment is not defined
    at Parser.<anonymous> (evalmachine.<anonymous>:9:2)
...

I think the Problem is that Parser dont know moment. But how can i realize that?

It seems the .bitN functions don't always respect options passed to them. However, .nest-ing them in a new parser does cause them to respect options, unless the nest-ed .bitN functions align to a byte.

Additionally, .nest-ed parsers consume up to the next byte boundary, even when only parsing .bitN functions.

Examples:
https://repl.it/Klzf/20