Giter Club home page Giter Club logo

dissolve's Introduction

Dissolve

Parse and consume binary streams with a neat DSL.

Overview

Dissolve allows you to parse packed binary data into numbers, buffers, strings and more*! With a simple syntax inspired by node-binary and a solid, minimal implementation, you can be up and running in no time.

(* implementing "more" is left as an exercise to the reader)

If you want to produce binary data, might I suggest concentrate?

Features

  • Accurate handling of [u]int{8,16,32} numbers in both signed and unsigned variants using fast, built-in Buffer methods
  • Fast approximation of [u]int64 numbers in signed and unsigned variants
  • Extendable base class for building your own parsers and implementing custom types
  • Tiny (~250 LoC) implementation, allowing for easy debugging

Installation

Available via npm:

$ npm install dissolve

Or via git:

$ git clone git://github.com/deoxxa/dissolve.git node_modules/dissolve

Usage

Also see example.js, example-complex.js and example-loop.js.

#!/usr/bin/env node

var Dissolve = require("./index");

var parser = Dissolve().loop(function(end) {
  this.uint8("id").tap(function() {
    if (this.vars.id === 0x01) {
      this.uint16be("a").uint16be("b");
    } else if (this.vars.id === 0x02) {
      this.uint32be("x").uint32be("y");
    }
  }).tap(function() {
    this.push(this.vars);
    this.vars = {};
  });
});

parser.on("readable", function() {
  var e;
  while (e = parser.read()) {
    console.log(e);
  }
});

parser.write(new Buffer([0x01, 0x00, 0x02, 0x00, 0x03])); // {id: 1, a: 2, b: 3}
parser.write(new Buffer([0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x05])); // {id: 2, x: 4, y: 5}
parser.write(new Buffer([0x01]));
parser.write(new Buffer([0x00, 0x02, 0x00]));
parser.write(new Buffer([0x03])); // {id: 1, a: 2, b: 3}

Methods

All parser methods are chainable and return the same parser instance they were called on.

Tap

tap(name, callback)

This method allows you to "tap" into the parser at an arbitrary point. The callback will be called bound to the parser instance, so you can use parser methods on this. Any additional parser steps you introduce inside the callback will be executed before any existing steps that are already scheduled to run after the tap call.

If you provide a name parameter, all actions performed in the callback will be applied to a child object that will be put into a new property named after name. Note that in the callback, even if you provide a name parameter, you can still pretend you were in the outer "scope" because of some prototype trickery done with the vars object under the hood. You don't need to worry about that too much, the examples should make it a bit clearer.

Loop

loop(name, callback)

This method is like tap except that the callback is called over and over until signalled to stop. You do this by calling the end function that's provided as the first argument to your callback. When you call the end function, you can provide an optional truthy/non-truthy flag to tell Dissolve to ignore the result of the iteration of the loop where end was called. This is useful if you are reading until a null entry or similar.

If you provide a name parameter, a new array will be placed into a property named for that parameter, and after each iteration of the loop, any new values will be appended to the array as an object. As with the name stuff on tap, the examples will make that explanation a lot clearer.

The same semantics for job ordering and "scoping" apply as for tap.

Basic Parsing Methods

For each basic parsing method, the name value is the key under which the value will be attached to this.vars.

Buffer/String Methods

For these methods, the length parameter tells the parser how many bytes to pull out. If it's a string, it will be assumed that it is the name of a previously-set this.vars entry. If it's a number, it will be used as-is.

  • buffer(name, length) - binary slice
  • string(name, length) - utf8 string slice
  • skip(length) - skip length bytes

Numeric Methods

  • int8(name) - signed 8 bit integer
  • sint8(name) - signed 8 bit integer
  • uint8(name) - unsigned 8 bit integer
  • int16(name) - signed, little endian 16 bit integer
  • int16le(name) - signed, little endian 16 bit integer
  • int16be(name) - signed, big endian 16 bit integer
  • sint16(name) - signed, little endian 16 bit integer
  • sint16le(name) - signed, little endian 16 bit integer
  • sint16be(name) - signed, big endian 16 bit integer
  • uint16(name) - unsigned, little endian 16 bit integer
  • uint16le(name) - unsigned, little endian 16 bit integer
  • uint16be(name) - unsigned, big endian 16 bit integer
  • int32(name) - signed, little endian 32 bit integer
  • int32le(name) - signed, little endian 32 bit integer
  • int32be(name) - signed, big endian 32 bit integer
  • sint32(name) - signed, little endian 32 bit integer
  • sint32le(name) - signed, little endian 32 bit integer
  • sint32be(name) - signed, big endian 32 bit integer
  • uint32(name) - unsigned, little endian 32 bit integer
  • uint32le(name) - unsigned, little endian 32 bit integer
  • uint32be(name) - unsigned, big endian 32 bit integer
  • int64(name) - signed, little endian 64 bit integer
  • int64le(name) - signed, little endian 64 bit integer
  • int64be(name) - signed, big endian 64 bit integer
  • sint64(name) - signed, little endian 64 bit integer
  • sint64le(name) - signed, little endian 64 bit integer
  • sint64be(name) - signed, big endian 64 bit integer
  • uint64(name) - unsigned, little endian 64 bit integer
  • uint64le(name) - unsigned, little endian 64 bit integer
  • uint64be(name) - unsigned, big endian 64 bit integer
  • floatbe(data) - big endian 32 bit float
  • floatle(data) - little endian 32 bit float
  • doublebe(data) - big endian 64 bit double
  • doublele(data) - little endian 64 bit double

License

3-clause BSD. A copy is included with the source.

Contact

dissolve's People

Contributors

arthurschreiber avatar deoxxa avatar leetreveil avatar mweibel avatar rgbkrk avatar trueinsider avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dissolve's Issues

How does 'readable' event really work?

What happens when parsing the buffer fails? (ex. sudden cut off, random data being inserted)
How am I going to manage that?

What does this piece of code do?

parser.on("readable", function() {
  var e;
  while (e = parser.read()) {
    console.log(JSON.stringify(e, null, 2));
  }
});

I think that loop is used for checking buffer read completion, am I right?

Thanks

Loop not terminated by EOF

I have following structure:

Header
- frame 1
- frame 2
- frame ...N
[EOF]

The header does not contain number of frames, there's just EOF after the last frame. The problem is that the loop never finishes and I never get any data from parser. Any idea?

Peeking ahead at new values

The structure looks like this

- Track 0
  - Event 0
  - Event 1
- Track 1
  - Event 0
  - Event 1

etc...

After parsing the header for the track I start parsing the variable number of events using a loop. When I hit the start of the header for the next track I need to stop the loop. The problem I'm having is that if I parse the next 4 bits to look for the track header the offset gets moved and now I lose the data.

Since you wrote this you probably have encountered this before and I figured you would have some insight. My first thought was to write a peek function that is basically a .buffer() but doesn't increment the offset.

skip() ?

is there any method similar to node-binary.skip() ?

Update npm package

Hi,

the npm package seems to be quite outdated and nowadays reports a peerDependency error:

» npm install --save dissolve
npm WARN engine [email protected]: wanted: {"node":"~0.10.0"} (current: {"node":"5.3.0","npm":"3.3.12"})
npm WARN engine [email protected]: wanted: {"node":"~0.10.0"} (current: {"node":"5.3.0","npm":"3.3.12"})
[email protected] /path/to/code/node-etf
└─┬ [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  └── UNMET PEER DEPENDENCY stream-browserify@*

npm WARN EPEERINVALID [email protected] requires a peer of stream-browserify@* but none was installed.
npm WARN EPACKAGEJSON [email protected] No repository field.

Is there a possibility you could update it on npm and maybe also check if newer node version would be doable?

example-complex.js Question with mcstring16

Hi,

In the example-complex.js you defined mcstring16.

I was wondering why you are deleting [0x00, 0x02] from the buffer before reading [0x00, 0x61, 0x00, 0x62]? What was the purpose of buffering the hex values you were trying to extract?

Nested loops

I have been trying to get the parser work with nested loops but for some reasons, I'm getting TypeError: Cannot call method 'push' of undefined for this.vars[_job.name].push(this.vars.__loop_temp);

It will be helpful if the example-loop.js file is updated with additional nesting levels for loops.

Thanks!

Asynchronous .tap()

Implement...

parser.tap(function(done) {
  doSomethingAsync(function(err) {
    return done(err);
  });
});

...with all of the expected semantics.

Bump bl to 1.0.0

bl has a peer dependency of stream-browserify in older versions. Now that peer dependencies require being pinned themselves, this forces users of this library to have to install that themselves.

If you switch to the latest bl, this isn't the case. I have no idea if that affects your current code, considering it's a major bump in versions on bl.

.scan to search for a buffer

node-binary has a method .scan:

var b = binary()
    .scan('line', new Buffer('\r\n'))
    .tap(function (vars) {
        console.log(vars.line)
    });

I'll be happy to propose a PR to add this if @deoxxa thinks it's a good idea.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.