Giter Club home page Giter Club logo

engine.io-parser's Introduction

engine.io-parser

Build Status NPM version

This is the JavaScript parser for the engine.io protocol encoding, shared by both engine.io-client and engine.io.

How to use

Standalone

The parser can encode/decode packets, payloads, and payloads as binary with the following methods: encodePacket, decodePacket, encodePayload, decodePayload.

Example:

const parser = require("engine.io-parser");
const data = Buffer.from([ 1, 2, 3, 4 ]);

parser.encodePacket({ type: "message", data }, encoded => {
  const decodedData = parser.decodePacket(encoded); // decodedData === data
});

With browserify

Engine.IO Parser is a commonjs module, which means you can include it by using require on the browser and package using browserify:

  1. install the parser package

    npm install engine.io-parser
  2. write your app code

    const parser = require("engine.io-parser");
    
    const testBuffer = new Int8Array(10);
    for (let i = 0; i < testBuffer.length; i++) testBuffer[i] = i;
    
    const packets = [{ type: "message", data: testBuffer.buffer }, { type: "message", data: "hello" }];
    
    parser.encodePayload(packets, encoded => {
      parser.decodePayload(encoded,
        (packet, index, total) => {
          const isLast = index + 1 == total;
          if (!isLast) {
            const buffer = new Int8Array(packet.data); // testBuffer
          } else {
            const message = packet.data; // "hello"
          }
        });
    });
  3. build your app bundle

    $ browserify app.js > bundle.js
  4. include on your page

    <script src="/path/to/bundle.js"></script>

Features

  • Runs on browser and node.js seamlessly
  • Runs inside HTML5 WebWorker
  • Can encode and decode packets
    • Encodes from/to ArrayBuffer or Blob when in browser, and Buffer or ArrayBuffer in Node

API

Note: cb(type) means the type is a callback function that contains a parameter of type type when called.

Node

  • encodePacket

    • Encodes a packet.
    • Parameters
      • Object: the packet to encode, has type and data.
        • data: can be a String, Number, Buffer, ArrayBuffer
      • Boolean: binary support
      • Function: callback, returns the encoded packet (cb(String))
  • decodePacket

    • Decodes a packet. Data also available as an ArrayBuffer if requested.
    • Returns data as String or (Blob on browser, ArrayBuffer on Node)
    • Parameters
      • String | ArrayBuffer: the packet to decode, has type and data
      • String: optional, the binary type
  • encodePayload

    • Encodes multiple messages (payload).
    • If any contents are binary, they will be encoded as base64 strings. Base64 encoded strings are marked with a b before the length specifier
    • Parameters
      • Array: an array of packets
      • Function: callback, returns the encoded payload (cb(String))
  • decodePayload

    • Decodes data when a payload is maybe expected. Possible binary contents are decoded from their base64 representation.
    • Parameters
      • String: the payload
      • Function: callback, returns (cb(Object: packet, Number:packet index, Number:packet total))

Tests

Standalone tests can be run with npm test which will run the node.js tests.

Browser tests are run using zuul. (You must have zuul setup with a saucelabs account.)

You can run the tests locally using the following command:

npm run test:browser

Support

The support channels for engine.io-parser are the same as socket.io:

Development

To contribute patches, run tests or benchmarks, make sure to clone the repository:

git clone git://github.com/socketio/engine.io-parser.git

Then:

cd engine.io-parser
npm ci

See the Tests section above for how to run tests before submitting any patches.

License

MIT

engine.io-parser's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

engine.io-parser's Issues

Tag new release with #55 PR

In an effort to get this noticed a bit more, I am creating an issue to request the release of a new version to npm that includes the merged PR #55 so that Socket.io will work with React Native.

A few of us have been trying in vain to get an ETA on a new release but have not heard anything since April so this is a gentle effort to get a bit more love on this issue. Excuse my forwardness with this issue in advance! ๐Ÿ˜„

CC @rauchg

Unable to bundle socket.io client due to undeclared dependency on base64-arraybuffer

I'm attempting to bundle the socket.io client with webpack 5 and yarn 2. I believe these have more strict dependency checks which would explain why this issue hasn't been opened for webpack 4 and/or npm users.

Webpack is giving the error

ERROR in ../../../.yarn/cache/engine.io-parser-npm-4.0.1-6bdb879e8a-3b71ef8b5a.zip/node_modules/engine.io-parser/lib/decodePacket.browser.js 7:18-47
Module not found: Error: engine.io-parser tried to access base64-arraybuffer, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Required package: base64-arraybuffer (via "base64-arraybuffer")
Required by: engine.io-parser@npm:4.0.1 (via /home/alex/GitProjects/tools/webpack-hot-module-replacement/.yarn/cache/engine.io-parser-npm-4.0.1-6bdb879e8a-3b71ef8b5a.zip/node_modules/engine.io-parser/lib/)
 @ ../../../.yarn/cache/engine.io-parser-npm-4.0.1-6bdb879e8a-3b71ef8b5a.zip/node_modules/engine.io-parser/lib/index.js 2:21-46
 @ ../../../.yarn/cache/engine.io-client-npm-4.0.2-9adb255344-af3880585b.zip/node_modules/engine.io-client/lib/index.js 14:24-51
 @ ../../../.yarn/cache/socket.io-client-npm-3.0.3-512b232ec1-8bb891554f.zip/node_modules/socket.io-client/build/manager.js 4:12-39
 @ ../../../.yarn/cache/socket.io-client-npm-3.0.3-512b232ec1-8bb891554f.zip/node_modules/socket.io-client/build/index.js 5:18-38 69:16-36
 @ ../../@skoville/webpack-hmr/client/universal/default-socket-io-accessor~/build/socket-io-accessor.js 13:27-54
 @ ../../@skoville/webpack-hmr/client/universal/default-socket-io-accessor~/build/package.js 13:13-44
 @ ../../@skoville/webpack-hmr/client/web/default~/build/entry.js 13:66-142

seems like this is accurate since it's being required here

base64decoder = require("base64-arraybuffer");

is there any reason why this isn't included in dependencies and is instead only in devDependencies?

"base64-arraybuffer": "0.1.5",

In case there's a native component to this library which causes it to fail compiling for some users, consider following the approach that was eventually taken here websockets/ws#1220, where we would move base64-arraybuffer from devDependencies to optionalDependencies.

I can open a PR if this approach (move to optional dependencies) works for you.

Typescript build error cjs

I just stumbled upon a weird error:
I am currently working on a typescript project which is using socket.io in its backend. (nodejs)

When trying to build it, I got this error:

node_modules/engine.io/build/engine.io.d.ts:3:25 - error TS7016: Could not find a declaration file for module 'engine.io-parser'. 'C:/Users/Lenovo/Code/wehub_development/wehub/node_modules/engine.io-parser/build/cjs/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/engine.io-parser` if it exists or add a new declaration (.d.ts) file containing `declare module 'engine.io-parser';`

3 import * as parser from "engine.io-parser";
                          ~~~~~~~~~~~~~~~~~~

node_modules/engine.io/build/transport.d.ts:4:24 - error TS7016: Could not find a declaration file for module 'engine.io-parser'. 'C:/Users/Lenovo/Code/wehub_development/wehub/node_modules/engine.io-parser/build/cjs/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/engine.io-parser` if it exists or add a new declaration (.d.ts) file containing `declare module 'engine.io-parser';`

4 import { Packet } from "engine.io-parser";
                         ~~~~~~~~~~~~~~~~~~


Found 2 errors in 2 files.

Errors  Files
     1  node_modules/engine.io/build/engine.io.d.ts:3
     1  node_modules/engine.io/build/transport.d.ts:4

When i tried copying the .d.ts files in your package from the esm folder into the cjs folder, it suddenly worked without complaint.

Could it be possible that you forgot to add the .d.ts files to the cjs export of your package or are my tsconfig configurations incorrect in some way?

Thank you! :)

CONTRIBUTING.md

We should come up with some baseline contributing guidelines for this repo (and all the other socket.io related repos) So that when people create issues or PRs they get some guidance on what to do and where to post issues.

This issue can be a starting discussion on what to include.

License?

Can you add a license to this please?

Unable to bundle socket.io

My bundle throw this error :
Uncaught Error: Cannot find module "engine.io-parser/keys" from "engine.io-parser"

I use Brunch.io and Babel.

Some extract from my bundle :

require.register('engine.io-parser', function(exports,req,module){
      var require = __makeRequire((function(n) { return req(n.replace('./', 'engine.io-parser//')); }), {}, 'engine.io-parser');
      (function(exports,require,module) {
        /**
 * Module dependencies.
 */

var keys = require('./keys');

if I replace here : require('./keys') by require('./lib/keys') that's work !

Or I need to replace this register by require.register('engine.io-parser/keys',...

require.register('engine.io-parser/lib/keys', function(exports,req,module){
      var require = __makeRequire((req), {}, 'engine.io-parser');
      (function(exports,require,module) {

/**
 * Gets the keys for an object.
 *
 * @return {Array} keys
 * @api private
 */

module.exports = Object.keys || function keys (obj){
  var arr = [];
  var has = Object.prototype.hasOwnProperty;

  for (var i in obj) {
    if (has.call(obj, i)) {
      arr.push(i);
    }
  }
  return arr;
};

      })(exports,require,module);
    });

How I can avoid this problem ?

Parser broken on Android 3.*

Hello!

I've encountered a pretty weird issue for Android Honeycomb.

The problem comes from the fact that there is specific handling for "android" devices as per this article.

It assumes that "android" can handle sending things as ArrayBuffer in here however this only works on android >= 4.

The check that is done for "Android" here works for 2.* because those versions doesn't identify as "Android" but the 3.* versions does.
The problem then arises here because Uint8Array isn't available until Android 4.

I'm currently testing on a Samsung Galaxy Tab 7.7 on Android 3.2 and will try to find some way to either fix this or work around it.

Publish latest commits via npm.

There have been a couple of commits and one in particular is quiet important - so i'd like to ask if you could publish a new version on npm soon?

Parsing ArrayBuffer

If the packet that should be decoded isn't a string (in decodePacket), the parser runs into the following problem in browsers:

Excerpt from exports.decodePacket:

var asArray = new Uint8Array(data);
var type = asArray[0];
var rest = sliceBuffer(data, 1);
...
return { type: packetslist[type], data: rest };

The problem is that 'type' will contain an useless number that refers to nothing contained in packetlist[type]. Making the returned object look like:
{ type: undefined, data: ArrayBuffer }... which makes everything that follows fail.
This only happens when using XHR.

exports.decodePacket is called from exports.decodePayloadAsBinary.

component.json

I would submit a PR, but none of the dependencies seem to support component. Is component support planned for engine.io?

2.0.1 has broken multi-byte characters

It appears since 2.0.1, multi-byte characters are breaking in browser clients.

In practice, we have an application that happened to send a JSON message containing a no-break space (U+00A0) somewhere, and the browser client received the message with the final } cut-off.

With a normal install of [email protected], zuul tests also fail there. When the engine.io-parser dep is then replaced with 2.0.0 (by simply swapping the directory in node_modules), tests succeed again.

All tests seem to work fine in Node.js.

(macOS 10.12.3, Chrome 56.0.2924.87. Same results in Firefox 52.0.)

Cannot find name 'TransformStream' after updating to v5.2.0

Since I updated socket.io from the v4.7.1 to the v4.7.2 (and so engine.io-parser from the v5.1.0 to the v5.2.1), I'm getting the following errors while building my projects:

node_modules/engine.io-parser/build/cjs/index.d.ts:6:54 - error TS2304: Cannot find name 'TransformStream'.
6 export declare function createPacketEncoderStream(): TransformStream<Packet, any>;
                                                       ~~~~~~~~~~~~~~~
node_modules/engine.io-parser/build/cjs/index.d.ts:7:96 - error TS2304: Cannot find name 'TransformStream'.
7 export declare function createPacketDecoderStream(maxPayload: number, binaryType: BinaryType): TransformStream<Uint8Array, any>;
                                                                                                 ~~~~~~~~~~~~~~~

Found 2 errors in the same file, starting at: node_modules/engine.io-parser/build/cjs/index.d.ts:6

Funnily, I'm only getting this error when compiling using tsc but not when compiling using esbuild.

npx tsc -> error
npx esbuild src/index.ts --platform=node -> ok

Here is my tsconfig.json:

{
  "compileOnSave": false,
  "compilerOptions": {
    "baseUrl": "./",
    "outDir": "./dist",
    "strict": true,
    "sourceMap": true,
    "declaration": false,
    "downlevelIteration": true,
    "experimentalDecorators": true,
    "moduleResolution": "node16",
    "importHelpers": true,
    "target": "es2022",
    "module": "node16",
    "lib": [
      "es2022"
    ]
  },
  "include": [
    "src/**/*.ts"
  ]
}

Reading nodejs's documentation, when using TransformStream, it looks like they are importing it with

import { TransformStream } from 'node:stream/web';

which is not the case for engine.io-parser, so maybe that's the problem.

Test omission

        it('should encode/decode empty payloads', function () {
          encPayload([], function(data) {
            decPayload(data,
              function (packet, index, total) {
                expect(packet.type).to.eql('open');
                var isLast = index + 1 == total;
                expect(isLast).to.eql(true);
              });
          });
        });

This test did not pass because it did not execute the assertion

UTF-8 stopped working

utf-8 strings worked just fine before. Now they are broken.
Characters that are represented as multiple bytes now get split into individual characters.

Allow sending binary data without ArrayBuffer (e. g. IE8)

I am working with binary data in the browser and โ€“ as far as I can see โ€“ I can receive binary data which comes as an object { base64: true, data: base64Data }.

BUT there is no way to do the same to send binary data. Sending binary data strictly requires an ArrayBuffer which is not available in older browsers.

I propose to introduce the same concept for sending binary data as for receiving binary data.
Sending an object { base64: true, data: base64Data } should send the data with binary flag to the wire.
This would allow older browsers to work binary. No need to modify server code nor client code. Only the
engine.io-parser needs a small modification.

I am working on a patch, if interested I will provide a pull request.

Badly handle ArrayBuffer type

This issue is originated from nwjs/nw.js#5034. When using Socket.io in latest NW.js, binary data cannot be handles properly.

The root cause is that engine.io-parser doesn't handle ArrayBuffer properly. See lib/index.js:L148-L157.

  if (binaryType === 'arraybuffer') { // <-- [1]
    var type = data[0]; // <-- [2]
    var intArray = new Uint8Array(data.length - 1);
    for (var i = 1; i < data.length; i++) {
      intArray[i - 1] = data[i];
    }
    return { type: packetslist[type], data: intArray.buffer };
  }
  var type = data[0]; // <-- [3] 
  return { type: packetslist[type], data: data.slice(1) };
  1. In Node.js or NW.js, binaryType is always null. I assume it's a bug as well.
  2. ArrayBuffer cannot be directly read using array styled syntax data[0]. One should construct a TypedArray form the buffer, such as Uint8Array, and read the data inside.
  3. I guess here it assumes data could only be a Node Buffer. However, in NW.js data can be of type ArrayBuffer from browser builtin WebSocket.

error TS2304: Cannot find name 'Blob'

Describe the bug
An error is thrown while running tsc in my typescript project using engine.io-parser on version v5.0.3.

Run cd backend && tsc
  cd backend && tsc
  shell: /usr/bin/bash -e {0}
Error: ../node_modules/engine.io-parser/build/esm/commons.d.ts(7,81): error TS2304: Cannot find name 'Blob'.
Error: Process completed with exit code 2.

To Reproduce

Please fill the following code example:

not applicable

engine.io-parser version: v5.0.3

Expected behavior
tsc runs without any issues as it does on version v5.0.0

Platform:

  • Software: tsc
  • OS: ubuntu

Additional context
The logs of the workflows can be found here https://github.com/jojomatik/blockcluster/runs/6233313889 (v5.0.0) and here https://github.com/jojomatik/blockcluster/runs/6233323368 (v5.0.3). The respective commits are linked from within the workflows.

It might be possible that the problem has to be solved somewhere else in the socket.io dependency chain but the error message lead me here. Here is the link to the commit that fails jojomatik/blockcluster@5cecdce. It shows all upgraded dependencies of that might cause the issue.

Expose synchronous "encodePacket" function

Currently, the encodePacket function always returns the result in its callback argument. I understand that's likely done to prevent blocking the thread in some situations.

I'd like to propose exporting a new function that'd be Packet -> string.

function encodePacketInline(
  packet: Packet,
  supportsBinary: boolean
): string

The name is illustrational. I'm open to suggestions.

I'd like to use such a function in my custom message parser:

client.on('message', (event) => {
  server.send(
    encodePacketInline({
      type: 'message',
      data: encoder.encode({ type: 2, data: ['message', 'payload'], nsp: '/' })
    })
  )
})

This is intentionally simplified example of my setup. In reality, I have a transport.parser instance that automatically wraps outgoing events to the server in transport.parser.encode(data) before providing the payload to the MessageEvent. This is also code that the consumer of my library is intended to write (their own parser).

How?

The logic for encoding is already synchronous:

supportsBinary ? data : "b" + toBuffer(data, true).toString("base64")

return callback(PACKET_TYPES[type] + (data || ""));

I simply propose moving it to a new standalone public function.

Blob vs global.Blob

I'm a little confused at line 106 of lib/browser.js:

  if (global.ArrayBuffer && data instanceof ArrayBuffer) {
    return encodeArrayBuffer(packet, supportsBinary, callback);
**  } else if (Blob && data instanceof global.Blob) { **
    return encodeBlob(packet, supportsBinary, callback);
  }

The Blob comes as a local variable from require('blob') while global.Blob is not. Is that on purpose?

Release new version

engine.io-parser includes various fixes since the last release 1.1.0...master (I need the PhantomJS fixes) Is there a chance to get an updated version published to npm?

Original data mutated during encoding?

Apologies if this is the wrong repo for this.

I have noticed that when I pass binary data to socket.io's emit function, that data is transformed in place. For example, File objects become ArrayBuffer objects. It makes sense why this needs to happen, but would it be possible to generate a modified copy of the data for transmission instead of mutating it in place?

As it is now, I find myself defensively deep cloning anything I pass to socket.io which is actually kind of hard to do well (JSON encoding and parsing leads to bugs, structured clone would be great but isn't exposed by browsers, circular references create issues, etc).

Thanks for the great software, btw.

utf8encode

A few questions:

  • why are there so many public methods? Shouldn't encodePayload and decodePayload be enough?

Currently, it seems that:
- encodePayload/decodePayload is used for polling transport
- encodePacket/decodePacket is used for websocket transport

Quoting socketio/engine.io-client#140 (comment):

encodePacket is more efficient in this case so we can leverage the WS framing

Is that really more efficient? Shouldn't we rather concatenate two packets instead of sending them one by one? (I think that's why, currently, sending a buffer with socket.io results in two frames: socketio/socket.io#2444)

  • according to the following comment, I understand that utf8 is needed for mixing binary/string content

The reason we are using utf8.js is that when encoding mixed binary and string contents, we need to support multibyte strings. It is easiest to encode the multibyte characters so that we can do a transformation to a binary representation one byte at a time.

Yet it seems that, since encodePayload always call encodePacket with utf8encode to true, the strings transmitted over polling transport are UTF8-encoded twice (socketio/engine.io#315):

https://github.com/socketio/engine.io-parser/blob/1.3.2/lib/index.js#L229

exports.encodePayload = function (packets, supportsBinary, callback) {
  ...
  if (supportsBinary) {
    return exports.encodePayloadAsBinary(packets, callback);
    // which call encodePacket with utf8encode = true => OK
  }
  ...
  function encodeOne(packet, doneCallback) {
    exports.encodePacket(packet, supportsBinary, true, function(message) {
      doneCallback(null, setLengthHeader(message));
    });
    // since the payload is not encoded as binary, shouldn't utf8encode be set to false here?
  }
}

cc @calzoneman @rase- @rauchg

esm types break TypeScript compilation with lib check enabled.

/node_modules/engine.io-parser/build/esm/index.d.ts (3,57): Relative import paths need explicit file extensions in EcmaScript imports when '--moduleResolution' is 'node12' or 'nodenext'. Did you mean './commons.js'?

Extensions ARE REQUIRED in esm type modules.

Version 5.2.1 not available

Hi, im getting an error when trying to install the version 5.2.1.

image

Looks like that version is not published in the npm repository yet. The latest published is 5.2.0.

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.