dudleycarr / snappystream Goto Github PK

JavaScript library for supporting Snappy framing format

License: MIT License

JavaScript 0.77% TypeScript 98.73% Shell 0.50%

snappystream's Introduction

snappystream

A Node.js library for supporting the Snappy framing format via streams. See the Snappy Framing Format Description for details.

Usage: SnappyStream and UnsnappyStream are Transform streams.

const {SnappyStream} = require('snappystream')
const fs = require('fs')

const inStream = fs.createReadStream('snappy.txt')
const snappyStream = new SnappyStream()
const outStream = fs.createWriteStream('snappy_frame.txt')

inStream.pipe(snappyStream).pipe(outStream)

UnsnappyStream constructor takes an optional argument verifyChecksums which is false by default.

const {UnsnappyStream} = require('snappysteam')
const fs = require('fs')

const inStream = fs.createReadStream('snappy_frame.txt')
const unsnappyStream = new UnsnappyStream(true)

unsnappyStream.on('end', function () {
  console.log(unsnappyStream.read())
})

inStream.pipe(unsnappyStream)

snappystream's People

Contributors

Stargazers

Watchers

Forkers

movableink korbin lavab samhogg tsjing jiripospisil grimen ts-project-fork luminati-io fossabot

snappystream's Issues

Can you please upgrade the snappy version to >= 5.0.5

Hi..can you please update snappy to latest version? I get the security warning in one of our Nodejs App audit process.

Thanks

An in-range update of snappy is breaking the build 🚨

The dependency snappy was updated from `6.0.4` to `6.1.0`.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

snappy is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

Commits

The new version differs by 12 commits.

e5034d5 6.1.0
e974c8f Merge pull request #160 from kesla/prebuild-release
1870a4b this seem more correct
d7ae42a let's see if this works
51e7000 run "npm ci" in travis & appveyor
7697656 run npm audit and fix
ca1cf06 empty
1a62395 Merge pull request #159 from mbroadst/prebuild-support
1451b7e chore(lock): update package lock file
44900c3 chore(appveyor): update configuration to provide prebuilt binaries
53f8765 chore(travis): update travis to provide prebuild on linux/osx
3c8bae1 feat(prebuild): add support for pre-built binaries

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Use Buffer#readUIntLE instead of external dep

Buffer#readUIntLE has been available since v0.11.15 (many years before node v12) so using the native function will not decrease support surface.

Under the hood it uses an internal readUInt24LE implementation..

Compatibility with nodejs 6

Hello Guys,
Could you please update "require" property of package.json to use the latest version of sse4_crc32 version? Current one isn't compatible with nodejs v6, "npm install" fails with

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1

I asked maintainers of sse4_crc32 repo and they tell that it works fine for them. So i believe it is old version issue.

Could not locate the bindings file

I have found this bug only on linux. This build buildpack-deps:jessie https://github.com/nodejs/docker-node/blob/master/4.2/Dockerfile
NODE_VERSION 4.2.1

Error: Could not locate the bindings file. Tried:
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/build/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/build/Debug/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/build/Release/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/out/Debug/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/Debug/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/out/Release/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/Release/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/build/default/binding.node
→ /node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/compiled/4.2.1/linux/x64/binding.node
at bindings (/node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/node_modules/bindings/bindings.js:88:9)
at Object. (/node_modules/nsqjs/node_modules/snappystream/node_modules/snappy/snappy.js:2:34)
at Module._compile (module.js:435:26)
at Object.Module._extensions..js (module.js:442:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:311:12)
at Function._load (/usr/local/lib/node_modules/pm2/node_modules/pmx/lib/transaction.js:62:21)
at Module.require (module.js:366:17)
at require (module.js:385:17)
at Object. (/node_modules/nsqjs/node_modules/snappystream/lib/snappystreams.js:9:10)

Question please?

I have a requirement that reads as follow:

/**
 * All streams should start with the "Stream identifier", containing chunk
 * type 0xff, a length field of 0x6, and 'sNaPpY' in ASCII.
 */

This is how DEV team is saving Json Objects in a Redis DB using Snappy Compression, my question is, how can I add the same Stream Identifier with this library? Thank you so much

const fs = require('fs')
const JSONStream = require('JSONStream')
const { SnappyStream } = require('snappystream')

const snappyStream = new SnappyStream()
const outStream = fs.createWriteStream('./compressed.sz')

// const inStream = fs.createReadStream('./works.txt')
const inStream = fs.createReadStream('./fails.txt')

inStream
    .pipe(snappyStream)
    .pipe(JSONStream.stringify())
    .pipe(process.stdout)
    .pipe(outStream)

Input: fails.txt

{
    "foo": "bar"
}

Output:

➜  snappstream-test node compress.js
[
{"type":"Buffer","data":[255,6,0,0,115,78,97,80,112,89]}internal/buffer.js:58
    throw new ERR_OUT_OF_RANGE('value', range, value);
    ^

RangeError [ERR_OUT_OF_RANGE]: The value of "value" is out of range. It must be >= 0 and <= 4294967295. Received 4_428_318_634
    at checkInt (internal/buffer.js:58:11)
    at writeU_Int32LE (internal/buffer.js:678:3)
    at Buffer.writeUInt32LE (internal/buffer.js:691:10)
    at /home/jonas/Desktop/snappstream-test/node_modules/snappystream/lib/snappystreams.js:83:22
    at /home/jonas/Desktop/snappstream-test/node_modules/snappystream/node_modules/async/dist/async.js:1140:9
    at /home/jonas/Desktop/snappstream-test/node_modules/snappystream/node_modules/async/dist/async.js:473:16
    at iteratorCallback (/home/jonas/Desktop/snappstream-test/node_modules/snappystream/node_modules/async/dist/async.js:1064:13)
    at /home/jonas/Desktop/snappstream-test/node_modules/snappystream/node_modules/async/dist/async.js:969:16
    at /home/jonas/Desktop/snappstream-test/node_modules/snappystream/node_modules/async/dist/async.js:1137:13

Input: works.txt

{
    "foo": "bar"
}

➜  snappstream-test node compress.js
[
{"type":"Buffer","data":[255,6,0,0,115,78,97,80,112,89]}
,
{"type":"Buffer","data":[1,23,0,0,237,49,157,176,123,10,32,32,32,32,34,102,111,111,34,58,32,98,97,114,10,125,10]}
]