stieneee / simple-zstd Goto Github PK

hello, is it possible and can you provide an example of decompressing/compressing a uint8array that would build and work in a browser environment (in my case webpack)

Hi

Great library, thanks for contributing.

This might be me being naive; apologies if so.

I want to uncompress a large text file on the fly, and I have different phases in the code (i.e. the file starts with metadata lines which I want to read all of, then data lines, which I want to expose as an async iterator. However, I'm at a simpler point so far.

When I run the following code in Mocha, it succeeds, but then hangs. If I take out the break it works just fine.

import * as readline from "readline";
import * as fs from "fs";
import * as path from "path";
import { ZSTDDecompressMaybe } from "simple-zstd";

it("process actual lines", async () => {
  console.log(__dirname);
  let filePath = path.join(__dirname, "/data/NA24385.trio.subset.vcf.zst");
  const stream = fs.createReadStream(filePath).pipe(ZSTDDecompressMaybe());
  const lines = readline.createInterface({
    input: stream,
    crlfDelay: Infinity,
  });

  let n = 0;
  for await (const line of lines) {
      n += line.length;
      if (n > 1024*1024) {
          break;
      }
  }
  console.log(n);
});

I presume I need to clean up somehow. Is there a better way to do this?

I assume the behaviour doesn't particularly depend on the specific compressed data. My file is 4.4Mb, which is a bit fat for a repro, but if you think it necessary, I can cut it down and see if the behaviour still exists.

I am facing error while using function ZSTDCompress function for compression of stream data to compressed buffer.

Node Version: 10.16.0
Npm Version: 6.9.0
OS: Ubuntu 18.04

What can be the solution to resolve this error?

Following are the error logs :

`error: uncaughtException: Command failed: Incorrect parameters
Usage :
zstd [args] [FILE(s)] [-o file]

FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
-# : # compression level (1-19, default:3)
-d : decompression
-D file: use file as Dictionary
-o file: result stored into file (only if 1 input file)
-f : overwrite output without prompting and (de)compress links
--rm : remove source file(s) after successful de/compression
-k : preserve source file(s) (default)
-h/-H : display help/long help and exit

{ date: 'Thu Jul 18 2019 11:00:01 GMT+0530 (India Standard Time)',
process:
{ pid: 8622,
uid: 0,
gid: 0,
cwd: '/home/gdmtrunk',
execPath: '/usr/local/bin/node',
version: 'v10.16.0',
argv:
[ '/usr/local/bin/node',
'/home/gdmtrunk/server.js' ],
memoryUsage:
{ rss: 170455040,
heapTotal: 129097728,
heapUsed: 112122016,
external: 72180011 } },
os:
{ loadavg: [ 1.04345703125, 1.24267578125, 1.625 ],
uptime: 2035 },
trace:
[ { column: 12,
file:
'/home/gdmtrunk/node_modules/duplex-child-process/index.js',
function: 'ChildProcess.onExit',
line: 116,
method: 'onExit',
native: false },
{ column: 20,
file: 'events.js',
function: 'Object.onceWrapper',
line: 286,
method: 'onceWrapper',
native: false },
{ column: 13,
file: 'events.js',
function: 'ChildProcess.emit',
line: 198,
method: 'emit',
native: false },
{ column: 20,
file: 'domain.js',
function: 'ChildProcess.EventEmitter.emit',
line: 448,
method: 'emit',
native: false },
{ column: 16,
file: 'internal/child_process.js',
function: 'maybeClose',
line: 982,
method: null,
native: false },
{ column: 11,
file: 'internal/child_process.js',
function: 'Socket.stream.socket.on',
line: 389,
method: 'on',
native: false },
{ column: 13,
file: 'events.js',
function: 'Socket.emit',
line: 198,
method: 'emit',
native: false },
{ column: 20,
file: 'domain.js',
function: 'Socket.EventEmitter.emit',
line: 448,
method: 'emit',
native: false },
{ column: 12,
file: 'net.js',
function: 'Pipe._handle.close',
line: 606,
method: 'close',
native: false } ],
stack:
[ 'Error: Command failed: Incorrect parameters',
'Usage : ',
' zstd [args] [FILE(s)] [-o file] ',
'',
'FILE : a filename ',
' with no FILE, or when FILE is - , read standard input',
'Arguments : ',
' -# : # compression level (1-19, default:3) ',
' -d : decompression ',
' -D file: use file as Dictionary ',
' -o file: result stored into file (only if 1 input file) ',
' -f : overwrite output without prompting and (de)compress links ',
'--rm : remove source file(s) after successful de/compression ',
' -k : preserve source file(s) (default) ',
' -h/-H : display help/long help and exit ',
'',
' at ChildProcess.onExit (/home/gdmtrunk/node_modules/duplex-child-process/index.js:116:12)',
' at Object.onceWrapper (events.js:286:20)',
' at ChildProcess.emit (events.js:198:13)',
' at ChildProcess.EventEmitter.emit (domain.js:448:20)',
' at maybeClose (internal/child_process.js:982:16)',
' at Socket.stream.socket.on (internal/child_process.js:389:11)',
' at Socket.emit (events.js:198:13)',
' at Socket.EventEmitter.emit (domain.js:448:20)',
' at Pipe._handle.close (net.js:606:12)' ] }
18 Jul 11:00:01 - [red] Uncaught Exception:
18 Jul 11:00:01 - Error: Command failed: Incorrect parameters
Usage :
zstd [args] [FILE(s)] [-o file]

FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
-# : # compression level (1-19, default:3)
-d : decompression
-D file: use file as Dictionary
-o file: result stored into file (only if 1 input file)
-f : overwrite output without prompting and (de)compress links
--rm : remove source file(s) after successful de/compression
-k : preserve source file(s) (default)
-h/-H : display help/long help and exit

at ChildProcess.onExit (/home/gdmtrunk/node_modules/duplex-child-process/index.js:116:12)
at Object.onceWrapper (events.js:286:20)
at ChildProcess.emit (events.js:198:13)
at ChildProcess.EventEmitter.emit (domain.js:448:20)
at maybeClose (internal/child_process.js:982:16)
at Socket.stream.socket.on (internal/child_process.js:389:11)
at Socket.emit (events.js:198:13)
at Socket.EventEmitter.emit (domain.js:448:20)
at Pipe._handle.close (net.js:606:12)`

I'm not sure if there is a command line flag in zstd that would allow an RCE, especially because as far as I know, it is treated as only one process argument, so no other program can be started with it directly.

This can be fixed by checking the variable to be a number, for example by doing this:

if (!lvl || typeof lvl !== "number") lvl = 3;

never mind.
Please delete.

in node_modules\simple-zstd\index.js

the regexp to remove zstd newline should not be just \n but \r?\n to support windows platform