readlines-ng's Introduction

Tiny and fast async generator to read streams line by line

Node 10+ is required. This package provides very simple but feature-reach functionality to consume readable streams or files line by line. It's also quite fast (up to 30% faster than readline) and has no dependencies.

Features:

reading of files (by name)
reading of readable streams
generator mode via readlines
readable stream mode via LineStream
customizable separator (default is \n)
raw bytes mode if encoding option is null
configurable lookahead buffer size

Documentation

Have a look at the index.js, it's really tiny. Or check the usage example below.

Usage examples:

const { readlines, LineStream } = require('readlines-ng');

// As a generator
for await (const line of readlines(filename, { encoding: 'utf8' })) {
  console.log(line);
}

// As a readable stream
const reader = new LineStream(filename, {
  chunkSize: 256*1024,
  encoding: null,
});
for await (const line of reader) {
  console.log(line);
}

// Custom separator
const { PassThrough } = require('stream');
const datasource = new PassThrough();
datasource.write('qux42');
datasource.write('foobar42');
datasource.end();

const reader = new LineStream(filename, { sep: '42' });
for await (const line of reader) {
  console.log(line);
}

Tests

npm run test

Benchmark

Trivial wc -l impl using this package exposes the performance better than the standard readline module.

npm run bench </path/to/huge/file>  # Node 11+ is required

# File size ~2.8 GB
> Benchmark: lines counter (readline module)
> 5974719 lines found
> Done in 5.757290349 sec

> Benchmark: lines counter (async generator)
> 5974719 lines found
> Done in 4.707053160 sec

> Benchmark: lines counter (async generator UTF-8)
> 5974719 lines found
> Done in 3.937784544 sec

> Benchmark: lines counter (readable stream)
> 5974719 lines found
> Done in 4.984460949 sec

> Benchmark: lines counter (readable stream UTF-8)
> 5974719 lines found
> Done in 4.354594750 sec

Recommend Projects