Giter Club home page Giter Club logo

node-stream-batch's Introduction

stream-batch

Batch streaming items. A batch is created whenever:

  • maxWait ms since last items has arrived (default: 100)
  • maxTime ms since first item in current batch arrived (default: infinity)
  • maxItems items are in the batch (default: infinity)

A batch is an array of the collected events, and can be of different sizes. Within the array, the sequence is the same order as they came in.

Usage example code:

var batch = require('stream-batch');

var someStream = ... // whatever stream
    .pipe( batch() )
    .pipe .... // whatever you want to do next

Example configuration

  • 200 ms after the last item is received: batch({ maxWait: 200})
  • 200 ms after the last item is received, but no longer as 500 ms after the first item was received: batch({ maxWait: 200, maxTime: 500})
  • 200 ms after the last item is received, but no longer as 500 ms after the first item was received or there are 100 items: batch({ maxWait: 200, maxTime: 500, maxItems: 100})

It could be used eg:

  • streams where new items are received in bursts of unknown size, where you want to react to the burst and not to the individual items (but still need them for you actions)
  • streams where new items are received that you want to work on in batches, but the speed of the receipt is unpredictable, eg due to network conditions, load of external servers etc
  • batching output to a server, so that data is written at least every second and or earlier when the batch is of a certain size

node-stream-batch's People

Contributors

hans-d avatar

Watchers

 avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.