michalcz / scramjet Goto Github PK

Scramjet is a powerful, yet simple framework written on top of node.js object streams, somewhat similar to the well-known event-stream or highland module, but with a much simplier API and written fully in ES6. It is built upon the logic behind three well known javascript array operations - namingly map, filter and reduce. This means that if you've ever performed operations on an Array in JavaScript - you already know Scramjet like the back of your hand.

The main advantage of scramjet is running asynchronous operations on your data streams. First of all it allows you to perform the transformations both synchronously and asynchronously by using the same API - so now you can "map" your stream from whatever source and call any number of API's consecutively.

The benchmarks are punblished in the scramjet-benchmark repo.

How about a CSV parser of all the parkings in the city of Wrocław from http://www.wroclaw.pl/open-data/...

const request = require("request");
const StringStream = require("scramjet").StringStream;

let columns = null;
request.get("http://www.wroclaw.pl/open-data/opendata/its/parkingi/parkingi.csv")
    .pipe(new StringStream())
    .split("\n")
    .parse((line) => line.split(";"))
    .pop(1, (data) => columns = data)
    .map((data) => columns.reduce((acc, id, i) => (acc[id] = data[i], acc), {}))
    .on("data", console.log.bind(console))

Here's the list of the exposed classes, please review the specific documentation for details:

• scramjet.DataStream - the base class for all scramjet classes.
• scramjet.BufferStream - a DataStream of Buffers.
• scramjet.StringStream - a DataStream of Strings.
• scramjet.MultiStream - a DataStream of Strings.

Note that:

• Most of the methods take a callback argument that operates on the stream items.
• The callback, unless it's stated otherwise, will receive an argument with the next chunk.
• If you want to perform your operations asynchronously, return a Promise, otherwise just return the right value.

The quick reference of the exposed classes:

DataStream is the primary stream type for Scramjet. When you parse your stream, just pipe it you can then perform calculations on the data objects streamed through your flow.

Detailed DataStream docs here

A stream of string objects for further transformation on top of DataStream.

Detailed StringStream docs here

A factilitation stream created for easy splitting or parsing buffers

Detailed BufferStream docs here

An object consisting of multiple streams than can be refined or muxed.

Detailed MultiStream docs here

Scramjet works in the browser too, there's a nice, self-contained sample in here, just run it:

    git clone https://github.com/signicode/scramjet.git
    cd scramjet
    npm install .
    cd samples/browser
    npm start # point your browser to http://localhost:30035 and open console

If you need your scramjet version for the browser, grab browserify and just run:

    browserify lib/index -standalone scramjet -o /path/to/your/browserified-scramjet.js

With this you can run your transformations in the browser, use websockets to send them back and forth. If you do and fail for some reason, please remember to be issuing those issues - as no one person can test all the use cases and I am but one person.

Scramjet uses functional programming to run transformations on your data streams in a fashion very similar to the well known event-stream node module. Most transformations are done by passing a transform function. You can write your function in two ways:

1. Synchronous

Example: a simple stream transform that outputs a stream of objects of the same id property and the length of the value string.

   datastream.map(
       (item) => ({id: item.id, length: item.value.length})
   )

2. Asynchronous (using Promises)

Example: A simple stream that fetches an url mentioned in the incoming object

   datastream.map(
       (item) => new Promise((resolve, reject) => {
           request(item.url, (err, res, data) => {
               if (err)
                   reject(err); // will emit an "error" event on the stream
               else
                   resolve(data);
           });
       })
   )

The actual logic of this transform function is as if you passed your function to the then method of a Promise resolved with the data from the input stream.

As of version 2.0 Scramjet is MIT Licensed.

The project need's your help! There's lots of work to do - transforming and muxing, joining and splitting, browserifying, modularizing, documenting and issuing those issues.

If you want to help and be part of the Scramjet team, please reach out to me, signicode on Github or email me: [email protected].