Giter Club home page Giter Club logo

asyncblock's Introduction

                                        ______  ______              ______  
______ ______________  _________ __________  /_ ___  /______ __________  /__
_  __ `/__  ___/__  / / /__  __ \_  ___/__  __ \__  / _  __ \_  ___/__  //_/
/ /_/ / _(__  ) _  /_/ / _  / / // /__  _  /_/ /_  /  / /_/ // /__  _  ,<   
\__,_/  /____/  _\__, /  /_/ /_/ \___/  /_.___/ /_/   \____/ \___/  /_/|_|  
                /____/                                                      

==================================================================

A fully fledged flow control library built on top of fibers. Don't want to use Fibers and on node v4+? Check out asyncblock-generators.

###Installation

npm install asyncblock

See node-fibers for more information on fibers

Why should I use asyncblock?

  • Write async code in synchronous style without blocking the event loop
  • Effortlessly combine serial and parallel operations with minimal boilerplate
  • Produce code which is easier to read, reason about, and modify
    • Compared to flow control libraries, asyncblock makes it easy to share data between async steps. There's no need to create variables in an outer scope or use "waterfall".
  • Simplify error handling practices
    • If an error occurs in an async step, automatically call your callback with the error, or throw an Error
  • Improve debugging by not losing stack traces across async calls
    • Line numbers don't change. What's in the stack trace maps directly to your code (You may lose this with CPS transforms)
    • If using a debugger, it's easy to step line-by-line through asyncblock code (compared to async libraries)

Overview

Check out the overview to get an at-a-glance overview of the different ways asyncblock can be used.

Examples

A few quick examples to show off the functionality of asyncblock:

Sleeping in series

asyncblock(function(flow){
    console.time('time');

    setTimeout(flow.add(), 1000);
    flow.wait(); //Wait for the first setTimeout to finish

    setTimeout(flow.add(), 2000);
    flow.wait(); //Wait for the second setTimeout to finish

    console.timeEnd('time'); //3 seconds
});

Sleeping in parallel

var ab = require('asyncblock');

ab(function(flow){
    console.time('time');

    setTimeout(flow.add(), 1000);
    setTimeout(flow.add(), 2000);
    flow.wait(); //Wait for both setTimeouts to finish

    console.timeEnd('time'); //2 seconds
});

Trapping results

var ab = require('asyncblock');

ab(function(flow) {
    //Start two parallel file reads
    fs.readFile(path1, 'utf8', flow.set('contents1'));
    fs.readFile(path2, 'utf8', flow.set('contents2'));
    
    //Print the concatenation of the results when both reads are finished
    console.log(flow.get('contents1') + flow.get('contents2'));
    
    //Wait for a large number of tasks
    for(var i = 0; i < 100; i++){
        //Add each task in parallel with i as the key
        fs.readFile(paths[i], 'utf8', flow.add(i));                                    
    }
    
    //Wait for all the tasks to finish. Results is an object of the form {key1: value1, key2: value2, ...}
    var results = flow.wait();
    
    //One-liner syntax for waiting on a single task
    var contents = flow.sync( fs.readFile(path, 'utf8', flow.callback()) );
    
    //See overview & API docs for more extensive description of techniques
});

With source transformation

//asyncblock.enableTransform() must be called before requiring modules using this syntax.
//See overview / API for more details

var ab = require('asyncblock');

if (ab.enableTransform(module)) { return; }

ab(function(flow) {
    //Start two parallel file reads
    var contents1 = fs.readFile(path1, 'utf8').defer();
    var contents2 = fs.readFile(path2, 'utf8').defer();
    
    //Print the concatenation of the results when both reads are finished
    console.log(contents1 + contents2);
    
    var files = [];
    //Wait for a large number of tasks
    for(var i = 0; i < 100; i++){
        //Add each task in parallel with i as the key
        files.push( fs.readFile(paths[i], 'utf8').future() );
    }
    
    //Get an array containing the file read results
    var results = files.map(function(future){
        return future.result;
    });
    
    //One-liner syntax for waiting on a single task
    var contents = fs.readFile(path, 'utf8').sync();
    
    //See overview & API docs for more extensive description of techniques
});

Returning results and Error Handling

var ab = require('asyncblock');

if (ab.enableTransform(module)) { return; }

var asyncTask = function(callback) {
    ab(function(flow) {
        var contents = fs.readFile(path, 'utf8').sync(); //If readFile encountered an error, it would automatically get passed to the callback

        return contents; //Return the value you want to be passed to the callback
    }, callback); //The callback can be specified as the 2nd arg to asyncblock. It will be called with the value returned from the asyncblock as the 2nd arg.
                  //If an error occurs, the callback will be called with the error as the first argument.
});

API

See API documentation

Stack traces

See stack trace documentation

Error handling

See error handling documentation

Formatting results

See formatting results documentation

Parallel task rate limiting

See maxParallel documentation

Task timeouts

See timeout documentation

Concurrency

Both fibers, and this module, do not increase concurrency in nodejs. There is still only one thread executing at a time. Fibers are threads which are allowed to pause and resume where they left off without blocking the event loop.

Risks

  • Fibers are fast, but they're not the fastest. CPU intensive tasks may prefer other solutions (you probably don't want to do CPU intensive work in node anyway...)
  • Not suitable for cases where a very large number are allocated and used for an extended period of time (source)
  • It requires V8 extensions, which are maintained in the node-fibers module
    • In the worst case, if future versions of V8 break fibers support completely, a custom build of V8 would be required
    • In the best case, V8 builds in support for coroutines directly, and asyncblock becomes based on that
  • When new versions of node (V8) come out, you may have to wait longer to upgrade if the fibers code needs to be adjusted to work with it

Compared to other solutions...

A sample program in pure node, using the async library, and using asyncblock + fibers.

Pure node

function example(callback){
    var finishedCount = 0;
    var fileContents = [];

    var continuation = function(){
        if(finishedCount < 2){
            return;
        }

        fs.writeFile('path3', fileContents[0] + fileContents[1], function(err) {
            if(err) {
                throw new Error(err);
            }

            fs.readFile('path3', 'utf8', function(err, data){ 
                console.log(data);
                console.log('all done');
            });
        });
    };

    fs.readFile('path1', 'utf8', function(err, data) {
        if(err) {
            throw new Error(err);
        }

        fnishedCount++;
        fileContents[0] = data;

        continuation();
    });

    fs.readFile('path2', 'utf8', function(err, data) {
        if(err) {
            throw new Error(err);
        }

        fnishedCount++;
        fileContents[1] = data;

        continuation();
    });
}

Using async

var async = require('async');

var fileContents = [];

async.series([
    function(callback){
        async.parallel([
            function(callback) {
                fs.readFile('path1', 'utf8', callback);
            },

            function(callback) {
                fs.readFile('path2', 'utf8', callback);
            }
        ],
            function(err, results){
                fileContents = results;                                    
                callback(err);
            }
        );
    },

    function(callback) {
        fs.writeFile('path3', fileContents[0] + fileContents[1], callback);
    },

    function(callback) {
        fs.readFile('path3', 'utf8', function(err, data){
            console.log(data);
            callback(err);
        });
    }
],
    function(err) {
        if(err) {
            throw new Error(err);
        }
        
        console.log('all done');
    }
);

Using asyncblock + fibers

var ab = require('asyncblock');

ab(function(flow){
    fs.readFile('path1', 'utf8', flow.add('first'));
    fs.readFile('path2', 'utf8', flow.add('second'));
    
    //Wait until done reading the first and second files, then write them to another file
    fs.writeFile('path3', flow.wait('first') + flow.wait('second'), flow.add()); 
    flow.wait(); //Wait on all outstanding tasks

    fs.readFile('path3', 'utf8', flow.add('data'));

    console.log(flow.wait('data')); //Print the 3rd file's data
    console.log('all done');
});

Using asyncblock + source transformation

//Requires asyncblock.enableTransform to be called before requiring this module
var ab = require('asyncblock');

if (ab.enableTransform(module)) { return; }

ab(function(flow){
    var first = fs.readFile('path1', 'utf8').defer();
    var second = fs.readFile('path2', 'utf8').defer();
    
    fs.writeFile('path3', first + second).sync();

    var third = fs.readFile('path3', 'utf8').defer();

    console.log(third);
    console.log('all done');
});

No prototypes were harmed in the making of this module

asyncblock's People

Contributors

axkibe avatar horiuchi avatar scriby avatar vkarpov15 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

asyncblock's Issues

No fibers found...

in 2.2.4, I'm getting the following error:

[dtsgen]    Error: Cannot find module 'fibers'
[dtsgen]        at Function.Module._resolveFilename (module.js:336:15)
[dtsgen]        at Function.Module._load (module.js:278:25)
[dtsgen]        at Module.require (module.js:365:17)
[dtsgen]        at require (module.js:384:17)
[dtsgen]        at /Users/arthur/code/shopstyle-node-boilerplate/node_modules/asyncblock/lib/asyncblock.js:13:23
[dtsgen]        at handleFiber (/Users/arthur/code/shopstyle-node-boilerplate/node_modules/asyncblock/lib/asyncblock.js:95:21)
[dtsgen]        at asyncblock (/Users/arthur/code/shopstyle-node-boilerplate/node_modules/asyncblock/lib/asyncblock.js:28:16)
[dtsgen]        at module.exports (/Users/arthur/code/shopstyle-node-boilerplate/node_modules/asyncblock/lib/asyncblock.js:44:5)
[dtsgen]        at processGenerate (/source/cli.ts:36:3)
[dtsgen]        at Object.<anonymous> (/source/cli.ts:31:3)

i'm no expert, so if this doesn't look like you're issue, sorry.

I see you're last commit changed from fibers to fibers-scriby, but appears that something is still looking for fibers?

storing deferred results in a hash

When I try to store the results of a deferred call in a hash, I'm getting the value of the deferred object rather than the result it points to. Here's a simple program to demonstrate what's happening:

(function() {
    var fs = require("fs");
    var asyncblock = require("asyncblock");

    if (asyncblock.enableTransform(module)) {
        return;
    }

    asyncblock(function() {
        var synced = fs.readFile(__filename, 'utf8').sync();
        var deferred = fs.readFile(__filename, 'utf8').defer();
        var hash = {};
        hash.sync = synced;    // this works just fine
        hash.defer = deferred; // this doesn't work
        console.log(hash);

        console.log(deferred); // this works just fine too

        var d = deferred;      // using an intermediate works
        hash.defer = d;
        console.log(hash);
    });
})();

Here's the suspicious output from the hash assignment:


  defer: 
   { _flow: 
      { _parallelCount: 1,
        _parallelFinished: 0,
        _fiber: [Object],
        _taskQueue: [],
        _forceWait: false,
        _light: true,
        _finishedTasks: {},
        errorCallback: null,
        taskTimeout: null,
        timeoutIsError: null,
        _originalError: [Error],
        _parentFlow: undefined,
        _isGenerator: false,
        maxParallel: 0,
        firstArgIsError: true },
     _key: '__defaultkey__',
     _result: null,
     _resultObtained: false } }

undefined return value

asyncblock(function(){

}, callback);

In the above example, callback(undefined) will be called instead of callback(). It's probably better to call the callback with no arg at all in this case.

some codes can't work with source transformation

Hello. I was in trouble with asyncblock (using source transformation), and I found the minimum case that couldn't work well. Do I make any mistakes?

Here is the codes.

// foo.js

var asyncblock = require('asyncblock');

if (asyncblock.enableTransform(module)) {
    return;
}

asyncblock(function() {
    var ary = getArray().sync();
    var num = getNum().defer();
    var unused1 = getNum().sync();
    var unused2 = plus1(num).defer();
    // console.log(num); // it becomes to work if this line is activated
    ary.push(num);
    console.log(ary); // [1, 2, 3] is the desired result
});

function getArray(callback) {
    setImmediate(function() {
        callback(null, [1, 2]);
    });
}
function getNum(callback) {
    setImmediate(function() {
        callback(null, 3);
    });
}
function plus1(num1, callback) {
    setImmediate(function() {
        callback(null, num1 + 1);
    });
}

And when I run node foo.js, the following error occurs.

foo.js:30
        callback(null, num1 + 1);
        ^
TypeError: undefined is not a function
    at Object._onImmediate (foo.js:30:9)
    at processImmediate [as _immediateCallback] (timers.js:330:15)

I think it is a minimum case because it becomes to work if either unused variable is removed, either defer() is changed to sync(), or function plus1's 1st argument is removed.

I use asyncblock 2.1.20 (the newest for now), node 0.10.24.

Sorry for my bad English. Thanks.

flow.sync within add/wait

Hey, I noticed an interesting issue as illustrated below:

asyncblock(function(flow) {

    var foo = function(callback) {
        flow.sync.doSomething();
        callback();
    };

    foo(flow.add());
    flow.wait();

});

It seems the flow.wait pauses indefinitely. Should I expect to be able to make a sync call within an add/wait scenario?

--wampleek

asyncblock.enableTransform() doesnt working

Hi. I'm trying this:

fs = require('fs');
asyncblock = require('asyncblock');

asyncblock.enableTransform();

asyncblock(function(flow) {
  var content = fs.readFile("/etc/passwd", "utf-8").sync();
  console.log(content);
});

Console says:
TypeError: Cannot call method 'sync' of undefined

What can be wrong?

Request support for n parallel threads

Hey, I've been using asyncblock for a couple weeks now, and I love the concise syntax. Recently I ran into a situation where I would like to have n parallel execution threads going at once. I've settled on something like:

for(var i = 0; i < numberOfExecutions; i++) {
    doTask(flow.add());
    if(i % 10 === 0) {
        flow.wait();
    }
}
flow.wait();

This works somewhat well. It will kick off 10 tasks at once, but will wait for all 10 to complete before issuing new tasks. It would be nice if I could have 10 tasks going at once until they all complete. What do you think?

Is it possible to use asyncblock with the Q promise library?

A have a library that returns Q promises to make async callbacks a little easier to use on clients. However, I have a use case where I need to block and wait for the promise to be fulfilled.

I'd like to use asyncblock to achieve this, but I can't figure out how to get it to work with promises.

For example, i'd like to use it to block the return from the following function until the last promise has been fulfilled.:

function authenticate() {
    return getUsername()
    .then(function (username) {
        return getUser(username);
    })
    // chained because we will not need the user name in the next event
    .then(function (user) {
        return getPassword()
        // nested because we need both user and password next
        .then(function (password) {
            if (user.passwordHash !== hash(password)) {
                throw new Error("Can't authenticate");
            }
        });
    })
    .done();

    // HERE i'd like to block and not return from authenticate() until the last promise has been fulfilled
}

Idea

What if I could pass informations to doneAdding() and add()?

I mean this:

asynchronous(function(){
var doc = {foo : bar};
flow.doneAdding(doc);
});

console.log(flow.forceWait()); // Output {foo : bar}

Thanks for your last modifications and good holly days for you!

x().sync() optimization

When a function call occurs with .sync() we know it must finish before continuing the current program flow. So, any asyncblocks created directly within the original fiber can reuse the fiber without altering program results.

Global variable leaks

File lib/flow.js contains global leaks for variables fn and fiber in function errorHandler :

var errorHandler = function(self, task){
    if(task != null && task.result && task.result[0] && task.firstArgIsError){
        if(!task.ignoreError) {
            // Some code removed... 

            fn = null;
            fiber = null;

            throw task.error;
        }
    }
};

"use strict" throws Fatal error

Hello,

I'm using you library, but with "use strict" enabled, my node module always throws

Fatal error: Illegal access to a strict mode caller function.

If I remove "use strict" all works fine.
So I guess your lib won't work with "use strict"?

I'm using the latest node.js version.

BR,
mybecks

How to emulate a simple blocking call?

I'm trying to re-use an existing asyncblock scope to avoid sprinkling a new scope in every function that needs to call async APIs and to support blocking calls. The module.exports.getCurrentFlow seems to be what I need. Is getCurrentFlow() officially supported?

"use strict";

console.log("main start");
console.time("main finish");

let asyncblock = require("asyncblock");

//let optionForceError = true;
let optionForceError = false;

function nonblockingRoutine (done) {
  if (optionForceError)
    setTimeout(() => done(new Error("something bad happened"), "test data"), 1000);
  else
    setTimeout(() => done(null, "test data"), 1000);
}

function blockingRoutine () {
  console.log("    blockingTest 1");
  let flow = asyncblock.getCurrentFlow();
  nonblockingRoutine(flow.add());
  let result = flow.wait();
  console.log("    blockingTest 2");
  return result;
}

asyncblock((flow) => {
  console.log("  enter level 1");
  try {
    let result = blockingRoutine();
    console.log("  result A = " + JSON.stringify(result));
  } catch (e) {
    console.log("  caught exception:", e.stack);
  }
  console.log("  leave level 1");
});

console.timeEnd("main finish");

Transform breaks on "comment" object key

When transforming a source file that uses an object key named "comment", the parser spits out a mangled line. For example:

{
...
    reason: request.param("reason"),
    comment: request.param("comment"),
    userInfo: request.sessionHelper.getUserInfo()
...    
}

gets transformed to:

{
...
    reason: request.param("reason"),
    //call,dot,name,request,param,,,string,comment

    userInfo: request.sessionHelper.getUserInfo()
...
}

Fortunately for this case the line number changes as well. However, I've seen this other places as well (after knowing what to look for), and it can happen without affecting the line numbers too.

interactive?

I'm trying to integrate your module with an interactive solution.

I'm colliding with an error:

/home/gianfri/Scrivania/Synthia_0.2/proto2/synthia/bin/lib/asyncblock/asyncblock/lib/flow_fiber.js:26
        this._fiber.run(task);
                    ^

TypeError: Cannot set property '_asyncblock_flow' of null
    at fiberContents (/home/gianfri/Scrivania/Synthia_0.2/proto2/synthia/bin/lib/asyncblock/asyncblock/lib/asyncblock.js:88:32)

in the non-interactive version it works perfectly but in the interactive one it gives me this problem

I try to explain how it works

we have an opening process that at a certain point launches

this.synchronizer ((line) => {
   ...
}

like ab(function(flow) { ... }

at some point I define a context with

  this._iVm = require('vm'),
  this._isandbox = require('vm').createContext({
      require: require,
      process:process,
      console: console,
      module:module,
      app: this,
      global:global,
  })

then I execute my instruction in my sandbox

this._iVm.runInContext(this._iteraction.buffer.join("\n"), this._isandbox);

from the same problem even if I use require('vm').runInThisContext(...)

repeat it works correctly until I try to use the module, but only in the interactive version.

some idea?
(I also tried with the normal version 'node -i' and the probema is the same)

asyncblock.enumerator leaks memory if not exhausted

If an enumerator isn't fully exhausted, the fiber doesn't appear to be getting garbage collected.

See if there's anything we can do about this, and at the very least add an "abort" method to clean up the enumerator without moving all the way through it.

enabling Transforms in exported module

I'm using the following code, which works great:

module.exports = {
  'bla': function() {
    var flow = asyncblock.getCurrentFlow();
    an.asyncCall('params', flow.add('a'));
    return flow.get('a');
  }
};

It might be nice to enable Transforms and use something like this:

module.exports = {
  'bla': function() {
    return an.asyncCall('params', flow.add('a')).sync();
  }
};

But since the exports are not wrapped in an asyncblock, the sync call doesn't get transformed and the result is TypeError: an.asyncCall(...).sync is not a function

Since this all lives in an Express app, my asyncblock gets launched elsewhere, but if I do explicitly wrap the exports in an asyncblock in order to force the transform, I get this:

/bla/node_modules/asyncblock/lib/flow_fiber.js:26
        this._fiber.run(task);
                    ^

TypeError: Cannot set property '_asyncblock_flow' of null
    at fiberContents (/bla/node_modules/asyncblock/lib/asyncblock.js:88:32)

Am I doing something wrong? Is there another way/is it possible to enable Transforms?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.