I'm trying to use threads.js do some some processing but am encountering this error:

Uncaught TypeError: importScripts.map is not a function(…)

The workflow goes like this. I first import Pool from the threads module:

import {Pool} from 'threads';

And then call it as in the documentation:

 let jobA = this.threadPool.run(function(input, done) {
                workerGetTilesetInfo(input);
                done(input);
            }, {
                workerGetTilesetInfo: 'worker'
            })

           .on('done', function(job, message) {
                console.log('done', message);
           })
           .on('error', function(job, error) {
                console.log('error', error);
           })
            .send(outUrl);

The function getTilesetInfo is defined in another javascript file called worker.js. Is this a valid issue, or am I simply doing something wrong?

Hi,

I was using the thread pool 0.7 for my company app to handle processing of large array of json objects.
Seems like pool.send(objs) will leave objs inside the job .send returns and stays in memory for quite a long time. I had to call

job.once('done', function(stats){
.......
delete job.sendArgs; // to mark it for GC
delete job.runArgs;
job.removeAllListeners();
job = null;
})

to prevent the leak. Please use memwatch to measure the memory footprint.

Thanks,

Hello, first of all sorry for my english.

I'm using the last version of threads.js in a electron project. I'm doing something like (its only a sample.... maybe are syntax errors...):

function myClass(){
    var my_sub_class = new Subclass();
    const Pool = require('threads').Pool;
    const pool = new Pool();
    const _pool_method = pool.run(function(opts, done){
        my_sub_class.method(opts);
        done();
    }
    this.public_method = funciton(opts){
        _pool_method.send(opts)
        .on('done', fucntion(){ console.log('SUCCESS'); })
        .on('error', function(e){ console.log(e); })
    }
   return this;
}
var a = new myClass();
a.public_method({hello: "world"})

This return an error: "Error: my_sub_class.method is not a function ......"

Can I access to my local variable, that is instance of another class (my_sub_class) inside the execution of thread pool? How?

Thanks a lot!!

Long running workers should be able to inform their progress to the main thread. For example,

spawn(...)
.on('progress', function(current, total) { 
   console.log(current/total + " % complete");
});

If this is already achievable in the current implementation, a simple example in Readme would be of great help.

We code:

const Pool = require('threads').Pool;

const pool = new Pool(2);

pool.run(function (input, done) {
  done(input * 1000);
});

pool.send(10);
pool.send(20);
pool.send(30);
pool.send(40);
pool.send(50);
pool.send(60);

pool
  .on('done', function (job, message) {
    console.log('Pid:', job.thread.slave.pid, ', Job done:', message);
  })
  .on('error', function (job, error) {
    console.error('Pid:', job.thread.slave.pid, ', Job error:', error);
  })
  .on('finished', function () {
    console.log('Everything done, shutting down the thread pool.');
    pool.killAll();
  });

The results:

Pid: 2326 , Job done: 10000
Pid: 2327 , Job done: 20000
Pid: 2326 , Job done: 30000
Pid: 2327 , Job done: 40000
Everything done, shutting down the thread pool.

Hi, I have noticed there is an issue when you try to access or even define global on any NodeJS thread. Is there some way that this sort of functionality can be added to new spawned processes? I realize globals aren't typically best practice, have several scripts which rely on a few variables that I would like to be able to migrate to/from the code in relation to directories/file structure.

After reading the doc for 2 times I still do not know the clear API of Worker, so if I want to spawn a thread to run function a() and then function b(), is this the correct code?

spawn(function a() {})
.send({/*data for fn a*/)
.run(function b(){})
.send(/* data for fn b*/)
.promise()
.then(function(finalResult) { /* ... */})

This is good work. Thanks for sharing.

Wondering how to specify the pool size (no. of threads to be created), and its scheduling characteristics?

Also, how are the jobs scheduled in the pool? Do the threads pull the jobs from a waiting-queue or are the jobs queued/pushed onto the threads?

In the later-case, would it be possible to specify thread-scheduling criteria (such as 'roundrobin', 'least-used' etc...). ?

I can't figure out how to get a worker that requires/imports an npm lib working in the browser. Works fine on the nodejs side. Is there an example or a guide that shows how to bundle this properly?

There may be a very obvious answer but is there any reason why I cannot pass a function as a parameter into a thread?

Thanks.

I've been banging my head for a while, and at this point, I honestly have no idea what's going on or how to fix it. I'm using Node 8.5 on Mac OSX. I've verified that the worker is loading properly. Every aspect of this module works for me except this one piece. Any thoughts?

THREADER.JS

module.exports.setup = async worker =>
{
    const threads = require('threads');
    const pool = new threads.Pool();
    pool.run(worker);
    let someFunction = (async foo =>
    {
         return foo;
    });
    const job = await pool.send([someFunction, 'bar']);
}

WORKER.JS

module.exports = async(input, done) =>
{
    const [theFunction, theArg] = input;
    console.log(theFunction(theArg)); // Fails with: theFunction is not a function
}

As the title says I'm trying to spawn a thread which connects to a mssql instance and then returns the result. But for some reason I won't get any result at all. Here is the code

const spawn = require('threads').spawn;

const thread = spawn(function(input, done) {
  const sql = require('mssql');
  sql.connect(`mssql://SQLadmin:SQLadmin@${input.ip}/master`)
    .then(() => {
      return sql.query`select * from sys.databases`;
    })
    .then(result => {
      console.log(result);
      done({ string : result });

    }).catch(err => {
      console.log(err);
      done({ string : err });
    });
});

thread
  .send({ ip : '192.168.5.179' })
  .on('message', function(response) {
    console.log(response.string);
    thread.kill();
  })
  .on('error', function(error) {
    console.error('Worker errored:', error);
  })
  .on('exit', function() {
    console.log('Worker has been terminated.');
  });

Shouldn't this work?

Sometimes we may want to abort a job:

const jobA = pool
  .run((input, done) => {
    somethingTakeALongTime()
    done(hash, input))
  .send({ do : 'something' })
// ...
jobA.abort()

I think that it can be implemented by adding a field isAborted in job object.
When user calls job.abort(),

1. If the job is executing on a thread, kill the thread and spawn a new one in the pool
2. If the job has not been executed, set the field isAborted to true, and add a judgement here to check if a job is marked as aborted before execution.

While worker.node/slave.js subscribes and nicely handles 'uncaughtException' event, similar 'unhandledRejection' is not handled there. And hence, promise-based workloads don't have the same "last resort" error catching as callback-based.

Here's an example:

<html>
    <body>
    <script src="https://cdn.rawgit.com/andywer/threads.js/v0.7.2/dist/threads.browser.js"></script>

    Hi there
<script>
    var pool = new thread.Pool();
    var scriptPath = document.location.href;

    function calcStuff(v) {
        console.log('v:', v);

        pool.run(function(input, done) {
                    let newVal = input.val + 1;
                    done(newVal);
                 })
           .on('done', function(job, message) {
                console.log('done', message);
           })
           .on('error', function(job, error) {
                console.log('error', error);
           })
        .send({val: v});
    }

    calcStuff(1);
    calcStuff(2);

</script>
    </body>
</html>

I would expect this code to output:

v: 1
v: 2
done 2
done 3

Instead, it outputs the done commands twice:

v: 1
v: 2
done 2
done 2
done 3
done 3

Is this intended behavior? It doesn't seem like what one would normally expect.

Hi,

I ran into the following issue: I am using threads in a typescript project of mine. I use webpack to transpile. I successfully build my project, but when I run it, I get the error Error: Cannot find module '/slave.js'.

I have been digging a bit, and the error originates from node_modules/threads/lib/worker.node/worker.js, line 37:

_this.slave = _child_process2.default.fork(_path2.default.join(__dirname, 'slave.js'), [], options);

Problem statement:

• When I use webpack to build my project, __dirname points to "/"
• however, when I simply execute with ts-node, __dirname does contain the correct path, and the thread successfully runs the referenced script.

Could somebody perhaps assist me in solving this issue (other than hacking in worker.js :) )

Thanks,
Willem

edit: this concerns code which is to be run on the server.

Hi. This is more a question than a real bug report:

I'd like to use the IndexedDb from within the worker script (I'm using the pool on the non-worker side) and share that IndexedDb with the main application. Sadly that's not possible, because the worker gets injected by a Blob-URL and - as a result - runs in a different security domain. There's no shareable IndexedDb possible with that setup - except when commenting some lines to use the IE fallback.

Any idea how that can be achieved?

Check: http://stackoverflow.com/questions/10343913/how-to-create-a-web-worker-from-a-string

Maybe using blob or some library would be a better solution

I ran the example code in the documentation but the application never completes. The output I get is:

123 * 2 = 246
Worker has been terminated.

But I have to CTRL+C to force quit the application. Did I miss something? I'm using Fedora 20. Thanks

This is rather an enhancement idea than a defect.

When a job executes in the pool, it reloads the worker with the script to execute. When this script is loaded using a URL, Chrome requests the script with Cache-Control: no-cache from the server.

In my use case with 20000 executions of the same job script, I therefore see 20000 requests to the server, each costing about 10ms according to Chrome developer tools.

I believe it should not be necessary to re-read the script or even re-initialize the worker behind the job as long as the job is using the same script as the worker is already initialized with.

I think the time has come to look into more-or-less rewriting threads with the lessons-learned in mind, making the code clearer and bumping the version to 1.0.

To Do

• Focus API on observables and promises instead of event emitting
  • Pool job can be promise-centric, Thread & Pool observable
• Use sync, async and possibly generator functions as worker code (instead of done callback)
• Thread API should look more like what workerize does
• Deduplicate browser and node worker code (might wanna use tiny-worker)
• Make sure all previous features still work

Nice to have

• Try to write code in a more functional, but still performance-sensitive fashion
• SharedArrayBuffer polyfill for node + easy usage with the lib
• Nicer integration of map-reduce patterns and thread pooling
• thread.send() and thread.run() are not the best names
• Pool.prototype.send() might be better called Pool.prototype.queue()
• Solution to clean up after finished pool job / support sending data messages and distinct metadata messages to thread/job (#48)
• Turn config into something else than a singleton
• Proper API docs

In nodejs.
Example:
I have a file storage.js, it has a variable to save some data, I wanna save data in a thread, and get data in another thread.

i have the following script:
var thread = spawn(function(input, done) {
var blahblah = require('./inc/blahblahblah.js');
});
i am not being able to import modules from within the process.
Error: Cannot find module

I am trying to follow one of the examples, the Progress Update Example. I get the progress messages, but the master thread never receives the done event. This is occurring on all other examples I try too.

Here is the code I'm running:

const threads = require('threads');
const config  = threads.config;
const spawn   = threads.spawn;

const thread = spawn(function() { });

thread
 .run(function(input, done, progress) {
    setTimeout(done, 1000);
    setTimeout(function() { progress(25); }, 250);
    setTimeout(function() { progress(50); }, 500);
    setTimeout(function() { progress(75); }, 750);
  })
  .send()
  .on('progress', function(progress) {
    console.log(`Progress: ${progress}%`);
  })
  .on('done', function() {
    console.log(`Done.`);
  });

Most of the examples are not full source code examples. I have no idea where thread is supposed to come from - perhaps I got it wrong.

Here is the output I get:

$ node index.js
Progress: 25%
Progress: 50%
Progress: 75%

I have to CTRL+C to get out, as nothing further ever happens.

I would like to know what I'm doing wrong here. Perhaps some working examples could be provided with the package, since at the moment the code fragments on the documentation are incomplete.

From what I can tell, no.

Unless there is some way to pass in options... Cursory look, I don't see that it's possible.

I've been developing a feature around this module, but may need to have this ability in order to move forward. I'd consider putting in a PR but wanted to get a read on whether you think this is a feature you'd like to have.

Thanks!

Just playing around to see if this library is something I could use and I like it!

I do however have an issue with the code below, I tried to keep it as simple as possible.

const threadPool = new Pool();

for (const letter of ['a', 'b', 'c']) {
  threadPool.run((params, message) => {
    message(`The letter was: ${letter}`);
  }).send({ letter });
}

threadPool
  .on('done', (job, message) => {
    console.log('Job done:', job);
  })
  .on('error', (job, error) => {
    console.error('Job errored:', job);
  })
  .on('finished', () => {
    console.log('Everything done, shutting down the thread pool.');
    threadPool.killAll();
  });

Outputs:

$ node index.js
Job done: The letter was: a
Job done: The letter was: b
Job done: The letter was: c
Everything done, shutting down the thread pool.

However, the threadPool.killAll(); doesn't seem to to anything as the console stays idling?

Am I doing this wrong?

Environment is Mac OS Sierra with node v8.3.0.

Thanks!

Another one I was caught in:

I have a script I intend to use for a pool. Using webpack, I generated the path to this file and this results in the following:

    /path/worker.js

Further, I configure the base path like this:

    basepath: {
      web: "http://localhost:8084",
      node: ""
    }

When constructing the Pool, Chrome complains about not being able to load the script, because this URL is generated:

  http://localhost:8085//path/worker.js

Note the additional / in the path. My workaround for this is to remove the leading slash from the otherwise perfectly valid path. But I believe the additional / is just wrong, because when combining script path and basepath, the script path needs to be absolute anyway, thus the leading slash would need to be always there.

Trying out threads.js

Ran into a problem building my app for distribution w webpack:

ERROR in ./~/threads/lib/worker.node/worker.js
Module not found: Error: Cannot resolve module 'child_process' in /theApp/node_modules/threads/lib/worker.node
 @ ./~/threads/lib/worker.node/worker.js 11:21-45

Similar to (?):
webpack/webpack#744

which suggests adding:

browser: { fs: false, child_process: false } to package.json

Thoughts? Thanks!

Hey,

First of all thanks for this great library.

Let's say I need to run a pool with different runners but they share the same imports, for example:

  pool.run((input, done) => {
        importScripts('https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.17.1/moment.min.js');
        ....process one
        done(input);
      }
    ).send({data: 1});

 pool.run((input, done) => {
        importScripts('https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.17.1/moment.min.js');
        ... process two
        done(input);
      }
    ).send({data: 2});

In this case, the browser will download the moment library twice, how can I overcome this problem? thanks!

Not every handler is supposed to call done when they need to return something. I think maybe slave-code.js should call preparedHandlerDone if the handler.call is returning some data. Currently it all depends on the method itself to call done.

Tried to used threads.js and maybe I'm doing something wrong, but:

https://jsfiddle.net/aorx7stg/

in javascript console:

Uncaught SecurityError: Failed to construct 'Worker': Script at 'data:text/javascript;charset=utf-8,/*eslint-env%20worker*/%0A/*global%20imp…ndlerDone,%20handlerProgress.bind(this));%0A%20%20%7D%0A%7D.bind(self);%0A' cannot be accessed from origin 'https://fiddle.jshell.net'.

Another question :)
I would like to have a bidirectionnal communication between processes.
For the slave to main direction, progress function can be used.
What about the main to slave direction ?

Tried to use send method , but it seems to spawn another process id.

Here is a simple use case :
In the slave process, a web server is running (in my case it's a listener on serial port).
At a moment, the main process should update some informations to the web server.
Using send will re-spawn a process, raising a "Port already binded" error.

How to send informations from main process to slave ?

Thanks for helping,

The progress event might not work with thread pools right now. See job.js, line 36.

To Do

• Write test(s)
• Fix job (and maybe pool)

I think I found another one. I run into this with IE9 with some polyfills for URL and WebWorker.

worker.js has this:

  initWorker() {
    try {
      this.worker = new window.Worker(slaveCodeDataUri);
    } catch (error) {
      const slaveScriptUrl = getConfig().fallback.slaveScriptUrl;
      if (slaveScriptUrl) {
        // try using the slave script file instead of the data URI
        this.worker = new window.Worker(slaveCodeDataUri);
      } else {
        // re-throw
        throw error;
      }
    }
  }

It rejects to use the data-url for creating the worker. The catch then reads the slave script URL (which I have set in the config). But it then again uses the slaveCodeDataUri for creating the worker. I believe it should use the slaveScriptUrl at this point.

Make it DRY.

Hi. Can you add typescript definitions?

As start point you can use this:

declare module 'threads' {
    type PostMessage <U> = (data: U) => void;
    type SpawnCallback <T, U> = (data: T, postMessage: PostMessage <U>) => void;
    type ResponseCallback <U> = (response: U) => void;

    class Thread <T, U> {
        public killed: boolean;

        public send (data: T): Thread <T, U>;
        public on (eventType: string, responseCallback: ResponseCallback<U>): Thread <T, U>;
        public kill (): void;
    }

    export function spawn <T, U> (spawnCallback: SpawnCallback <T, U>): Thread <T, U>;
}

I encounter uncaught exception: out of memory errors for jobs in a pool that returns json. The pool handle 21 jobs, but some jobs throw that error randomly. Some returned json are big (8mb and more), but that's not always the biggests that throw the error...

The error occurs only in FF, not in Chrome or IE.

var Pool = require("threads").Pool;

var pool = new Pool();
pool
    .on("finished", function() {
        pool.killAll();
    });

var loadRemoteZippedJson = function(url) {
    var job = pool.run(
        function(url, done) {
            importScripts('./jszip.min.js');
             var xhr = new XMLHttpRequest();
             xhr.open('GET', url, true);
             xhr.responseType = "arraybuffer"
             xhr.onreadystatechange = function() {
                    if (xhr.readyState == 4) {
                        var zip = new JSZip(xhr.response);
                        var c = zip.file(url.split("/")[1]).asText();
                        done(JSON.parse(c), url);
                    }
               }
               xhr.send(null);
         }
    ).send(url);

    job.on("done", function(c, url) {
        doSomething(c, url);
    });
}

Hi,

I would like to thread a class method. Is it possible ?
Here is my non working code :

const threads = require("threads");

class A {
    constructor() {
        let thread = threads.spawn(this.asyncProcess);
        thread.send();
    }

    asyncProcess(input, done) {
        // Do stuff
        done();
    }
}

let a = new A();

And the following error appears :

evalmachine.<anonymous>:1
module.exports = asyncProcess(input, done) {
                                           ^
SyntaxError: Unexpected token {
    at Object.exports.runInNewContext (vm.js:71:16)
    at runAsSandboxedModule (/Users/smizrahi/Documents/hautomation/core-node/node_modules/threads/lib/worker.node/slave.js:35:6)
    at process.<anonymous> (/Users/smizrahi/Documents/hautomation/core-node/node_modules/threads/lib/worker.node/slave.js:72:22)
    at emitTwo (events.js:106:13)
    at process.emit (events.js:191:7)
    at process.nextTick (internal/child_process.js:719:12)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Is it possible to use a class method of an instantiated object in spawn ?

Thanks

Hi @andywer!

An issue actually related with audreyt/node-webworker-threads#99.
I tried both webworker-threads and threads libs to implement multithreading in nogl-shader-output, and both packages get very slow at returning an array from threads. It takes 800ms on my laptop to return 300×300 array from the thread.

Whether something I don’t know or is there a way to speed things up?

Thank you.

Currently when job.send is called, it is emitting readyToRun to wait for a thread, so when calling .promise() on it, as the job has no this.thread, it will throw Cannot return promise, since job is not executed.

.promise should wait until the thread is assigned to the job.

Hello,
Is there a way to capture the stdout of a single worker in the thread pool?

maybe by passing a config function like such:

function(err, stdout, stderr) { 
    console.log(stdout); 
}

My node terminal logging is all mixed together by the different threads, is there a way to fix that?

Cheers!

As the document says, "Threads Uses web workers when run in browsers and child processes when run by node.js", I am able to write a piece of code which runs on the browser(using web worker model) but same code does not work on Node(Server side - using child process).

Note: I could not find enough support for above library in Typescript, I am using the js package in typescript code.

import { spawn } from "threads/lib";
const thread = spawn((value: any,  callback: (_: any) => any ) => { console.log("X: " + value.x + " Y:" + value.y );  callback({ result : value.x + value.y } ); });
thread.on("message", (value: any) => {console.log("Addition: " + value.result); } );
thread.on("error", (message: any) => { console.log("Error: " + message); });
thread.send({x : 1, y : 5});

Above written code works fine while running on the browser(returns 6). But it neither returns anything nor throws any error on the Node.

I have a complete node js setup and I have been using it since a bit long[On windows 10].

Could someone please take a look as why its not working for Node or if I have missed any step in between, or let me know if more details are needed.

Thank you in advance.

I got a use case where I am submitting a large number of similar jobs to a pool.

Problem is, on my machine with 8 cores processing stops after 10 jobs. The reason for this seems to be the way events are handled.

When I submit my 20000 jobs, there are 19992 listeners registered for the threadAvailable event.
When a first job is done and a thread is available, the event emitter removes all listeners (due to primus/eventemitter3#63) and then calls dequeue for all listeners 19992 events. When the event processing is finished, we end up with a single listener for the threadAvailable event. This is because in the event emitter, the event array is cleaned for every iteration of the loop.

When the next job finishes, we have a single listener for the threadAvailable event. Event emitter clears the event listeners for the event and calls dequeue once. Because at this time, a thread is available and the job queue is not empty, there is no listener registered.

Thus, job processing stops at this point.

I am able to work around this issue by calling dequeue when a job is done. But I believe this is just a workaround.

It would be handy to be able to configure the debug listener port used by child threads. Right now child threads attempt to use the same port as the parent thread causing EADDRINUSE exception to be thrown.

When follow the 'basic usage' example to test this module in the browser. I get the error that the module 'threads' is not found. I installed the module using Bower and included it using <script>. Is there something else I need to do to make it available?

Running this code prints done 10 times, but finished does run.

  for (let i = 0; i <= 10; i++) {
        pool
          .run(function(i, done) {
            done(i)
          })
          .send(i)
          .on('done', function (i) {
            console.log('done', i)
          })
          .on('error', function (e) {
            console.error(e)
          })
          .on('finished', function () {
            console.log('finished')
          })
      }

I use the thread pool to process bunch of tasks, but I need to do some cleanup in every worker thread before shutting it down (at least in case when no unexpected errors occurred). As child_process.kill() is used under the hood for pool.killAll(), I cannot use process.on("exit"/"beforeExit") to schedule such cleanup. Currently I could not find a nice way to do so.

I could come up with two a bit hacky solutions:

const pool = new Pool(SIZE).run(__dirname + "/runner");
Promise.all(
  workItems.map(work => pool.send({ work }).promise()
)
  .then(result => {
      return Array.from(new Array(SIZE)).map(() => pool.send({ exit: true }).promise()); // 1
      return pool.threads.map(thread => thread.send({ exit: true }).promise()); // 2
  })
  .then(() => pool.killAll())

• but the solution 1 relies on the way how Pool schedules the work to threads, which is not documented and might change in the future;
• and solution 2 relies on not public api of Pool + breaks in cases when workItems.length < SIZE as it produces No thread logic initialized. errors.

Can you maybe suggest a better way of solving this? Or even supporting a graceful shutdown on the library level? Other option might be to have some api like sendAll(message) in Pool.

It looks like the code is written with ES6 modules but the built version is using babel to transpile down and uses commonjs modules to support that. As people are adopting newer versions of ES, would it be useful to build an ES6 module as well?

See title.