Hi!

I'm currently facing an issue when i use strict with your library.

/node_modules/worker-nodes/lib/pool.js:3
const { TimeoutError, ProcessTerminatedError, MaxConcurrentCallsError } = require('./errors');

test.js

import workerNodes from 'worker-nodes';
const test1 = new workerNodes(path.resolve(__dirname, "test-model"));

test-model.js

import lib from './lib.js';   // **report cant use import,and export**
module.exports =async function(){
};

lib.js

export default {
  test(){
  }
}

Hi, In windows os all promise remain in "pending" code, while in linux it works.

i want use es6 import,class, in worker_module.
how i can do it ?
thx

Thank you so much for your great library! I've been hitting an issue when trying to transfer large JS objects - they seem to be especially slow to encode/decode. I've put together a simple test case to analyze:

https://gist.github.com/jeresig/af003831559a99af63133189e088b4e6

It seems like passing through any JS object ends up being slow - however if we use our own JSON.stringify/JSON.parse and pass through just a string then it ends up being much faster.

We may just encode everything as a big string (non-JS object) and just parse that result, to get optimal speed. Any suggestions you have for improving the performance would be appreciated!

Hi,
i happen to call browserify in a worker node: upon failure it returns an error with a lot of additional structures.
It took me a while to realize I had to rewrite the error using:

const err2 = new Error(err.message);
err2.code = err.code;
err2.stack = err.stack;
reject(err2);

i wonder if there is a way for worker-nodes to avoid the headache.

I didn't understand that workerStopTimeout force kill the process when the time is reached after we call Worker.stop().
I was expecting same behavior than taskTimeout and I was confused between this 2 options.

I think it's more clear to call this option workerForceKillAfterTimeout like execa.

Hi,

Its been few days that I am struggling to use this package on typescript.
Here is an online test implementation: https://repl.it/@JasimKhan/worker-nodes-test

Locally, I am trying to debug it using ts-node, but i dont get any break point in the worker tasks... Nor do I see console traces of their execution.

Can some one point me to some typescript usage of this package?

Thanks in advance.

The following configuration results in spawning 8 workers on my machine:

{
    "autoStart": true,
    "maxWorkers": 4
}

I expect 4 workers to be spawned instead.

Seems that with latest versions of node.js (7.5.0+) this library is not outperforming its competitors as it is previously stated in a benchmark results.

Hey,

We’re using this lib in our service to run worker processes which bundles code with webpack.
following @yurynix PR everything seemed to work fine.

We use this configuration for the pool

const POOL_OPTIONS = {
  minWorkers: 4,
  maxWorkers: 6,
  // Not passed in production, only used for testing.
  autoStart: process.env.POOL_TESTS_AUTO_START !== "false",
  // Not passed in production, only used for testing.
  taskTimeout: process.env.POOL_TESTS_TASK_TIMEOUT ?? seconds(55),
  workerType: "process",
  workerEndurance: 1,
};

After a while, we noticed that our node processes were leaking memory.
It took some time for us to understand that when a process is being killed after it finishes 1 job (workerEndurance: 1 configuration) we don’t close the server connection here. This is causing server socket files to accumulate in os.tmpdir() since the server never closes

I created a fork which fixes the issue (close server connection when child process exists)
I already tested it in production and it works great.

I'll create a PR to fix it in this package so I can continue use it in production.

workerNodes.call : Proxy

This exposes the api of a module that the worker nodes are working on. If the module is a function, you can call this directly. If the module exports multiple functions, you can call them as they were properties of this proxy.

Kind: instance property of WorkerNodes

nothing is mentioned about promises, and the example shows usage of promises

The minWorkers option is only respected when initially reporting pool readiness, but should also be treated as a number of workers that the pool needs to keep running all the time, i.e.:

• on any accidental worker termination, pool should boot a new worker immediately (if minWorkers number is greater than the current number of healthy workers)
• a planned worker exit (for example as a result of WorkerEndurance value exceedance) should result in a spawn of a new worker before starting the process of termination the exhausted one (maybe even at cost of some extra calls above declared endurance).

On circleci, my program crash at onExit callback.
The program work on local environement.

Edit: A worker crash, but I don't know how to get more error output. It's seems like the real error is hidden by node-worker-nodes :/

I use node v8.7.0 and npm v5.5.1

> ERROR: ProcessTerminatedError: cancel after 0 retries!
    at tasks.filter.forEach.task (/home/circleci/project/node_modules/worker-nodes/lib/pool.js:111:39)
    at Array.forEach (<anonymous>)
    at WorkerNodes.handleWorkerExit (/home/circleci/project/node_modules/worker-nodes/lib/pool.js:110:14)
    at Worker.worker.on.exitCode (/home/circleci/project/node_modules/worker-nodes/lib/pool.js:160:44)
    at emitOne (events.js:115:13)
    at Worker.emit (events.js:210:7)
    at WorkerProcess.Worker.process.once.code (/home/circleci/project/node_modules/worker-nodes/lib/worker.js:39:18)
    at Object.onceWrapper (events.js:316:30)
    at emitOne (events.js:115:13)
    at WorkerProcess.emit (events.js:210:7)
    at ChildProcess.WorkerProcess.child.once.code (/home/circleci/project/node_modules/worker-nodes/lib/worker/process.js:42:41)
    at Object.onceWrapper (events.js:318:30)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)

{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: read ECONNRESET
    at _errnoException (util.js:1021:11)
    at Pipe.onread (net.js:608:25) code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
{ Error: This socket has been ended by the other party
    at Socket.writeAfterFIN [as write] (net.js:354:12)
    at Transport.send (/home/circleci/project/node_modules/worker-nodes/lib/worker/transport.js:57:19)
    at Promise.then.result (/home/circleci/project/node_modules/worker-nodes/lib/worker/child-loader.js:36:28)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7) code: 'EPIPE' }

We have the need to initialize some global data structures on our workers before they can start doing work. In addition, the initialization depends on some runtime data known by the parent process.

We'd like to do something like this:

const workerNodes = require("worker-nodes");
const cacheSize = 100;
const workers = new workerNodes(myfile, options, () => initCache(cacheSize));

though that particular implementation probably wouldn't work.

Is such a thing possible? It seems like it should be, by calling handleCall on worker startup, in addition to calling it every time we are starting a new task. But I'm afraid I'm not up to the task of figuring out all the details.

mod.js

module.exports = async function(){
  return new Promise((succ, fail ) => {
      fail();
  })
}

test.js

let worker = ......
worker.then().catch()//not catch it

(node:28649) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): RangeError: Maximum call stack size exceeded

const msg = require('msgpack-lite');
msg.decode(msg.encode({a:undefined})); // { a: null }

See related issue in msgpack-lite repo

Please consider changing the packing library to msgpack-js as the same issue exists in msgpack, where both undefined and null are casted to the same type.

FFMPEg runs in a worker, and in the main a node webserver who use sockets is waiting for instructions.
But the web server can't respond during ffmpeg is running : why ??

Hi 👋
Thank you for the great library! 🙏

When moving from 1.x to 2.x, the implementation of the workers was switched from process (child_process) to threads (worker_threads), which is great for many workloads, however, for some workloads, processes are preferred.

Specifically what bothers me is workloads that do process.chdir() which is not supported inside worker_threads.

Would you be open to a new option in WorkerNodesOptions -> .workerType which can get thread (default) or process,
that will control the underlying implementation?

I don't think it's a big code change, however one downside is that WorkerNodesOptions.resourceLimits is only going to be supported in worker_threads.

What do you think? 🙃

When launching a Node 8 service with the --inspector flag that is passed an IP Address, the following error is printed to the service console when each worker launches:

Unable to open devtools socket: address already in use

I have also sometimes observed this error continuing to print to the service console.

This is down to each worker using the same debugPort as the main service process.

I am creating a Pull Request that will reference this issue and should solve this error, allowing each worker to listen in on a different port.

Hi,
Right now this library is using bson from mongo. there is a problem with the library here.
Bson is using a predefined buffer with max size of 17mb, once this buffer is over node will shut down the process for security reasons without any way for the system to recover from it (no process events, try catch or anything else i could think of).
There aren't many ways to resolve this, bson does not allow us to modify the buffer value which leaves us with two basic options:

1. switch bson with something else, MessagePack can be a viable option (and a simple drop in replacement)
2. leave things the way they are but allow to customize the message serializer (leaving the door open for many other binary serializers, like protobuf).