Binding outer variables to mapped function (without webpack) about parallel.es HOT 4 CLOSED

I see. However, this can not work because of how functions are serialized:

1. bind is a native function. So, the library uses (formatAddressesBound.bind(null, zip)).toString() to serialize the function (I believe it should be formatAddresses?) which returns:

function () { [native code] }

So there is no way this function can be serialized / deserialized.

2. Even if it is not a native function the example would not work. Think about how bind is implemented:

function bind(fn, this, ...args) {
    return function bound() {
        return bind.call(this, ...[...args, ...arguments]);
    }
}

However, the bind call only returns the inner function that makes use of the fn variable in its closure. But closures are not --- and cannot --- be supported when just using runtime serialization.

So the way to go in this case is to pass zip explicitly:

function formatAddress(address, { zip }) {
  const {num, street, city} = address
  return `${num} ${street} ${city} ${zip}`
}

parallel.from(addresses)
  .inEnvironment({ zip: zip })
  .map(formatAddress)
  .then(result => console.log(result))

from parallel.es.

Ok, what you posted does work. I really appreciate all of your help.

I'm struggling to make parallel-es work for my use case - running complicated, CPU-intenstive calculations in node.js. I've tried to convert my code to using parallel-es and am running into problems. For the real calculations, I have probably 10 functions used, some calling others which call others. It would be hard to include them all in inEnvironment.

This isn't my real code, but it's probably along these lines:

const zip = '80305'

// not `inEnvironment`
function getCity() {
  return 'Boulder'
}

function formatRegion(zip) {
  return `${getCity()} ${zip}`
}

function formatAddresses(address, {zip, formatRegion}) {
  const {num, street} = address
  const region = formatRegion(zip)
  return `${num} ${street} ${region}`
}

parallel.from(addresses)
  .inEnvironment({zip, formatRegion})
  .map(formatAddresses)
  .subscribe((subresult, taskIndex) => console.log(`The result of the task ${taskIndex} is`, subresult))
  .then(result => console.log(result))

I could pass in formatRegion into inEnvironment but it depends on other functions, which depend on other functions. Passing all of them in from the top would be really cumbersome. I want it to work like webpack - look up all the functions needed and include it. I guess that's what the webpack plugin does, which I can't use because this is a node.js project.

Also I'm not clear why closures can't be serialized (I haven't tried). Are we bumping into a fundamental limitation of parallelizing JS with that? You also mentioned lodash functions can't be included because how they are defined.

I really want this to work, for this specific case and lots of future functions I write - the goals of your paper great. But after spending time with this getting a little discouraged.

Can you see a way for this to work? If not, I may look elsewhere. There is a gpu.js library, but that requires a lot of careful rewriting. And of course, there are other languages like Clojure that parallelize easily, but that requires a lot more time to shift to that paradigm.

Regardless, I'll still try to help with documentation. Thanks again.

from parallel.es.

I could pass in formatRegion into inEnvironment, but it depends on other functions, which depend on other functions. Passing all of them in from the top would be really cumbersome. I want it to work like webpack - look up all the functions needed and include it. I guess that's what the webpack plugin does, which I can't use because this is a node.js project.

That's the idea behind the webpack plugin. Unfortunately, it is not supported for node. However, Pull requests are always accepted ;)

Also I'm not clear why closures can't be serialized (I haven't tried). Are we bumping into a fundamental limitation of parallelizing JS with that? You also mentioned lodash functions can't be included because how they are defined.

I tried to explain this in the previous answer. Again, use the previous example of bind:

function bind(fn, self, ...args) {
    return function bound() {
        return bind.call(self, ...[...args, ...arguments]);
    }
}

const zip = ...;
const bound = formatAddress.bind(undefined, zip);

Parallel.es uses Function.toString to get the source code of the function (there is no other way to get the source of a function) wich outputs for bound:

function bound() {
  return bind.call(self, ...[...args, ...arguments]);
}

As you can see, it only returns the inner function of bind. However, the values of self, args and arguments are missing in the serialized representation. And here we come to the root problem (as explained in the paper). .toString returns the string representation of a function. However, what we would need is the string representation of the function instance together with its scope that contains the values of self, args and arguments. There is no way in JavaScript to access the scope of a function (this would be a huge security concern!). So, there is no possible way to get what you want at runtime.

However, the example project contains some quite advanced examples that also work without transpiling. The main idea is that you wrap all you needed functions in a single function.

For example, take a look at the monte-carlo implementation.

There are only two functions createMonteCarloEnvironment and calculateProject that wrap other functions. I think this approach is quite acceptable and not that uncommon in JavaScript (to reduce visibility). Create the bounded functions in the inEnvironment function and you can access them in the processing steps.

I hope this helps.
Micha

from parallel.es.

I rewrote my long-running scripts to use inEnvironment. Which is fine if you're writing something from scratch but tedious if you are re-writing something complicated. Also the lack of Lodash is a bummer because in applications (but not libs) I consider it now part of the standard library. I got everything converted but without debugging I was working blind and couldn't make it work. I'm moving my long-running scripts to Clojure.

But I'm really excited about the potential for this library. There is some friction and traps, especially in Node.js, but hopefully that can smoothed out as more people use and contribute to the library. Thanks for making it.

from parallel.es.