Comments (9)
Ok, I figured it out, EMS generates a script called EMSThreadStub.js
:
// Automatically Generated EMS Slave Thread Script
// To edit this file, see ems.js:emsThreadStub()
var ems = require("ems")(parseInt(process.argv[2]));
process.on("message", function(msg) {
eval("func = " + msg.func);
func.apply(null, msg.args);
} );
I'm guessing that process.argv[2] is the reason for those messages.
Is there a way to go around this? (without modifying ems code)
from ems.
Hi,
The short answer is code modifications are required.
To execute EMS barriers and joins, every process needs to know how many processes there are and a way to determine which process started first so initialization only happens once. There is more than one way of solving this, and command-line arguments was a regrettable choice.
This problem also turned up in the Python port, which seems like a good reason to fix this now before Python is released.
I hope to remove command line arguments from EMS this weekend and have a fix by Monday. My apologies for delays getting back to you and implementing a better fork-join scheme.
-Jace
from ems.
Hi,
No apologies are necessary! this is an awesome project, way to go!
May I inquire as to what approach are you going to take to circumvent this?
from ems.
Please update to v1.3.6 (git/npm) which no longer depends on command line arguments and should fix the problem you ran into. That information is now communicated from the master to slave process via environment variables.
I was hoping to rework initialization for both Python and JS but wasn't able to finish this weekend, that version also gets rid of the thread stub program file created by EMS.
Let me know if this patch didn't help and/or there are other assumptions that are causing problems!
from ems.
I might just be paranoid, but environment variables feels like a risky play... perhaps send the information to slaves via stdin?
from ems.
I didn't think there was much risk setting environment variables in the forked processes, was there a particular risk I hadn't considered?
I did experiment with using stdin/out for slave processes but Node detects the slave processes' stdin is not attached to a terminal and a naive implementation did not work. The version of this feature on the Python branch is, in fact, using stdin/out to set up the slave processes, but isn't complete yet. Another constraint is I'd like to keep the Python and Node implementations similar to the extent that's possible so they behave similarly for end-users and for development/debugging.
from ems.
The only "risk" I can think of right now is the fact that the environment might be altered or changed somehow without your control.
Are you just passing msg.func and msg.args?
I did experiment with using stdin/out for slave processes but Node detects the slave processes' stdin is not attached to a terminal
Are you referring to this: https://github.com/SyntheticSemantics/ems/blob/master/nodejs/ems.js#L632 ?
from ems.
The environment variables are per-process in UNIX, they are copied from the parent process to the child process' environment when it is forked, and once executing neither process may see or modify another's environment variables.
The fork at ems.js:633
is creating the child processes whether Bulk Synchronous Parallelism (bsp) or Fork-Join (fj) is being used. For BSP, the script executed is the currently executing script, for FJ the script is the the thread stub declared a few lines above. In both cases, the original command line arguments are passed to the slave process, however the FJ thread stub ignores them.
In the Python branch this fork
is a spawn
. fork
has additional semantics where Node sets up a socket communication channel, the child knows Node forked it, and it does something special with stdin. spawn
is a simple process creation that doesn't have Node specific semantics, so Node behaves as if it were run from a bash script.
The msg.func
and msg.args
are used by fork-join -- they are the actual function code and arguments from the master process to be executed by the thread stub. BSP processes are all running the user's script and start execution at the top.
from ems.
The environment variable instead of command line argument is now working for Python as well, so I'm going to close this since it seems to work as a language independent solution.
v1.4 is now available, which supports object sharing with Python3. Py2 on the way.
from ems.
Related Issues (20)
- Fork-join threading type does not work HOT 8
- potential GCC optimizations based on undefined behaviour HOT 1
- kv_store seems to limit "clusters" module to one core HOT 2
- hello world using python and nodejs HOT 3
- EMS crash with JSON objects saved HOT 2
- unexpected JSON format! HOT 2
- How to free up memory? HOT 2
- Key Deletion and Memory Leak HOT 1
- Clarify what kinds of data can be shared. HOT 1
- NPM Install Fails HOT 4
- typedarrays / buffers? HOT 4
- N-API HOT 3
- Future of EMS HOT 6
- Bug memory leak when used to run custom next.js server. HOT 3
- Assertion `0' failed HOT 6
- Survey Question on the For loop in source file accum_omp.c line 45 HOT 2
- Survey Question on the For loop in source file Tests/accum_omp.c line 52 HOT 2
- Instantiating unbounded EMS array HOT 1
- Killing child threads HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ems.