mogill / ems Goto Github PK

Hi,

I did some testing using Examples\kv_store.js as a starting point. And even if I was not using any of the ems function (other than initializing it), the cluster module suddenly was not able to do more than one core's worth of cpu, even though I could see it had several slave processes running.

It do seem to be related to the "threadAffinity" option of ems. When I set this to false in the initialization, it seems that 'cluster' recovers its powers.

Isn't it weird to make an example use the node js cluster module, but limit its strength back to where you would be without it. (or in fact worse)

Hi! I was shocked to see today that our docs are broken (images + API Documentation + EMS Website) and EMS is going to die for Node.js. I've rather limited funds and I usually avoid C(++), can I help in some other way? Looking at http://syntheticsemantics.com/ it indeed seems it's used mostly by people who know how to set up proper caching. ;-) Or do people just not need the performance anymore and prefer a broker like AMQP instead?

In your word count example the following line:

var text = fs.readFileSync('/Data/Gutenberg/all/' + dir[bufNum], 'utf8', "r")

should be:

var text = fs.readFileSync('/Data/Gutenberg/all/' + dir[docNum], 'utf8', "r")

After running "npm install ems"

I get a bunch of errors like:
../src/ffi.cc:126:11: error: ‘class v8::Object’ has no member named ‘ForceSet’

Is it possible to delete keys from an EMS array object that uses key-index mapping? Presumably, this would free up memory when certain items are no longer needed. Nodejs 'delete' does not do anything here as far as I can tell.

After working for a while loading data and updating them, with write (), it is giving me the following error, how is it solved?
Thank you

node: ../src/ems_alloc.cc:185: void emsMem_free(emsMem*, size_t): Assertion `0' failed.
Aborted (core dumped)

Hello Sir/ Madam
We are from a research group at Iowa State University, USA. We want to do a survey on Github developers on the methods they used for paralleling their code. To do the survey, We want to ask three questions:

1. Have you ever tried to add pragma for that 'for' loop?.

2. How much confidence do you have about the correctness of this implementation? You can choose from 1-5 with 1 as the lowest confidence score and 5 as the highest confidence.

3. (Optional) Do you actually run (interpret the code with compilation and pass input/get output) the code? Yes/No

• If yes, can you provide the information of what are the input and expected output of this program (the input that caused the program to run through this for-loop).

The for loop is from line 52 of file https:/github.com/mogill/ems/blob/master/Tests/accum_omp.c
Here is a part of the code:

nan
for (i = 0; i < 3200; i++)
{
sum += a[i];
}

Sincerely thanks

Is there a way to exit all threads that are not the master thread? In other words this is what I want to do:

1. Create child threads
2. Do work
3. Exit child threads
4. The main thread continues to run

wondering if it's within reasonable effort to read/write typedarrays/buffers? I can see everything goes through JSON stringify & parse, is this a hard constraint or just convention?

@mogill, are you implying here #20 that there's something inherently wrong with buffers in node or just sharedbuffers? ems far outshines anything possible with master-worker shared mem, would love to hear a few more of your thoughts on all this...

Hello,

ems is initialised in multiprocess environment

if (cluster.isMaster) {
var start = Date.now();
var persistent_data_options = {
dimensions: [100000],
heapSize: [10000000],
useExisting: false,
useMap: true,
setFEtags: 'full',
filename: '/tmp/persistent_shared_web_data.ems'
};
....
}

in other process the useExisting flag is true

I add JSON objects (28 in number with fairly small structure of object) all is well till now.

With 29th object in array I get following error

SyntaxError: Unexpected token `
at Object.parse (native)
at EMSreturnData (/home/telemon/node_modules/ems/nodejs/ems.js:211:25)
at Object.EMSread [as read] (/home/telemon/node_modules/ems/nodejs/ems.js:308:12)

when I try to read it returns a partial object string

{"name":"FSM1","mbAddress":0,"mbSize":17,"` - notice the last quote

with

SyntaxError: Unexpected token `

please let me know how do I resolve this?

setting the threading type to 'fj' prints the following error:

code:

var ems = require('ems')(8, true, 'fj');

console:
node test.js
> EMS: Must declare number of nodes to use. Input:NaN

Hello
Could you help me do the same thing as in the gift inside readme, aka communicate between modems and python?
I'm lost in the documents and I could not find what should I do.
Thanks

Hello Sir/ Madam
We are from a research group at Iowa State University, USA. We want to do a survey on Github developers on the methods they used for paralleling their code. To do the survey, We want to ask three questions about this for loop:

Can you briefly explain the purpose of using pragma for this case? If the pragma contained reduction and private clauses, can you briefly mention the purposes of variables in those clauses?

How much confidence do you have about the correctness of this implementation? You can choose from 1-5 with 1 as the lowest confidence score and 5 as the highest confidence score.

(Optional) Do you actually run (interpret the code with compilation and pass input/get output) the code and see the optimization of parallelization? Yes/No

If yes, can you provide the information of what are the input and expected output of this program (the input that caused the program to run through this for-loop).
The for loop is from line 45 of file
https://github.com/mogill/ems/blob/master/Tests/accum_omp.c

for (i = 0; i < 100000000; i++)
{
int retval = __sync_fetch_and_add(&a[i % 3200], 1);
}

The readme states

EMS operations may performed using any JSON data type, read-modify-write operations may use any combination of JSON data types. like operations on ordinary data.

… but I'm not sure whether I should read that as a limitation, or the guaranteed minimal set that works in all circumstance. For example: Is it possible to share python functions with other python processes, or JS functions from one NodeJS with other NodeJS instances?

N-API port to insulate from engine changes (#20)

Mapped indexes are implemented using a secondary address translation lookup from an ordinary EMS array. As a consequence, 'empty' target data at startup cannot be initialized.

Hi,

This situation was mentioned in #10, here is the code, but this time around I don't have any shared state with the spawning process:

index.js:

const ems = require('ems')(4, false, 'fj')

ems.parallel(function () {
    var x = 1
        console.log(x)
})

process.exit(0)

Running node index.js yields:

1
EMS: Must declare number of nodes to use.  Input:NaN
EMS: Must declare number of nodes to use.  Input:NaN
EMS: Must declare number of nodes to use.  Input:NaN

Running node index.js 4 yields:

1
1
1
1

Node version is v5.5.0

Anything I missed?

Hello,

We are using ems in a multiple process environment using nodjs, this is a small embedded computer which fetches data from multiple ports converts them and sends to another port.

We are randomly getting an error which is a JSON parse error on a saved memory object when we read it using the key. The object is accessed at many places and we are doing just read and write operations on these objects.

Here is the screen shot of the error.

Is this something to do with we are running out of memory? If so how come other operations continue?

By the way what we have in the shared memory is the set of JSON objects (read from a configuration file, some of them have an attribute called value(which is simple type mostly) are of type array of 200 odd integers.

we initialise ems like this (common file sharedMem.js)

`var ems = require('ems')(1, false, 'user');

var persistent_data;

var persistent_data_options = {
dimensions: [50000],
heapSize: [50000000],
useExisting: true,
useMap: true,
setFEtags: 'full',
filename: '/tmp/persistent_shared_web_data.ems'
};

module.exports = {
sharedMem : ems.new(persistent_data_options)
}
`
we use sharedMem in clients

var persistent_data = require('./sharedMem').sharedMem; //somewhere down the line prm = persistent_data.read(currPrm.name); //or write
main module (master since we are using cluster) initialisation is done as follows

var persistent_data_options = { dimensions: [50000], heapSize: [50000000], useExisting: false, useMap: true, setFEtags: 'full', filename: '/tmp/persistent_shared_web_data.ems' };

Here is the error

Any help in this regard will be greatly appreciated.

Regards,
Prasanna

The open-hashing scheme used to resolve key collisions means there is more than one possible place to find a key. Resolving insertion collisions is not atomic, resulting in race hazards when deletions create new places to find a key.

Deletion in EMS is implemented as overwriting the value of the key, the key itself is never freed. Although this makes insertion deterministic, it leaks memory and hash map entries. The key itself is stored on the EMS heap when the key is written, or as a result of a read-modify-write, including modification of tag bits. That is, if the key foo does not already exist, readFF("foo") will store the key foo on the EMS heap, and it is possible to consume all the entries in the hash map by reading keys that have undefined values.

In EMS 1.x the solution is to replace the open hashing scheme with hashing with chaining where collisions are resolved by chaining performed sequentially (ie: while holding a per hash element lock).

In EMS 2.x the internal heap data structure will use both hash lookup and linked lists.

Is it possible to instantiate an EMS array object without specifying it's dimension? I'm at a case where I do not know how big the array will be.

CXX(target) Release/obj.target/ems/ems.o
../ems.cpp:250:5: error: use of undeclared identifier 'nanosleep'
NANOSLEEP;
^
../ems.cpp:155:5: note: expanded from macro 'NANOSLEEP'
nanosleep(&sleep_time, NULL);
^
../ems.cpp:265:5: error: use of undeclared identifier 'nanosleep'
NANOSLEEP;
^
../ems.cpp:155:5: note: expanded from macro 'NANOSLEEP'
nanosleep(&sleep_time, NULL);
^
../ems.cpp:352:7: error: use of undeclared identifier 'nanosleep'
NANOSLEEP;
^
../ems.cpp:155:5: note: expanded from macro 'NANOSLEEP'
nanosleep(&sleep_time, NULL);
^
../ems.cpp:392:32: error: no type named 'Arguments' in namespace 'v8'; did you mean 'v8::internal::Arguments'?
uint64_t EMSreadIndexMap(const v8::Arguments& args)
^~~~~~~~~~~~~
v8::internal::Arguments
/var/folders/mr/cr6pnwvs5578w22byhjcxkgc0000gn/T/.node-gyp/0.12.1/deps/v8/include/v8.h:127:7: note: 'v8::internal::Arguments' declared here
class Arguments;
^
../ems.cpp:394:19: error: calling a protected constructor of class 'v8::HandleScope'
v8::HandleScope scope;
^
/var/folders/mr/cr6pnwvs5578w22byhjcxkgc0000gn/T/.node-gyp/0.12.1/deps/v8/include/v8.h:816:13: note: declared protected here
V8_INLINE HandleScope() {}
^
../ems.cpp:395:3: error: unexpected namespace name 'Buffer': expected expression
EMS_DECL(args);
^
../ems.cpp:325:9: note: expanded from macro 'EMS_DECL'
node::Buffer buffer = node::ObjectWrap::Unwrapnode::Buffer(args.This()->GetHiddenValue(buffer_symbol)->ToObject());
^
../ems.cpp:395:3: error: use of undeclared identifier 'buffer'
../ems.cpp:325:17: note: expanded from macro 'EMS_DECL'
node::Buffer *buffer = node::ObjectWrap::Unwrapnode::Buffer(args.This()->GetHiddenValue(buffer_symbol)->ToObject());
^
../ems.cpp:395:3: error: no member named 'ObjectWrap' in namespace 'node'
EMS_DECL(args);
^~~~~~~~~~~~~~
../ems.cpp:325:32: note: expanded from macro 'EMS_DECL'
node::Buffer *buffer = node::ObjectWrap::Unwrapnode::Buffer(args.This()->GetHiddenValue(buffer_symbol)->ToObject());
~~~~~~^
../ems.cpp:395:3: error: unexpected namespace name 'Buffer': expected expression
../ems.cpp:325:57: note: expanded from macro 'EMS_DECL'
node::Buffer *buffer = node::ObjectWrap::Unwrapnode::Buffer(args.This()->GetHiddenValue(buffer_symbol)->ToObject());
^
../ems.cpp:395:3: error: member access into incomplete type 'const v8::internal::Arguments'
../ems.cpp:325:69: note: expanded from macro 'EMS_DECL'
node::Buffer *buffer = node::ObjectWrap::Unwrapnode::Buffer(args.This()->GetHiddenValue(buffer_symbol)->ToObject());
^
/var/folders/mr/cr6pnwvs5578w22byhjcxkgc0000gn/T/.node-gyp/0.12.1/deps/v8/include/v8.h:127:7: note: forward declaration of 'v8::internal::Arguments'
class Arguments;
^
../ems.cpp:395:3: error: use of undeclared identifier 'buffer'
EMS_DECL(args);
^
../ems.cpp:326:40: note: expanded from macro 'EMS_DECL'
char emsBuf = static_cast<char*>(buffer->handle_->GetIndexedPropertiesExternalArrayData())
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:224:4: note: expanded from macro 'EMSv8toEMStype'
arg->IsInt32() ? EMS_INTEGER :
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:225:4: note: expanded from macro 'EMSv8toEMStype'
arg->IsNumber() ? EMS_FLOAT :
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:226:5: note: expanded from macro 'EMSv8toEMStype'
(arg->IsString() && !stringIsJSON) ? EMS_STRING :
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:227:5: note: expanded from macro 'EMSv8toEMStype'
(arg->IsString() && stringIsJSON) ? EMS_JSON :
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:228:4: note: expanded from macro 'EMSv8toEMStype'
arg->IsBoolean() ? EMS_BOOLEAN :
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:229:4: note: expanded from macro 'EMSv8toEMStype'
arg->IsUndefined() ? EMS_UNDEFINED:
^
../ems.cpp:402:46: error: type 'const v8::internal::Arguments' does not provide a subscript operator
int idxType = EMSv8toEMStype(args[0], false);
~~~~^~
../ems.cpp:230:4: note: expanded from macro 'EMSv8toEMStype'
arg->IsUint32() ? EMS_INTEGER : -1
^
../ems.cpp:409:39: error: type 'const v8::internal::Arguments' does not provide a subscript operator
v8::String::Utf8Value argString(args[0]);
~~~~^~
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Release/obj.target/ems/ems.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:269:23)
gyp ERR! stack at ChildProcess.emit (events.js:110:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1067:12)
gyp ERR! System Darwin 14.0.0
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build"
gyp ERR! cwd /Users/eugene/node-jpeg-turbo-scaler-master/node_modules/ems/ems
gyp ERR! node -v v0.12.1
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok

Hi there, great library.

I have briefly looked into the code, and I have stumbled some potential problem with emsMem struct.

ems struct is declared as :

struct emsMem {
int32_t level;
uint8_t tree[1];
};

but tree is not an array of one byte, it is actually a slice of a bigger array allocated in emsMem_new:

struct emsMem *self = (struct emsMem *) malloc(sizeof(struct emsMem) + sizeof(uint8_t) * (size * 2 - 2));

yet, what GCC see in other functions is basically access to tree as uint8_t(*)[1], for example

self->tree[index] == BUDDY_UNUSED

where index can be any number.

for GCC, this is undefined behavior if index != 0 (and it is, for most cases ) , even though tree is a valid allocated block, but tree is declared as uint8_t[1].

GCC is known to use optimizations that rely on undefined behavior. an improvement is to turn tree from uint8_t[1] to uint8_t*. for GCC, it cannot determine if tree[index] is undefined behavior or not at compile time, thus minimizing the possibility of undefined-behavior based optimizations.

We made a custom server for next.js.

and adopted esm using es import and export syntax.

but esm make built pages each request and not free.

So we adopted babel-core & babel-cli to custom server instead of esm.

I hope you will fight with code without suffering.

Good luck!