I have seen that there already was an issue on memory leaks (it's #45), and that things should be resolved by returning oboe.drop
as documented here.
Unfortunately, 2.1.1 seems to still have memory issues (or I am getting it absolutely wrong how to use oboe.drop
correctly).
My setup is as follows: I have a server based on Express that delivers an endless JSON array, and I have a client that uses Oboe to stream these data. I measure the memory consumption of this client, and within a few hours it uses hundreds of MBytes, and GC obviously does not clean up as expected.
The server looks like this:
var http = require('http');
var express = require('express');
var app = express();
app.get('/', function (req, res) {
var i = 0;
var sendEventToClient = function (event) {
res.write(JSON.stringify(event) + ',');
};
req.setTimeout(0);
res.setTimeout(0);
res.writeHead(200, {
'content-type': 'application/json'
});
res.write('[');
setInterval(function () {
sendEventToClient({
foo: 'bar',
bar: 'baz',
nufta: 23,
counter: i++
});
}, 10);
});
http.createServer(app).listen(3000);
The client looks like this:
var url = require('url');
var oboe = require('oboe'),
Stethoskop = require('stethoskop');
var stethoskop = new Stethoskop({
from: {
application: 'client.js'
},
to: {
host: 'localhost',
port: 8125
},
enabled: true
});
oboe({
url: url.format({
protocol: 'http',
hostname: 'localhost',
port: 3000,
pathname: '/'
}),
cached: false
}).on('node:!.*', function (event) {
console.log(event);
return oboe.drop;
}).on('fail', function (err) {
console.log('Error!', err.thrown);
});
So, the server sends a new object every 10 ms, and the client should do nothing with it but drop it. I am using stethoskop to measure the client's fitness: It sends the CPU and memory data to a StatsD server.
I have run these two processes for 4.5 hours, and the memory consumption looks like this:
I also tried to run it returning null
instead of oboe.drop
, same result (the left part is the same test as above, the right part is with null
, so basically both options show the very same behavior. The drop to 0 in the middle is not because of GC, it's because I stopped and restarted the processes):
So, to cut a long story short, basically I have two questions:
- Am I doing something wrong in the client? If so, what do I need to correct? Any hints?
- If not, this seems to be a bug in Oboe. Can I do something to help fix it? Pointing out and giving some hints where to look might already be quite helpful :-)
Moreover, the documentation states:
Dropping from an array will result in an array with holes in it. This is intentional so that the array indices from the original JSON are preserved:
My guess is that dropping nodes only works (wrt memory consumption) for objects, but not for arrays. Since I am using an array as outer container here, and since sparse arrays seem to capture more memory than dense ones, this might be the cause of the problem. Please note that I'm not too sure about memory behavior of sparse arrays, so any answer in this direction will be appreciated.