loganathanb / socket.io-benchmark Goto Github PK

Simple socket.io benchmark.

For now, this project should be used to benchmark socket.io implementation itself, and not projects built using it.


We are building it as a tool to compare WebSocket vs Comet performance, in terms of CPU, Memory and Bandwith utilization.

We choose Socket.IO, because we could run the experiment using WebSocket, and then run it again, using XHR-Polling, or JSON-Polling, by simply changing the available Transports.


There are two tests you can run:
* echo -> each client sends a message to the server, which sends it back to the same client, which repeats the process forever.

* broadcast -> each client sends a message to the server, which broadcasts it to all other clients. The server notify the client if the broadcast went ok, and then the client repeats the process forever.


Installation:
1) 'git clone [email protected]:michetti/socket.io-benchmark.git'
2) on created dir, 'npm install -d'
3) set ulimit 'ulimit -u <desired ulimit>'

* set ulimit to increase max number of opened sockets.

Run:
1)  Start server: 'node app.js <transport>'
    * transport can be omitted.
    * see socket.io docs for available transport
    * only websocket and xhr-polling seens to work...
    * xhr-polling acts oddly under load...

2)  Start clients: 'node benchmark.js <users> <rampup in seconds> <broadcast> <host> <port> &'
    Ex: 'node benchmark.js 120 60' -> one new user every 0.5 seconds, until 120 users are connected.

Broadcast, host and port parameters are optional. If not specified, then echo behaviour will be used, and it will connect to localhost on port 3000.
Different Configu Example: 'node benchmark.js 120 60 broadcast 192.168.1.50 4040'

It is better to repeat step 2 multiple times than run it one time with a lot of users, since node is monothread.

Run 'node responsetime.js' to see current response time. It will connect another user, that will send messages to the server, using the echo behaviour. The cicle time is logged in miliseconds.

Output:
app.js will log the following line each second:
U: 100, MR/S: 2500, MS/S: 2500, MR/S/U: 25, MS/S/U: 25, CPU: 56, Mem: 1.8

U       -> Number of connected users
MR/S    -> Messages received per second
MS/S    -> Messages sended per second
MR/S/U  -> Messages received per second per user
MS/S/U  -> Messages sended per second per user
CPU     -> CPU usage percentage
Mem     -> Memory usage percentage

To verify global resources utilization (by server and multiple clients processes), use 'top', 'htop' (recomended) or something like it.


Debug/Profilling:
Both app.js and benchmark.js have support for v8-profiler/node inspector.

To debug/profile app.js:
1) 'node --debug app.js'
2) 'node inspector &'
3) open the printed url (must be a webkit compatible browser, like chrome)


TODO/Brainstorm:
*) Make benchmark.js fork new processes to run clients, instead of running the script multiple times.
*) Use redis store and create several server processes.


I'm quite new to node, socket.io and benchmarking. Any help is appreciated.

Tested only with node v0.4.12.