node-monitor

=====

This is an EC2 monitoring application which uses Node.js and the Amazon CloudWatch API. The application is primarly used to push custom metrics, via plugins, to CLoudWatch -- where we can create alarms for failed services, high memory usage, etc. Older branches pushed data to a NoSQL store + websockets (logs), but these functions are being ported to plugins as well.

Prerequisites

Amazon EC2 account

Setup

Clone the repository.

cd ~/
git clone git://github.com/franklovecchio/node-monitor.git

Copy the script out of the the directory into a usable location.

cp ~/node-monitor/bin/node-monitor.sh ~/node-monitor.sh
chmod a+x ~/node-monitor.sh

To install all the necessary packages (including node.js and npm):

sudo ~/node-monitor.sh install-debian # Debian
sudo ~/node.monitor.sh install-centos # CentOS

Configure AWS credentials/CloudWatch namespace in config/monitor_config:

cloudwatch_namespace=<Namespace Instance Metrics>
AWS_ACCESS_KEY_ID=<ID>
AWS_SECRET_ACCESS_KEY=<Key>

Start the monitor in production mode:

sudo ~/node-monitor.sh start

To create an Upstart job and be verbose (in Ubuntu):

 touch /var/log/node-monitor.log
 cat >> /etc/init/node-monitor.conf <<EOF
 description "node-monitor"
 start on (local-filesystems and net-device-up IFACE=eth0)
 stop on shutdown
 exec sudo -u root sh -c "cd /home/ubuntu/node-monitor/run && /usr/local/bin/node client.js ec2=true debug=false console=true cloudwatch=true >> /var/log/node-monitor.log 2>&1 &"     
 EOF

Plugins

df.sh

To monitor disk size, you need to specify disks in the df_config file:

 /dev/disk0s2
 /dev/sdb

filesize.js

To monitor file sizes (like logs), and empty when they hit a maximum, you need to specify files and their max size (in KB) in the filesize_config file:

 /Users/franklovecchio/Documents/my.log=1024

free.js
health.js
lsof.js
services.js

To monitor running services, you can specify a service name only for a ps -ef command, or a service name and a port for an lsof command:

cassandra
redis=6379

top.js
uptime.js
who.js

There are 2 types of plugins, poll and step. Plugins are modules which have access to a global set of passed modules, and return asynchronously CloudWatch data. A plugin might look like:

 /* lsof.js */

 /**
  * A plugin for monitoring open files.
  */
 var fs = require('fs');

 var Plugin = {
   name: 'lsof',
   command: 'lsof | wc -l',
   type: 'poll'
 };

 this.name = Plugin.name;
 this.type = Plugin.type;

 Plugin.format = function (data) {
   data = data.replace(/^(\s*)((\S+\s*?)*)(\s*)$/, '$2');
   data = data.replace('%', '');
   return data;
 };

 this.poll = function (constants, utilities, logger, callback) {
   self = this;
   self.constants = constants;
   self.utilities = utilities;
   self.logger = logger;

   var exec = require('child_process').exec, child;
   child = exec(Plugin.command, function (error, stdout, stderr) {
     callback(Plugin.name, 'OpenFiles', 'Count', Plugin.format(stdout.toString()), Plugin.format(stdout.toString()));
   });
 };