This project is no longer in active development.

NAD is obsolete and has been replaced by the circonus-agent.

• Overview
• Installation
  • Automated recommended
  • Manual
    • Windows
  • Source
  • Updating
• Running NAD
  • Command line
  • As a service
  • NAD HTTP interface
• Configuration options
  • General
  • Reverse
  • API
  • SSL
  • Miscellaneous
• StatsD
  • Configuration
  • Group metrics
• Plugins
  • Enable
  • Disable
  • Verify
  • Run State Directory
  • Creating custom plugins
• NAD Development

NAD is a portable, extensible, lightweight metric collection agent. It is the recommended way to collect system metrics for the Circonus monitoring platform.

NAD comes with a rich set of plugins which collect:

• System metrics on Linux, Illumos (Solaris), FreeBSD and OpenBSD
• Application metrics for MySQL, PostgreSQL, HAProxy, Cassandra and more

Further, applications can be easily added using a simple but powerful plugin system. We welcome further contributions by the community. Just submit a pull request against this repository.

NAD Features:

• Simple HTTP interface for metric collection
• Metrics exposed in easy to parse JSON format
• Supports SSL for securing HTTP interface
• Full support for histogram metrics
• Support for Circonus real-time (1s) dashboards and graphing
• Provides local StatsD interface for application metric submission
• Multiple data submission paradigms:
  • pull - Circonus collects metrics using HTTP interface
  • reverse - Function behind NAT. NAD initiates secure TCP connection, Circonus uses connection to collect metrics
• Self-configure with Circonus via the command line with a user-provided JSON configuration file

The easiest, and recommended, method to install NAD for Linux and Illumos based systems is via the Circonus one-step Installer (COSI). See the COSI documentation for details.

Benefits of using COSI:

• one command install (fully automated, install NAD, create checks, graphs, worksheets and dashboards)
• or download the install script and customize to your needs
• customizable templates for checks, graphs, dashboards and worksheets
• supports automation via orchestration systems (e.g. ansible, puppet, shell scripts, etc.)
• cosi-site can be installed and run locally for complete control

For convenience and flexibility, pre-built packages are available for selected platforms from updates.circonus.net. These are self-contained omnibus packages built for the target OS. Packages contain the correct version of NodeJS, binaries for platform-specific plugins, and applicable service configuration. These packages will install NAD in /opt/circonus/nad and configure and start NAD as a service.

At the time of this writing, these are:

• deb packages - Ubuntu: 14.04, 16.04
• rpm packages - CentOS: EL6, EL7

An up-to-date list of currently supported platforms is available from COSI (list returned in JSON).

1. Download from NAD repository releases
2. Unpack NAD file downloaded from releases
3. Ensure node v6+ installed
4. Change to directory where NAD was unpacked
5. Create the default plugin directory etc/node-agent.d e.g. mkdir etc\node-agent.d
6. Run npm install npm install
7. Create nad directory in node_modules e.g. mkdir node_modules\nad
8. Copy lib/* into node_modules/nad e.g. xcopy lib\*.* node_modules\nad /s /e
9. Run NAD node sbin\nad.js

For RPM or DEB based installs, go to updates.circonus.net and retrieve most recent package.

curl -O http://updates.circonus.net/node-agent/packages/<name of rpm file>
rpm -Uvh <name of rpm file>

curl -O http://updates.circonus.net/node-agent/packages/<name of deb file>
dpkg -i <name of deb file>

For pkg based systems (e.g. OmniOS) run pkg update field/nad.

• NodeJS v4.4.5+ must be installed, node and npm available in the PATH.
• A basic development environment (compiler, GNU make, etc.) in order to build certain plugins. For OmniOS/FreeBSD/etc. you must install and use gmake.

A basic install from source installs NAD in /opt/circonus/nad.

git clone https://github.com/circonus-labs/nad.git
cd nad
sudo make install

git clone https://github.com/circonus-labs/nad.git
cd nad
sudo gmake install

In addition to the basic install target, there are OS-specific installation targets. Which will build certain plugins for the specific OS platform, enable default plugins, and install an OS-specific service configuration.

• make install-ubuntu
• make install-rhel
• gmake install-illumos
• gmake install-freebsd
• gmake install-openbsd

/opt/circonus/nad/sbin/nad [options]

• Systemd based systems - CentOS 7.x and Ubuntu 16.04
  • Configuration: /lib/systemd/system/nad.service
  • Enable: systemctl enable nad
  • Start: systemctl start nad
• Upstart based systems - CentOS 6.x and Ubuntu 14.04
  • Configuration: /etc/init/nad.conf
  • Enable: presence of configuration
  • Start: initctl start nad
• SMF based systems - OmniOS/Illumos/etc.
  • Configuration: /var/svc/manifest/network/circonus/nad.xml
  • Enable: svccfg import /var/svc/manifest/network/circonus/nad.xml
  • Start: svcadm enable nad
• FreeBSD
  • Configuration: /etc/rc.d/nad
  • Enable: add nad_enable="YES" to /etc/rc.conf
  • Start: service nad start
• OpenBSD - manual service configuration/installation required

  For example, add the following to your /etc/rc.local:

  if [ -x /opt/circonus/nad/sbin/nad ]; then
     echo -n ' nad'
     /opt/circonus/nad/sbin/nad --daemon --syslog
  fi

  Will start NAD and redirect logging to syslog via the logger command. To redirect logging to a file or elsewhere, replace the --syslog option with redirection e.g. > /tmp/my.log 2>&1.

NAD exposes an HTTP endpoint, the default is to listen to TCP:2609 (e.g. curl http://127.0.0.1:2609/). The output from all requests is JSON.

Options are configured in /opt/circonus/nad/etc/nad.conf. Options can be set using their individual environment variables or added to a single NAD_OPTS variable (for backwards compatibility).

Is used by both Reverse and Self-configure.

1. Reverse will use it to search for a check if a cid is not provided and cosi was not used to setup the host.
2. Self-configure will use it to configure the check on the broker - it is the host (IP or FQDN) the broker will connect to in order to pull metrics.

Set up reverse connection for metric collection.

If the host was registered with COSI then the only required parameter is --reverse, the rest of the information will be retrieved from the COSI configuration.

If the host was not registered with COSI then a valid API Token Key must be supplied. If an explicit Check Bundle ID is supplied, NAD will use the check if it is still active. If no Check Bundle ID is supplied, NAD will search for a json:nad check where the check target matches the supplied (or default) --target.

• --reverse flag signals nad to setup a reverse connection to the broker.
• --api-key - optional if cosi configuration exists on host, otherwise, api key is required.

• --cid - will pull from cosi configuration, if available.
• --target - to enable searching for a check (e.g. on a host not registered by cosi).

DEPRECATED -- use of COSI is recommended.

Providing an API token key without the reverse flag will initiate a self-configuration attempt.

• --api-key
• --target - Note: environment variable NAD_REVERSE_TARGET is not used for self-configure, --target must be specified on command line.
• --brokerid
• --configfile

• --hostname

NAD-StatsD (nad-statsd) provides support for applications on the local system to send metrics using the StatsD line protocol <metricname>:<value>|<type>. The core StatsD metric types (c, g, s, ms) are supported, as well as, additional types specific to Circonus.

• h histogram, treated exactly the same as timing (ms) metrics.
• t text metrics.

Additionally, nad-statsd does not automatically generate derivative metrics from timings since they are represented as histograms in Circonus offering much more flexibility for analysis.

Note: nad-statsd uses a push method for submitting metrics to Circonus, as such, it is not fully compatible with real-time graphing (graphs will update as metrics are received rather than at the normal one second cadence).

Place configuration options in a file, the default is /opt/circonus/nad/etc/statsd.json. If configuration saved to a different location, use either the NAD --statsd-config command line option or set NAD_STATSD_CONFIG in nad.conf or environment to indicate where the configuration file is located. e.g. nad --statsd-config=/etc/project_configs/nad_statsd.json.

The default nad-statsd configuration is:

{
    "servers": [
        {
            "server": "udp",
            "address": "127.0.0.1",
            "port": 8125
        }
    ],
    "flush_interval": 10000,
    "group_check_id": null,
    "group_key": "group.",
    "group_counter_op": "sum",
    "group_gauge_op": "average",
    "group_set_op": "sum",
    "host_key": null,
    "host_category": "statsd",
    "send_process_stats": true    
}

• servers - array of objects - is the same as the StatsD servers list
• flush_interval - milliseconds - is the same as the StatsD flushInterval
• group_check_id - /^[0-9]+$/ - the ID (numeric portion of CID) for the group check, default is to retrieve from COSI installation
• group_key - string - metrics prefixed with this key will be sent to the group check (if enabled)
• group_counter_op - string - sum or average group counter metrics
• group_gauge_op - string - sum or average group gauge metrics
• group_set_op - string - sum or average group set metrics
• host_key - string - metrics prefixed with this key will be sent to NAD. Show up in Circonus in the host check
• host_category - string - the category to hold the host metrics e.g. with the default host_category of 'statsd', a metric named 'my_metric' would appear in Circonus as 'statsd`my_metric'
• send_process_stats - boolean - send nad-statsd module's processing statistics

The nad-statsd module can bifurcate metrics - sending some metrics to NAD (host) and some to a group check (intended to be used by multiple hosts - e.g. a group of web servers). If the --group parameter is provided to COSI when the system is registered, it will create a group check (or use the existing group check if one has already been created from another system registration with the same --group). Additional systems which use the same --group parameter when COSI registers them will also send group metrics to the same group check. This allows application metrics to go to either the host, the group, or both - providing more flexibility in viewing, aggregating and analytics. If an HTTPTrap group check is manually created using the UI, set group_check_id in the nad-statsd configuration file. Additionally, if an HTTPTrap check is created manually it must have asynchronous set to disabled, in the Advanced Configuration section, in order for metrics submitted by systems in the group to be handled correctly.

The host_key and group_key are metric name prefixes which determine the disposition of a given metric. The host and group key prefix is removed from the metric name when it is submitted to Circonus.

Note: If a group check is not enabled and a group_key is configured -- all metrics destined for group, prefixed with the group_key, will be ignored.

NAD plugins are located in the plugin directory (default: /opt/circonus/nad/etc/node-agent.d, configurable with --plugin-dir option). If the automated or manual install were used, platform specific plugins for the current OS are already built. If the source installation method was used - change to the appropriate directory for the current OS and run make or gmake to build the platform specific plugins for the OS. (e.g. cd /opt/circonus/nad/etc/node-agent.d/linux && make)

When NAD starts it scans the plugin directory for plugins to enable. Rudimentary filters are used to determine what is a plugin and what is not. e.g. entry is not a directory, entry has a name in the format name.ext, entry is executable, entry is not a configuration file (extension of .json or .conf), etc. It is recommended that plugins be stored in subdirectories of the plugin directory. Subdirectories are not scanned, those plugins will not be loaded and enabled without an additional step.

To enable a plugin, create a symlink in the plugin directory. For example:

cd /opt/circonus/nad/etc/node-agent.d  # change to plugin directory
ln -s linux/vm.sh .                    # create symlink

The plugin will be automatically found and loaded if file watching is enabled (the default). If file watching is disabled (--no-watch), send a SIGHUP to the NAD process to trigger scanning for plugins.

To disable a plugin, delete the symlink in the plugin directory. For example:

cd /opt/circonus/nad/etc/node-agent.d  # change to plugin directory
rm vm.sh                               # delete symlink

The plugin will automatically be purged from the loaded plugins if file watching is enabled (the default). If file watching is disabled (--no-watch), send a SIGHUP to the NAD process to trigger scanning for plugins.

The output from a plugin can be verified/inspected at any time by making a request for that specific plugin:

curl http://localhost:2609/run/name

where name is the name of the plugin without the extension. NAD will respond with the metrics from that plugin in JSON format.

The currently loaded plugin inventory can be seen by making a request to the inventory endpoint.

curl http://localhost:2609/inventory

NAD will respond with a list of the currently loaded plugins. The inventory endpoint supports one argument, ?full, which includes additional details on each plugin. The output of the inventory endpoint is JSON, enabling it to be used by orchestration and monitoring tooling.

Installation will create a run-state directory in the application directory that should be writable by the non-privileged user that NAD runs as. Plugins can use this directory as scratch space to save local state, such as caching the output of an expensive command or query.

The default location is $PREFIX/nad/var/run.

For information on creating custom plugins see the Plugin section of DEVELOPMENT.md.