If we enable trace, with appmetrics, and run ghost v0.7.1 application, it through exception repeately:

Unhandled rejection TypeError: Cannot read property 'clients' of undefined
    at renderIndex (/opt/node-v0.10.33/sample/ghost_v0.7.1/core/server/controllers/admin.js:22:72)
    at tryCatcher (/opt/node-v0.10.33/sample/ghost_v0.7.1/node_modules/bluebird/js/main/util.js:24:31)
    at Promise._settlePromiseFromHandler (/opt/node-v0.10.33/sample/ghost_v0.7.1/node_modules/bluebird/js/main/promise.js:454:31)
    at Promise._settlePromiseAt (/opt/node-v0.10.33/sample/ghost_v0.7.1/node_modules/bluebird/js/main/promise.js:530:18)
    at Promise.b (domain.js:183:18)
    at Async._drainQueue (/opt/node-v0.10.33/sample/ghost_v0.7.1/node_modules/bluebird/js/main/async.js:182:12)
    at Async._drainQueues (/opt/node-v0.10.33/sample/ghost_v0.7.1/node_modules/bluebird/js/main/async.js:187:10)
    at Async.drainQueues (/opt/node-v0.10.33/sample/ghost_v0.7.1/node_modules/bluebird/js/main/async.js:15:14)
    at process._tickDomainCallback (node.js:463:13)

Ghost v0.7.1:
https://ghost.org/download/

To run ghost application, you must change "127.0.0.1" or "localhost" to your IP, otherwise it cannot be accessed outside this machine.

The link for IBM Monitoring and Diagnostics Tools - Health Center client in the first section of README.md is broken. It has extraneous "**" on the end of the link.

My team uses postgresql as our database, but we'd love to give this package a try. Any chance you'd accept a PR for postgresql support?

Since much of the community has moved over to io.js, does this project work on it?

The set of modules that are instrumented and monitored is currently limited. With the huge growth of interest and usage of "Internet of Things" and MQTT as a messaging protocol, adding MQTT seems like a reasonable module to provide support for.

Testcase:

var assert = require('assert');
require('appmetrics').monitor().on('memory', function(data) {
  assert(data.physical_used === -1 || data.physical_used >= 0,
           'Memory data message does not contain a valid used physical memory ('
            + data.physical_used + ')');
});
setTimeout(function() {}, 3000);

Output:

assert.js:89
  throw new assert.AssertionError({
  ^
AssertionError: Memory data message does not contain a valid used physical memory (-4115607553)
    at API.<anonymous> ([eval]:1:80)
    at emitOne (events.js:77:13)
    at API.emit (events.js:169:7)
    at API.formatMemory (.../test/node_modules/appmetrics/appmetrics-api.js:125:14)
    at API.raiseEvent (.../test/node_modules/appmetrics/appmetrics-api.js:40:17)
    at events (.../test/node_modules/appmetrics/appmetrics-api.js:178:9)

Currently this is mentioned in an example, but not documented as an event.
It should also be mentioned in the getEnvironment() description that it should not be called prior to the 'initialized' event.

I installed appmetrics with Node.js v4.2.2:

npm install -g appmetrics

result:

[root@xxxx node_modules]# pwd
/root/Documents/node4/lib/node_modules
[root@xxxx node_modules]# ls
appmetrics  npm

Then add require("appmetrics") in my application entry, and start my application. It cannot find the global appmetrics:

[root@xxxx acmeair-webapp-nodejs]# /root/Documents/node4/bin/node app
module.js:339
    throw err;
    ^

Error: Cannot find module 'appmetrics'
    at Function.Module._resolveFilename (module.js:337:15)
    at Function.Module._load (module.js:287:25)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
    at Object.<anonymous> (/root/NodeApps/acmeair-webapp-nodejs/app.js:4:1)
    at Module._compile (module.js:435:26)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Function.Module.runMain (module.js:467:10)

Then I use:require("/root/Documents/node4/lib/node_modules/appmetrics"), it throw Module did not self-register error

[root@xxxx acmeair-webapp-nodejs]# /root/Documents/node4/bin/node app
module.js:460
  return process.dlopen(module, path._makeLong(filename));
                 ^

Error: Module did not self-register.
    at Error (native)
    at Object.Module._extensions..node (module.js:460:18)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
    at Object.<anonymous> (/root/Documents/node4/lib/node_modules/appmetrics/index.js:25:13)
    at Module._compile (module.js:435:26)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)

Mac OS is a Node.js platform that we would like to support.

In pr #7 we added the changes necessary to get appmetrics to compile on Mac OS, however, it will not run without a loader and core agent compiled for Mac OS.

We need to work with the IBM team that produces the HealthCenter core agent (which appmetrics consumes) to add support for Mac OS.

@sjanuary I should have picked this up during the PR review. In the following:
https://github.com/RuntimeTools/appmetrics/blob/master/probes/mqlight-probe.js#L130

the request is always set to being a root (ie, the start of a new request). That should only apply to messages received on a subscribed topic - outbound published messages may form part of a request but won't be the start of one.

Can you add a check for the method type and set true/false as appropriate?

This should describe the use of branches, tags and milestones/versions in the package.json and any license files.

The profiling plugin does not correctly escape method names (and probably script resource names) for CSV. These need to double-quote strings containing commas or double-quotes (and also backslash escape double-quotes in the value).

Example of problematic message content:

NodeProfData,Start,1448976339186
...
NodeProfData,Node,238,233,,RegExp: [,]+,0,1
...
NodeProfData,End

This affects both HealthCenter (MQTT) which send malformed CSV, and the event API which reports incorrect values for method, line number and sample count (eg: for the above, the method name would be "RegExp: [", the line number would be "]+" and sample count "0".)

The markdown needs spaces at the end of the 1.0.3 line to make a line break.

The README.md should describe (but currently doesn't) the keys / values available in the object returned by getEnvironment().

Provide the V8 max heap value in the node environment plugin data source. Since the max heap size will not change it should be fine to provide it once in the node environment data.

The duration value in events is currently millisecond based, generated using Date.now(). This isn't particularly useful for short (sub-millisecond) events, so we should move to using a more accurate timer, eg. process.hrtime();

This should include building from a clone of the github repository and installing directly from github with npm.

Summary
Clone from github

git clone https://github.com/RuntimeTools/appmetrics

You now have latest dev code. To change to the latest release:

git checkout latest-release

To build inside the source tree (without downloading Health Center core binaries):

npm install node-gyp [email protected]
node_modules/.bin/node-gyp configure rebuild

To build and install (and run install scripts), go up a level and install the project directory:

cd ..
npm install ./appmetrics

Install direct from github
Latest release: npm install git+https://github.com/RuntimeTools/appmetrics#latest-release
Latest dev: npm install git+https://github.com/RuntimeTools/appmetrics

Add appmetrics probe support for the redis npm module.

The data produced would be:

Event: 'redis'

Emitted when a Redis client sends commands to a Redis server emit a 'redis' event and create a request.

Event:

• data (Object) the data from the redis event:
  • time (Number) the time in milliseconds when the Redis event occurred. This can be converted to a Date using new Date(data.time)
  • args (Array) A truncated version of the arguments to the Redis command. No more than 10 arguments are returned and string arguments are truncated. Callbacks are not included in the arguments.
  • cmd (String) the Redis command sent to the server or 'batch.exec'/'multi.exec' for groups of command sent using batch/multi calls.
  • duration (Number) the time taken in milliseconds.

Request:

• type: 'redis' - a request with type: 'redis' is started when a call is made to Redis.
• root request: false - Redis requests are started by the running nodejs program not triggered by external events so they are children of a request triggered from outside.
• context - the context object will contain:
  • cmd (String) the Redis command sent to the server or 'batch.exec'/'multi.exec' for groups of command sent using batch/multi calls.
  • args:  (Array) A truncated version of the arguments to the Redis command. No more than 10 arguments are returned and string arguments are truncated. Callbacks are not included in the arguments.

Testcase:

• Connect to redis server and runs several basic commands.
• Create batch and multi objects and runs those.
• Create a publisher and subscriber test.
• Count the correct number of events have been received and that call backs passed to those calls were run correctly.

I'd like to add support for the MQLight npm module.

The data produced would be:

Event: 'mqlight'

Emitted when a MQLight message is sent or received.

• data (Object) the data from the MQLight event:
  • time (Number) the time in milliseconds when the MQLight event occurred. This can be converted to a Date using new Date(data.time)
  • clientid (String) the id of the client
  • data (String) the data sent if a 'send' or 'message', undefined for other calls. Truncated if longer than 25 characters.
  • method (String) the name of the call or event (will be one of 'send' or 'message')
  • topic (String) the topic on which a message is sent/received
  • qos (Number) the QoS level for a 'send' call, undefined if not set
  • duration (Number) the time taken in milliseconds.

I'm keeping a track of typos and minor errors in the readme in this issue:

• "or further information on the development process got to the appmetrics wiki." -- "got" should be "go"

When using Express with router level middleware, the req.url value at the start of the request (which the user makes requests of) is modified to remove the context root / mount URL before the request completes.

As the HTTP event currently uses the req.url value that's present at the end of the request, it doesn't match the external URL the request is made against.

Using the actual hosted URL makes more sense, especially as the filters configuration can be used to modify the URL to something else if required.

The 'request' capability can optionally create stack traces for any monitoring event in the request that exceeds a given duration. The default is currently set to 0ms, so stack traces are created for every monitoring event, which is very expensive.

This either needs to be disabled by default, or set to a reasonable default threshold value.

When running the following in REPL with the 1.0.2 development line:

> node
> var appmetrics = require('appmetrics');

The following TypeError is seen:

TypeError: Cannot read property '1' of null
    at getRootModuleDir (C:\Users\tunniclm\workspaces\jtctools\runtime.tools\build\Release\deploy\internal\node_modules\appmetrics\probes\trace-probe.js:187:30)
    at Object.<anonymous> (C:\Users\tunniclm\workspaces\jtctools\runtime.tools\build\Release\deploy\internal\node_modules\appmetrics\probes\trace-probe.js:198:21)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at C:\Users\tunniclm\workspaces\jtctools\runtime.tools\build\Release\deploy\internal\node_modules\appmetrics\index.js:44:25
    at Array.forEach (native)

This is caused by the redundant/unused getRootModuleDir() function in probes\trace-probe.js, which can be removed.

Crashes with stacks like the following:

Example failure on Linux ppc64le:

#0  0x0000000010a26928 in v8::internal::TemplateHashMapImpl<v8::internal::FreeStoreAllocationPolicy>::LookupOrInsert(void*, unsigned int, v8::internal::FreeStoreAllocationPolicy) [clone .isra.110] ()
#1  0x0000000010a2b0fc in v8::internal::ProfileTree::AddPathFromEnd(v8::internal::Vector<v8::internal::CodeEntry*> const&, int) ()
#2  0x0000000010a2b7a0 in v8::internal::CpuProfilesCollection::AddPathToCurrentProfiles(v8::base::TimeTicks, v8::internal::Vector<v8::internal::CodeEntry*> const&, int) ()
#3  0x0000000010a2bb94 in v8::internal::ProfileGenerator::RecordTickSample(v8::internal::TickSample const&) ()
#4  0x000000001071001c in v8::internal::ProfilerEventsProcessor::ProcessOneSample() ()
#5  0x0000000010710108 in v8::internal::ProfilerEventsProcessor::Run() ()
#6  0x0000000010ec6264 in v8::base::ThreadEntry(void*) ()
#7  0x00003fff826d89d8 in start_thread (arg=0x3fff8023f180)
#8  0x00003fff8262ef00 in clone ()

Example failure on Linux x64:

#0  0x0000000000c681ba in v8::internal::TemplateHashMapImpl<v8::internal::FreeStoreAllocationPolicy>::LookupOrInsert ()
#1  0x0000000000c6b5c9 in v8::internal::ProfileTree::AddPathFromEnd(v8::internal::Vector<v8::internal::CodeEntry*> const&, int) ()
#2  0x0000000000c6bb64 in v8::internal::CpuProfilesCollection::AddPathToCurrentProfiles(v8::base::TimeTicks, v8::internal::Vector<v8::internal::CodeEntry*> const&, int) ()
#3  0x0000000000c6be5f in v8::internal::ProfileGenerator::RecordTickSample(v8::internal::TickSample const&) ()
#4  0x0000000000a2e040 in v8::internal::ProfilerEventsProcessor::ProcessOneSample() ()
#5  0x0000000000a2e0d8 in v8::internal::ProfilerEventsProcessor::Run() ()
#6  0x0000000000fc2650 in v8::base::ThreadEntry(void*) ()
#7  0x00000032f90079d1 in start_thread () 
#8  0x00000032f8ce89dd in clone () 

Currently there's no mechanism provided to instrument constructors. This is useful as some modules provide access to methods only through constructed objects.

Examples are socket.io, which has the following construct:

var io = require('socket.io')();
io.sockets.emit('broadcast_event');

Here the emit function is only accessible through the sockets field of the constructed io object.

There is a similar pattern for leveldown where all the functions hang off the constructed leveldown instance.

The aspects patching framework allows you to specify an array of function names to patch. eg. for aspect.around:

var functionNames = [];
aspect.around(target.prototype, functionNames,
    function(target, args) {},
    function(target. args, rc) {}
);

This means you can patch multiple functions in the same way with the same code, but there's no way of knowing the name of the actual function that is running. This is useful if you want to add it to the raised monitoring event.

Passing the function name through to the function callbacks as an additional (last) parameter provides access to the name of the function without breaking existing code, eg:

var functionNames = [];
aspect.around(target.prototype, functionNames,
    function(target, args, funcName) {},
    function(target. args, rc, funcName) {}
);

Hey all,

I was looking through the licenses of appmetrics and noticed that things get complicated when you want to use the Eclipse plugin. What are your thoughts on releasing this without the IBM integration to keep the licensing simpler to just only Apache 2.0? What about a forked version not including the IBM/Eclipse integration?

Cheers,

Zach

the Health Center gui currrently displays the appmetrics version as the "Health Center agent version" which is basically the agent core version. I would like to have the correct appmetrics version shown, e.g. 1.0.4

I'd like to add support for the LevelDOWN npm module.

Iterator

Currently, the probe can monitor calls to put(), get(), del() and batch(). An iterator object can be made using Leveldown.iterator which can query the database using next() and seek(), so support for these calls should be considered.

The current implementation of aspectLvldownMethod() only supports methods which require a callback to be passed as an argument. Iterator.seek() doesn't require a callback as an argument and so this implementation will need to be changed to cater to this

aspect.afterConstructor()

Currently the implementation of prototype.attach() wraps the leveldown target in a new function as it uses a constructor. The afterConstructor() function has been added to the aspect API which replicates this, so it'd be best to change the implementation of attach() to use afterConstructor()

The prototype can be found in my fork of appmetrics

Emitting arguments

The data emitted by the probe's implementation of metricsEnd() currently emits:
- Time of the call
- the method being called
- the duration of the call

This should also emit the methodArgs variable in some form

We should add a page on the wiki where we can document any problems with particular versions of Node Application Metrics. This should complement the Troubleshooting section in the README and corresponding Troubleshooting page on the wiki.

A few examples:
Limitation in appmetrics-1.0.1: Mac OS X system CPU usage is not collected, will read 0
Known issue in appmetrics-1.0.1: Under some circumstances with io.js v2.5 on Mac OS X there is a warning printed and the command line is not collected, will read as blank

Profiling the overhead of appmetrics with the AcmeAir benchmark, and visualizing the data in the Health Center client, shows that 2.47% of CPU is being spent executing HttpProbe.filterUrl() and the functions it calls:

HttpProbe.filterUrl() itself is only using 0.021% (Self %) and the functions it calls account for the rest. The called methods shows that almost all of this time is spent executing Url.parse().

Whilst Url.parse() can parse the full range of URL types and break it down into constituent parts, the use case here is much more limited - we're just taking the req.url value from the HTTP request (which don't contain hostname/port etc), and stripping off any query on the end.

This can be done much more cheaply using a custom implementation rather than using Url.parse(), ie. stripping the URL at the first instance of '#' or '?'.

Document new features (Node v4 support) and any limitations/notes (eg runtime library compatibility changes).

Several probes are storing request data and timer data in global variables which means that multiple events can end up overwriting each others data.

"When installed globally you can access monitoring data via the Health Center client (but not the API) by using the node-hc command-line utility (see *The node-hc command)."

That '*' is an unmatched emphasis token.

The appmetrics loader code needs to be refactored and decoupled from the HealthCenter core agent so it can be added to the repository.

The goal is for it to be possible to build the Node.js addon (appmetrics.node) from the github source.

socket.io is one of the most used packages for interactive/real time client applications, with over 2,000,000 downloads and a month, so a use case we should definitely be supporting.

I've create a prototype in my public fork: https://github.com/seabaylea/appmetrics/tree/socket.io

This does very basic data collection only at the moment. It instruments the following:
io.emit() a broadcast to all clients
socket.emit() an emit to a single client
socket.on('event') a receive of an event from a client

For these events it produces the following data:

Event: 'socket.io'

Emitted when a WebSocket data is sent or received by the application using socket.io.

• data (Object) the data from the socket.io request:
  • time (Number) the milliseconds when the event occurred. This can be converted to a Date using new Date(data.time).
  • method (String) whether the data is for a broadcast or emit from the application, or a receive from a client .
  • event (String) the name used for the event.
  • duration (Number) the time taken for event to be sent or for a received event to be handled.

ie. for the moment it only produces the 'method' of broadcast, emit or receive and provides the name of the event (along with the standard time and duration information).

The readme needs the following updates

1. "https://www-01.ibm.com" should be changed to "https://www.ibm.com"
2. "To use the monitoring to register callbacks and request information" - not sure what "monitoring" is referring to here - should it be "the monitoring object"?

The mysql and mongo probes generate the context data for requests in requestStart() and requestEnd(). This only needs to be done once per request, the second context object will overwrite the first and both generate the same data in both cases.
As this involves making the relatively expensive call to JSON.stringify() we should remove the extra calls to improve performance.

The environment plugin (src/plugins/node/env/nodeenvplugin.cpp) message uses the field runtime.name to report the variation of the SDK being monitored.

It currently reports either:

1. IBM SDK for Node.js if monitoring the IBM SDK for Node.js
2. Node.js otherwise

Ideally it should report io.js if monitoring io.js.

Some links (for example, related to installing, developing and contributing) are shallow and go to the wiki index page, rather than deep links directly to the relevant pages.

These should be changed to be more like the deep link to the Troubleshooting page (which links both to the wiki and also to the specific page.)

Memcached is one of the more popular datastores used by Node.js applications. The 'memcached' module currently has > 150K downloads a month, so it seems reasonable to provide support for monitoring the module.

You can get appmetrics from 3 different places:

1. npmjs.org
2. Github
3. IBM SDK for Node.js

We should document the differences between these (slightly) different packages. These differences center around how the the native libraries are obtained.

Summary

For (1) all native libraries for your platform+SDK version are downloaded (in case you don't have a compiler).
For (2) the appmetrics native libraries are compiled from source (compiler required) and the Health Center core agent libraries for your platform+SDK version are downloaded.
For (3) the native libraries (along with the rest of appmetrics) come preinstalled in the SDK.

This has knock-on effects to the package.json, associated install scripts and licenses.

Currently its expected that each of the probes will set and check whether the __ddProbeAttached__ value is set on a module to denote that its already been probed.

This should be moved to the probe handler, both to remove complexity from the probes and reduce the risk that a probe implementor will fail to do the check properly.

Probes will still need to set the value themselves as probes that return constructors need to run every time and therefore should not set the value at all.

It appears you guys don't support 4.1.x. Please update the module for those using the newer versions of Node.js.

Unsupported version v4.1.1. Exiting.

When trace events are present in the request data, rather than having a context object being set for the event containing the arguments, eg:

{ arg0: 'customerSession',
  arg1: '03b05742-c43a-46c5-a921-d5ce6bc75157',
  arg2: 'function <anonymous>' }

what's actually present is a function, eg:

[Function]

It looks like the issue is that rather than calling the function to create the context data in probes\trace-probe.js and adding that to the request as the context, we're passing the function itself.

This failure may also happen on npm v2 if the nan module is installed as a peer module (ie at the same level as appmetrics -- I think this could happen if you install nan 2.x separately before appmetrics).

The fix is not to hard code the path, but instead use require('nan') as described in the nan documentation.

In lib/aspects.js the following check is done in two places:

if(process.version.split('.')[0] == 'v0' && process.version.split('.')[1] < 8){}

This isn't needed because Node Application Metrics doesn't versions before 0.10 (there is an install time check for this), so the check is redundant.

Under issue #40 we switched to the use of process.hrtime() to measure the duration of events in the probe framework. We also added it to timer.js for request tracking, but kept the original millisecond duration as as well in case it was being used.

Having been through more testing, it looks like its safe to replace the millisecond duration with the measurement from process.hrtime().

It is interesting to understand how long the period is between requesting a task run on the event loop and it beginning execution.

This can be tracked by periodically:

• taking a timestamp
• enqueuing a callback that emits a monitoring event containing the time elapsed since the timestamp

The callback could be enqueued by, for example: process.nextTick(), setImmediate(), setTimeout() or event at the native layer with uv_async_send().

When we run tracing of a constructor function, eg:

myObj = new mod.Construct(data);

the instrumentation that's injected to generate the tracing fails to return the constructed object, so myObj is undefined.