canonical / layer-metrics Goto Github PK

Some metrics are best reported per-application, and not per unit. For example, members of a zookeeper cluster. My metric simply does a word count of a conf file:

metrics:
  servers:
    type: gauge
    description: number of zookeeper servers in the cluster
    command: grep ^server /etc/zookeeper/conf/zoo.cfg  | wc -l

When displaying this metric, I only want one count across the whole application, regardless of how many units are deployed. Instead, I get this:

$ juju metrics zookeeper
UNIT       	           TIMESTAMP	 METRIC	VALUE
zookeeper/0	2017-03-19T17:09:43Z	servers	    3
zookeeper/1	2017-03-19T17:04:45Z	servers	    3
zookeeper/2	2017-03-19T17:07:11Z	servers	    3

One approach to address this would be to classify the metric in metrics.yaml as something like "leader-only", so it would only run collect-metrics on the leader of an application. That yaml might looks something like this:

metrics:
  servers:
    type: gauge
    leader-only: true
    description: number of zookeeper servers in the cluster
    command: grep ^server /etc/zookeeper/conf/zoo.cfg  | wc -l

Or perhaps introduce a new metric type like "gauge-leader" that does the same thing -- only run metric collections on the leader of the application:

metrics:
  servers:
    type: gauge-leader
    description: number of zookeeper servers in the cluster
    command: grep ^server /etc/zookeeper/conf/zoo.cfg  | wc -l

Needs verification.

We import the charm dir lib:
https://github.com/CanonicalLtd/layer-metrics/blob/master/hooks/collect-metrics#L5

But charmhelpers is installed in a venv which we don't have access to. We should be doing something like:

# Load modules from $CHARM_DIR/lib
import sys
sys.path.append('lib')

from charms.layer import basic
basic.bootstrap_charm_deps()
basic.init_config_states()

I've included the basic juju-units metric in my charm's metrics.yaml:

$ cat ~/charms/xenial/hadoop-namenode/metrics.yaml
metrics:
  juju-units: {}

This metric has no command key, so my logs are littered with these errors:

unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics error: no metrics specified
unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics Traceback (most recent call last):
unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics   File "/var/lib/juju/agents/unit-namenode-0/charm/hooks/collect-metrics", line 33, in <module>
unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics     check_call(command)
unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics     raise CalledProcessError(retcode, cmd)
unit-namenode-0: 18:07:30 INFO unit.unit-namenode-0.collect-metrics subprocess.CalledProcessError: Command '['add-metric']' returned non-zero exit status 2

Couple potential fixes in ./hooks/collect-metrics:

• inject a command: /bin/true if a metric does not define a command

    for metric, value in values.items():
        command.append("%s=%s" % (metric, value))
    else:
        # fwiw, i'm totally guessing that this will work
        command.append("%s=/bin/true" % (metric))
    return command

• guard the check_call with an if <command is valid>: check_call(command)