Collecting application level metrics about heapster HOT 8 CLOSED

Application metrics is something we have been wanting to collect in
heapster for a while. But most of the existing monitoring solutions use
either collectd or statsd and so we were looking at leveraging them for
application metrics. Native support in heapster sounds even better. My
concern is that of diverging from the open source community around
monitoring.

On Fri, Mar 20, 2015 at 3:21 AM, Jimmi Dyson [email protected]
wrote:

Currently Heapster collects metrics from cAdvisor to gather container
level metrics. Collecting application level metrics would provide an even
richer set of data. Applications would be required to implement an http
endpoint to expose the metrics that it wants to publish.

As a PoC, we have a heapster fork, jadvisor (
https://github.com/fabric8io/jadvisor), that collects JVM stats via JMX
exposed via Jolokia (http://jolokia.org/) for http access to JMX.
Currently the stats that are collected are hard-coded, or rather discovered
by what is available in the JMX tree. I would like to make this more
generic so it can handle any http endpoint & hence be stack agnostic.

The difference that this would introduce to heapster is the need to
connect to individual pods rather than just to the kubelet/cadvisor so
would increase the number of connections made significantly. This would
likely require work on clustering & sharding to implement properly.

—
Reply to this email directly or view it on GitHub
#173.

from heapster.

Agreed that heapster should support statsd & collectd, but running those would require extra setup rather than natively supporting collection in heapster.

Also heapster providing discovery of what to monitor would probably be easier than configuration of collectd & statsd to do the same. My initial thoughts on this were to use selectors to collect configured metrics over http. I'm not sure how you would achieve that simply with statsd/collectd. Adding a REST service to heapster to allow for dynamic configuration of metrics collectors would make this even better IMO.

from heapster.

Application type discovery is an open issue in kubernetes. Filed an issue to discuss
that.

Filed #177 for dynamic backend configuration.

from heapster.

Thanks for raising those. Wonder if it's worth pushing on with using labels while the discussion around discovering application types via the Kubernetes API kicks off?

from heapster.

from heapster.

CC @jszczepkowski

from heapster.

subscribe. @jjw27

from heapster.

Possible with custom metrics. Support in Heapster is done.

from heapster.