ganglia / jmxetric Goto Github PK

Write a utility class to scan the MBean server and identify all the metrics and generate sample XML configuration entries

It may be desirable to scale some values, e.g. to convert millisecond integer values to floats representing seconds with decimal places. This would ensure correct representation of the values on charts, where rrdgraph automatically adds prefixes such as 'm' for milli.

see the jboss issue in JIRA

Hello,
For the last couple of days I've been trying to monitor all the JVM's in our JBoss system.
We have 3 working VM (CentOS 6.5) in domain mode (1 master + 2 slaves).
As I type, that's the current situation:

• Master: 2 JVM processes, 1 host controller + 1 process controller

• Slave 1 : 3 JVM processes, 1 host controller + 1 process controller + 1 webapp (deployed jar)

• Slave 2 : 3 JVM processes, 1 host controller + 1 process controller + 1 webapp (deployed jar)
  There' s one jmxetric xml file for each JVM. The problem:

  We cannot make the webapp JVM's to be monitored. The 6 others are correctly monitored.

For each one of the xml files I uncommented the lines below:

           <mbean name="jboss.as:subsystem=web,connector=ajp" pname="ajp">
                  <attribute name="requestCount" type="int32" slope="positive"/>
           </mbean>

All JVM processes show references of oncrpc-1.0.7.jar, gmetric4j-1.0.7.jar and jmxetric-1.0.6.jar, with apparently correct configurations. Tried to increase the delay at the xml file, but no luck.

Any ideas?

Best!

In the XML output from gmond, the DMAX value for each metric from jmxetric is always 0, e.g. DMAX="0"

We need a way to configure this in jmxetric.xml so that DMAX can be set per metric (or for all the JMXetric metrics as a group)

Currently, JMXetric is configured by jmxetric.xml

It would be useful if jmxetric.xml could source metrics from additional XML files and merge all the attribute definitions.

This would enable sharing config fragments between several JVMs on the same host.

No matter what value is configured for the delay attribute in jmxetric.xml, the gmond XML output always shows TMAX="60" for each metric from jmxetric

This does not appear to be correct

Some applications only create the MBeans some time after starting

In these cases, jmxetric is logging warnings early in the startup process because it is trying to find MBeans that don't exist yet. It appears to find the MBeans eventually and start submitting their metrics.

It needs to be more tolerant of these MBeans. The logs should not contain warnings for such MBeans as it may cause undue concern to users.

Currently, the JVM must be restarted to detect any changes in the JMXetric config file.

It should be able to reload the file at runtime, possibly be detecting changes to the file or reloading when there is a signal or a command from JMX / JConsole.

Refer to MBeanSamplerTest.java, the methods of the Publisher interface is not implemented correctly. This probably happened because I wasn't working with the latest version of ganglia/gmetric4j.

I'm starting the daemon with config=
That config file contains a ganglia section with the wireformat31x flag set to true.
However, jmxetric uses the 30 wireformat.

The problem is that getTagValue( "wireformat31x", args, "false") will return the default string "false".
JMXetric thinks that i passed this "false" value in the config parameters and that i want to override the value in my XML.

In many environments where multicast can't be used for one reason or another, and multiple clusters within ganglia need to be maintained, the need arises to have multiple ganglia target hosts specified.

For example, in gmond for one cluster, I might have two udp_send_client statements like:

udp_send_channel {
bind_hostname = yes
host = 128.118.58.126
port = 8649
ttl = 1
}

udp_send_channel {
bind_hostname = yes
host = 128.118.58.163
port = 8649
ttl = 1
}

And then define a cluster within gmetad with those two hosts as data_sources.

It currently seems impossible to mix jmxetric into an environment like this. I propose one of two ways to correct this:

1. Either in gmetric4j or jmxetric itself, we parse a limited set of configuration directives from gmond.conf, if present, particularly the udp_send_channel stanzas. This would closely mimic the behavior of the gmetric command line tool.

2. Allow multiple configuration elements to be specified. In this way, I could push out a jboss-jmxetric.xml file alongside gmond.conf, with the same set of hosts specified in both files.

should be able to sample counters and convert them to rates for Ganglia

Ganglia expects rates rather than raw counter values

During pre-main class execution, there is a call to

ManagementFactory.getPlatformMBeanServer()

at line MBeanSampler.java:33

In JBoss, getPlatformMBeanServer() then tries to initialize a logger before it is safe to do so.

Therefore, we need to find a way to make this initialization take place later or catch the exception here and try again later.

If I specify:

in the xml configuration, and only invoke jmxetric with:

-javaagent:/path/to/jmxetric-1.0.6.jar=config=/path/to/jmxetric.xml

Then jmxetric never sends metrics. I have to specify host, port, and mode in the command line arguments to make it work.

When using gmond 3.0.x (versions less than 3.1.x) and the setting wireformat31x=false in jmxetric, the slope=positive attribute is ignored

These metrics appear in gmond XML output with slope=both

Hello according to http://docs.oracle.com/javase/7/docs/api/javax/management/ObjectName.html
it is possible to use patterns in the objectName.
If I use a pattern in the mbean parameter. f.i.
name="org.eclipse.jetty.server:type=connectorstatistics,*"
Then Jmxetric doesn't collect the counters

JMX provides the ProcessCpuTime metric. It is the number of nanoseconds of CPU time used by the JVM.

Graphing the raw value in Ganglia is unhelpful.

gmond itself sends cpu_* metrics (e.g. cpu_system) and these are percentages derived from the raw tick values of the host. jmxetric could derive percentages from the JVM values and send those instead. This would be more consistent with the Ganglia design and reduce network load when values are not changing.

We should also consider how this should work for multi-threaded Java apps on multi-CPU systems.

Other projects like jmxtrans have examples of working with these values.

While working on the source code, I noticed some minor inconsistencies in the formatting of source code. Open source projects are prone to such issues since there are multiple contributors and they change all the time.

Should there a style guide that we can follow, or perhaps room for such information to be recorded inside a CONTRIB file?

I will take a look at other popular open source java projects and see what method they adopt.

how to get metrics for all the beans ?

Below is the complete output of the test.
The problem is that in the configureJMXetricFromXML method call of XMLConfigurationService, while reading the first composite data, the missing dmax attribute results in an empty string "" being passed to Integer.parseInt on line 250.

Still investigating why this happens, on first look it looks like the default value of "0" should have been passed in instead.

java.lang.NumberFormatException: For input string: ""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:504)
    at java.lang.Integer.parseInt(Integer.java:527)
    at info.ganglia.jmxetric.XMLConfigurationService.configureJMXetricFromXML(XMLConfigurationService.java:249)
    at info.ganglia.jmxetric.XMLConfigurationService.configure(XMLConfigurationService.java:69)
    at info.ganglia.jmxetric.JMXetricAgentIT.testRun(JMXetricAgentIT.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
    at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
    at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
    at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
    at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
    at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
    at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
    at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
    at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
    at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)