eostermueller / heapspank Goto Github PK

Spring Boot's "Actuator" provides a dozen or more json http "endpoints" for performance/health monitoring.

Need to create a new Actuator endpoint that provides heapSpank data.

Here is one guide for creating the new endpoint.

The jmap -histo data ends with a summary line that totals the count of instances and the count of all bytes in use, like this:

Total 61297 10152040

heapSpank needs to display the latest version of these numbers in the line above the column headers.

Here are the current header column names:
LKY% (B-INC (JMH (I-INC (R-INC BYTES INSTANCES NUM CLASS

Some of these aren't very valuable, so here are proposed changes:

• Because (JMH is the same for each row, this needs to be moved up above the column headers.
• (I-INC and (R-INC need to just go away, b/c they provide too little bang for the buck. Not much help in finding a leak, too much effort to explain/document.
• (B-INC must stay, because users will need to understand how it contributes to LKY%.

heapSpank only displays the top 10 candidates for memory leaks, and an algorithm chooses which 10 to display.
As good stewards of the heap, we are often interested in memory consumption of other classes, like ones that

• do not leak, but take up ginormous portions of your heap, like cached data
• have caused memory problems before
• are suspect or of concern for some other reason

To address these needs, heapSpeak must let the user specify 'special' class names in a configuration file that heapSpank will always display, in addition to the regular 'top 10'.

Current, glowroot provides jmap -histo output -- but it just displays one run at a time.
Making decisions on a single sample like this is not a good idea.

Need to write a plugin that makes heapSpank data available from glowroot user interface.
Don't need live refresh, like in the command line version.

This doc on glowroot plugins might be helpful:
https://glowroot.org/instrumentation.html

if heapSpank is plugged into a process that is killed, you will see an error message like this:

Error executing HotSpotVirtualMachine.heapHist() for pid [10402] at time [1483212282896] java.lang.reflect.InvocationTargetException com.github.eostermueller.heapspank.leakyspank.tools.JMapHistoException Cause:null java.lang.reflect.InvocationTargetException Error executing HotSpotVirtualMachine.heapHist() for pid [10402] at time [1483212287894] java.lang.reflect.InvocationTargetException

Need to have a cleaner response and display a message like "process 10402 has been terminated" and then perhaps do a System.exit(-1); to terminate heapSpank.

Need to enhance heapSpank to read configuration from a text file.
As a shortcut, configuration must currently be done via code -- see 'Configuration' section of README.md.

Features I'd like:

• command line option to create a new a file with all all parameters with their default values, and perhaps even a little doc.
• look for config file in User's home folder, or specified on command line.
• This interface details the parameters that need to be in the file.
• Startup banner needs to report the location of the file in use.

Need to discover whether the jmap -histo data byte counts are shallow, deep, or somewhere between.

Need to use data from examples in issue #4.

Yes, heapSpank can detect a leak in just minutes.
BUT there is a catch. To be fully certain there is a leak, we need to see that the LKY% = 100 right after the first full GC occurs. With an overside "old" gen and a light application load, this could take hours.

This is an enhancement request for heapSpank to 'know' when a full GC event first happens.
At that point, confidence in the leak should be increased to a full on 'certainty' instead of a just merely a 'guess' or an 'indication'.

Issue #4 needs to be addressed before this one. Once we have examples from #4, we'll see whether this enhancement successfully distinguishes real leaks from increasing counts of objects that are collected in a full gc.

Testing will need to be done with and without the org.heapspank.jmap.histo.live parameter.

I didn't know about this project, although I started doing something pretty similar with check-leak:

https://github.com/check-leak/check-leak

although the way I'm doing things slightly different, (I'm calling a stream exactly like jcmd is doing)... I was thinking about stealing your code on the leaky calculation and pretty much perhaps "merge the projects"...

how do you feel about that?

What if your app starts leaking and crashes in the middle of the night, when there is noone around to watch/operate the heapSpank command line client.

To address this problem, would be helpful to have periodic dumps, perhaps every 15 minutes, of heapSpank data in the app log file.

Could use these instructions for an easy way to plug it in without any code changes.

need to include a handful of sample programs with leaks.

Start with some of these examples:
http://stackoverflow.com/questions/6470651/creating-a-memory-leak-with-java
http://blog.nimbledroid.com/2016/05/23/memory-leaks.html
https://github.com/CodingFabian/JavaCollectionAnalyzer/blob/master/src/main/java/de/codecentric/performance/LeakDemo.java

By default, heapSpank currently passes "live" parameter to jmap -histo, which means heapSpank will ask jmap to trigger a request for a full GC every 5 seconds. Whoops.

That sounds a bit intrusive/invasive, so I need to set the default back to "-all".

This link shows that C code in jmap that triggers the full gc.