jvm-profiling-tools / perf-map-agent Goto Github PK

View Code? Open in Web Editor NEW

1.6K 1.6K 261.0 127 KB

A java agent to generate method mappings to use with the linux `perf` tool

License: GNU General Public License v2.0

Java 8.57% C 61.15% Shell 20.82% CMake 4.38% Perl 5.08%

perf-map-agent's People

Contributors

Stargazers

Watchers

Forkers

maycat t3rmin4t0r philipa apurtell abak yuanke phoenix24 jerrinot giraldeau brendangregg davidcl gregnwosu smjavalangdev drawlerr boonchu bendalexis bitisony chrisvest gtrght spotify bzz ktoso eduaranega m4dc4p shlomiv rzs840707 anirishduck craiig noscripter phraktle xs3c tboettch jamsauce nileema nitsanw sauliusvl felipebetancur tjake thomasdarimont jujis008 bbc anil11392 sattishv elal normanmaurer conioh opentokix coca1ne chbatey incubos liuqian1990 gitter-badger lishuai1225 kmizu luozengbin tdrjnr rowhit dikang123 adfernandes hefei1986 yongjianxu kenengjiang sjanulonoks daodennis-zz kandeshvari epickrram hnhbdss ifarhankhan yansheng723 mastermind1981 tutty427 liumian97 prasun3 zhaog xiedingyuan dimrain codingcat zhipwang lizhanhui dakar24 leonzhouwei busing tlisonbee heiqiaoxiang neven7 vsarunas feesx zvonkok zhouheming lshm ebernhardson zhoujygit sernle 4v yzxlr qiangcai chenk008 yummly xiashuijun jarvis657

perf-map-agent's Issues

perf script using perf-java-report-stack output will get "Misordered timestamps: 0"

jdk: OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
cmake: version 2.8.12.2
perf: version 3.10.0-693.11.1.el7.x86_64.debug
OS: centOS 7

After using perf-java-record-stack <pid> get output in \tmp\perf-<pid>.data, but using perf script -i /tmp/perf-<pid>.data -d will result in Misordered timestamps: 0. If perf script not run in debug mod, nothing happens and got an empty stack file.

perf-java-top <pid> and perf-java-report-stack <pid> works as expected, but sometimes fail with empty sample, I guess that is normal.

I try this in several machine I've got, all the same.

Attached my .data and .map file.
perf-11484.zip

MacOS problem with bin/create-links-in

My old version of MacOS does not support the readlink -f option. I do not know if newer versions of MacOS support this option.

$ bash -xv bin/create-links-in /usr/local/bin
#!/bin/sh
set -e
+ set -e
#set -x

TARGET_DIR=$1
+ TARGET_DIR=/usr/local/bin
BIN_DIR=`dirname $0`
dirname $0
++ dirname bin/create-links-in
+ BIN_DIR=bin

if [ -z "$TARGET_DIR" ]; then
  echo "Please specify a target directory."
  exit
fi
+ '[' -z /usr/local/bin ']'

ln -s $(readlink -f $BIN_DIR/perf-java-top) $TARGET_DIR
readlink -f $BIN_DIR/perf-java-top
++ readlink -f bin/perf-java-top
readlink: illegal option -- f
usage: readlink [-n] [file ...]
+ ln -s /usr/local/bin
ln: ./bin: File exists

Annotate methods in flamegraphs with info from JIT compiler

I'm filing it here in lack of a better place so that I don't forget about it.

E.g. if a frame couldn't be inlined because it hit MaxInlineDepth, that info could be somehow annotated graphically in the flame graph. Similar for other inlining decisions.

The tool needed would take the output of -XX:+LogCompilation to produce annotations. Flamegraph would need to be able to merge those annotations into the flamegraph.

Unable to build with java 8

posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-8-oracle/include;/usr/lib/jvm/java-8-oracle/include/linux;/usr/lib/jvm/java-8-oracle/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ set -o vi
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake  CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make
Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:14:6: warning: conflicting types for ‘open_file’ [enabled by default]
 void open_file() {
      ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:11:9: note: previous implicit declaration of ‘open_file’ was here
         open_file();
         ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbCompiledMethodLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:46:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s.%s%s\n", code_addr, code_size, csig, name, msig);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbDynamicCodeGenerated’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:65:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s\n", address, length, name);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘Agent_OnLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:93:11: warning: incompatible implicit declaration of built-in function ‘memset’ [enabled by default]
     (void)memset(&capabilities,0, sizeof(capabilities));
           ^
Linking C shared library libperfmap.so
[100%] Built target perfmap
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ ls
CMakeCache.txt  cmake_install.cmake  libperfmap.so  perf-map-agent.c
CMakeFiles      CMakeLists.txt       Makefile       README.md
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake  CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .                      -- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make                         Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:14:6: warning: conflicting types for ‘open_file’ [enabled by default]
 void open_file() {
      ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:11:9: note: previous implicit declaration of ‘open_file’ was here
         open_file();
         ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbCompiledMethodLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:46:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s.%s%s\n", code_addr, code_size, csig, name, msig);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbDynamicCodeGenerated’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:65:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s\n", address, length, name);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘Agent_OnLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:93:11: warning: incompatible implicit declaration of built-in function ‘memset’ [enabled by default]
     (void)memset(&capabilities,0, sizeof(capabilities));
           ^
Linking C shared library libperfmap.so
[100%] Built target perfmap
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake  CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .                      -- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-8-oracle/include;/usr/lib/jvm/java-8-oracle/include/linux;/usr/lib/jvm/java-8-oracle/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make                         Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
In file included from /home/posix4e/projects/perf-map-agent/perf-map-agent.c:1:0:
/usr/lib/jvm/java-8-oracle/include/jni.h:45:20: fatal error: jni_md.h: No such file or directory
 #include "jni_md.h"
                    ^
compilation terminated.
make[2]: *** [CMakeFiles/perfmap.dir/perf-map-agent.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ ls9390

Missing method name from flamegraph

I am trying to build flamegraph, flamegraph is getting created successfully, but it is having a lot of methods without name / Generic symbol.

Out put which i get is below.

[aemauthor@local-aem62-test bin]$ ./perf-java-flames 17715
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.696 MB /tmp/perf-17715.data (618 samples) ]
Failed to open 3E, continuing without symbols
Flame graph SVG written to PERF_FLAME_OUTPUT='/home/aemauthor/perf-map-agent/bin/flamegraph-17715.svg'.

Perf data, method map and Flamegraph are attached.

Archive.zip

Can make in CentOS

Hi
I have ran cmake .
but show me as below
do you know what it is happened?
finished or unfinished?

   -- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at CMakeLists.txt:24 (include):
  include could not find load file:

    UseJava


CMake Error at CMakeLists.txt:30 (add_jar):
  Unknown CMake command "add_jar".


-- Configuring incomplete, errors occurred!

when do i have to create the map

The perf map should be created before or after the per record/top commands?

Clarify license of shell scripts

The repo as a whole is marked as GPLv2, though the C and java files have an explicit header indicating they're GPL2, but the shell scripts don't.

Are the shell scripts also GPL2?

I'm looking at using these for another JIT (i.e. not using the JIT, just the dtrace + perf map stuff), and considering packaging an 'easy profiling' dev-dependency - but we'd want to use that with projects licensed under licenses such as MIT (e.g. https://github.com/hhvm/hhast/blob/master/LICENSE) or more obscure combinations such as PHP+Zend (e.g. https://github.com/facebook/hhvm/ - parts are taken from PHP itself so we can not use other licenses there)

If they are GPL, we'll likely document how to use them, but avoid distributing them or adding any dependencies on them.

make will not build perf-map-agent: fatal error: 'error.h' file not found

$ make

[ 50%] Built target attach-main
[ 66%] Building C object CMakeFiles/perfmap.dir/src/c/perf-map-file.c.o
/Users/justinthomas/tools/perf-map-agent/src/c/perf-map-file.c:24:10: fatal error: 'error.h' file not found
#include <error.h>
         ^
1 error generated.
make[2]: *** [CMakeFiles/perfmap.dir/src/c/perf-map-file.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2

cmake error(cmake version is 2.8.12.2)

i installed cmake, perf, and cmake version is 2.8.12.2, but failed

[root@test perf-map-agent]# cmake .

-- Could NOT find JNI (missing:  JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
JAVA_INCLUDE_PATH (ADVANCED)
   used as include directory in directory /root/perf-map-agent
   used as include directory in directory /root/perf-map-agent
JAVA_INCLUDE_PATH2 (ADVANCED)
   used as include directory in directory /root/perf-map-agent
   used as include directory in directory /root/perf-map-agent

-- Configuring incomplete, errors occurred!
See also "/root/perf-map-agent/CMakeFiles/CMakeOutput.log".
See also "/root/perf-map-agent/CMakeFiles/CMakeError.log".

CMakeError.log

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags:

The output was:
No such file or directory


Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags: -c

The output was:
No such file or directory


Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags:

The output was:
No such file or directory


Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags: -c

The output was:
No such file or directory


Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":

Problem building in Centos 6

When running make I receive the following error:

src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
                           ^

Full output:

$ JAVA_HOME=/usr/java/jdk1.8.0_60/ cmake .
-- JNI_INCLUDE_DIRS=/usr/java/jdk1.8/include;/usr/java/jdk1.8/include/linux;/usr/java/jdk1.8/include
-- JNI_LIBRARIES=/usr/java/jdk1.8/jre/lib/amd64/libjawt.so;/usr/java/jdk1.8/jre/lib/amd64/server/libjvm.so
-- JAVA_INCLUDE_PATH=/usr/java/jdk1.8/include
-- JAVA_INCLUDE_PATH2=/usr/java/jdk1.8/include/linux
-- Configuring done
-- Generating done
-- Build files have been written to: /root/perf-map-agent
$ make
[ 20%] Building Java objects for attach-main.jar
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
                           ^
make[2]: *** [out/CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 1
make[1]: *** [CMakeFiles/attach-main.dir/all] Error 2
make: *** [all] Error 2
$

I run a strace -fe file make and it is finding tools.jar file:

[pid  2128] open("/usr/java/jdk1.8/include/../lib/tools.jar", O_RDONLY) = 5
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
                           ^
[pid  2154] +++ exited with 0 +++

include build info in out directory

I've found that perf-map-agent compiled for OracleJDK can crash OpenJDK (usually on the second invocation). I'm changing my tooling so that we have builds for each. This ticket is to suggest that build info is included in the out directory, so that other tooling can double check what it was built against. Eg, just a:

echo JAVA_HOME=$JAVA_HOME > out/build.txt

From the build process would be handy.

For reference, here's stack trace from one of the crashes:

Core was generated by `java -Xms24952m -Xmx24952m -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemory'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe7253fb700 (LWP 5078))]
Installing openjdk unwinder
(gdb) bt
#0  0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fe7a6d0e02a in __GI_abort () at abort.c:89
#2  0x00007fe7a6012db9 in os::abort(bool) (dump_core=<optimized out>) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
#3  0x00007fe7a61c6897 in VMError::report_and_die() (this=this@entry=0x7fe7253f7e20)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
#4  0x00007fe7a61c6fb7 in crash_handler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8070, ucVoid=0x7fe7253f7f40)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/vmError_linux.cpp:106
#5  0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007fe7a6e1a287 in __GI__dl_addr (symbolp=0x0, mapp=0x0, info=0x7fe7253f83f0, match=0x7fe6f0001800, addr=140630735942332) at dl-addr.c:61
#7  0x00007fe7a6e1a287 in __GI__dl_addr (address=0x7fe7250f72bc, info=0x7fe7253f83f0, mapp=0x0, symbolp=0x0) at dl-addr.c:137
#8  0x00007fe7a6018b49 in os::find(unsigned char*, outputStream*) (addr=addr@entry=0x7fe7250f72bc "\205W\255Ox\361\030\233\067O\200\262\266\067\071_\321\357\335+\204\360\344_\223_", st=st@entry=0x7fe7253f8b90) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:5166
#9  0x00007fe7a600d61d in os::print_location(outputStream*, long, bool) (st=0x7fe7253f8b90, x=140630735942332, verbose=<optimized out>)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:1030
#10 0x00007fe7a601cfbb in os::print_register_info(outputStream*, void*) (st=0x7fe7253f8b90, context=0x7fe7253f8e00)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:852
#11 0x00007fe7a61c557a in VMError::report(outputStream*) (this=0x7fe7253f8c90, st=st@entry=0x7fe7253f8b90)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:542
#12 0x00007fe7a61c63eb in VMError::report_and_die() (this=this@entry=0x7fe7253f8c90)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:971
#13 0x00007fe7a601c29f in JVM_handle_linux_signal(int, siginfo_t*, void*, int) (sig=sig@entry=11, info=info@entry=0x7fe7253f8f30, ucVoid=ucVoid@entry=0x7fe7253f8e00, abort_if_unrecognized=abort_if_unrecognized@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
#14 0x00007fe7a600fef8 in signalHandler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8f30, uc=0x7fe7253f8e00)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4346
#15 0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#16 0x00007fe7a72b9cda in check_match (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", ref=ref@entry=0x0, version=version@entry=0x0, flags=flags@entry=2, type_class=type_class@entry=0, sym=sym@entry=0x6f8, symidx=44, strtab=0x7a0 <error: Cannot access memory at address 0x7a0>, map=0x7fe6f0001800, versioned_sym=0x7fe7253f9338, num_versions=0x7fe7253f9334) at dl-lookup.c:92
#17 0x00007fe7a72ba64b in do_lookup_x (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", new_hash=new_hash@entry=1336760197, old_hash=old_hash@entry=0x7fe7253f9400, ref=0x0, result=result@entry=0x7fe7253f9410, scope=<optimized out>, i=0, version=0x0, flags=2, skip=0x0, type_class=0, undef_map=0x7fe6f0001800) at dl-lookup.c:423
#18 0x00007fe7a72bab1f in _dl_lookup_symbol_x (undef_name=0x7fe6f00076b0 "Agent_OnAttach", undef_map=0x7fe6f0001800, ref=0x7fe7253f94e8, symbol_scope=0x7fe6f0001b88, version=0x0, type_class=0, flags=2, skip_map=0x0) at dl-lookup.c:829
#19 0x00007fe7a6e1a873 in do_sym (handle=0x7fe6f0001800, name=0x7fe6f00076b0 "Agent_OnAttach", who=<optimized out>, vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:168
#20 0x00007fe7a6e1ad1d in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at dl-sym.c:273
#21 0x00007fe7a68ba034 in dlsym_doit (a=a@entry=0x7fe7253f96f0) at dlsym.c:50
#22 0x00007fe7a72c0564 in _dl_catch_error (objname=0x7fe6f00017c0, errstring=0x7fe6f00017c8, mallocedp=0x7fe6f00017b8, operate=0x7fe7a68ba020 <dlsym_doit>, args=0x7fe7253f96f0) at dl-error.c:187
#23 0x00007fe7a68ba571 in _dlerror_run (operate=operate@entry=0x7fe7a68ba020 <dlsym_doit>, args=args@entry=0x7fe7253f96f0) at dlerror.c:163
#24 0x00007fe7a68ba088 in __dlsym (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach") at dlsym.c:70
#25 0x00007fe7a60138f4 in os::dll_lookup(void*, char const*) (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach")
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:2052
#26 0x00007fe7a600c34c in os::find_agent_function(AgentLibrary*, bool, char const**, unsigned long) (agent_lib=agent_lib@entry=0x7fe6f00079a0, check_lib=check_lib@entry=false,---Type <return> to continue, or q <return> to quit--- 
 syms=syms@entry=0x7fe7253f9860, syms_len=syms_len@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:490
#27 0x00007fe7a5e8f5f3 in JvmtiExport::load_agent_library(AttachOperation*, outputStream*) (op=0x7fe6f0000a30, st=0x7fe7253fad20)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/prims/jvmtiExport.cpp:2252
#28 0x00007fe7a5a43db1 in attach_listener_thread_entry(JavaThread*, Thread*) (thread=<optimized out>, __the_thread__=<optimized out>)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/services/attachListener.cpp:453
#29 0x00007fe7a616aa07 in JavaThread::thread_main_inner() (this=this@entry=0x7fe708001000)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1699
#30 0x00007fe7a616aecc in JavaThread::run() (this=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1679
#31 0x00007fe7a6011d82 in java_start(Thread*) (thread=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:790
#32 0x00007fe7a66a36ba in start_thread (arg=0x7fe7253fb700) at pthread_create.c:333
#33 0x00007fe7a6dde3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Report missing tools.jar in build and run script instead of failing with NoClassDefFoundError

Hello,

After succesfully creating flamegraphs a few days ago I tried it again today and only get this error:

Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/attach/AgentInitializationException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.AgentInitializationException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more

I haven't changed anything on the machine. Is there a solution to this or a way to get more error information?

Not able to resolve symbols

I'm trying to use your project on a fedora 24 using openjdk 1.8.0.101 . The problem can be seen using perf-java-top and flame graphs as well . The create-java-perf-map seems to work well but it did saw how could get its output with perf to plot with frame. Jstack works as well (but as you mention it would be a terrible poor performance and quality).

I thought it would be something with preserving the framepointer but the jdk version is compatible and I'm using the following JVM options:

-XX:+PreserveFramePointer -XX:+StartAttachListener -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:-OmitStackTraceInFastThrow -XX:+ShowHiddenFrames

Do you have any suggestions about how to solve this problem?

Thanks in advance
Luan

CMake issue

Hi
I have ran cmake. Some error happened. Do you have any idea?

-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- JAVA_INCLUDE_PATH=/usr/lib/jvm/java-7-openjdk-amd64/include
-- JAVA_INCLUDE_PATH2=/usr/lib/jvm/java-7-openjdk-amd64/include
CMake Error at /usr/share/cmake-2.8/Modules/UseJava.cmake:269 (string):
string sub-command LENGTH requires two arguments.
Call Stack (most recent call first):
CMakeLists.txt:30 (add_jar)

-- Configuring incomplete, errors occurred!

JDK inline info not accurate in all cases

For some methods there can be some chunks of code for which proper inline information seems to be missing (compared to PrintAssembly output).

I don't think there can be done anything about that realistically inside of perf-map-agent but if we find some clues why that would be the case it could be reported to openjdk (maybe it is but I couldn't find it).

For reference, here are the code locations where this info is gathered:

Through the agent/JVMTI interface:

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/hotspot/share/prims/jvmtiExport.cpp#L2093

In contrast, for PrintAssembly:

From here:

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/3e78ed84f13da94b98f6365935b2192cf0ac2ee0/src/hotspot/share/compiler/disassembler.cpp#L359

to finally:

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/3e78ed84f13da94b98f6365935b2192cf0ac2ee0/src/hotspot/share/code/nmethod.cpp#L2652

From my quick investigation, the issue might be the way that scopes_pcs_begin reports interesting (= scoped) code regions, in contrast to what scopes are reported if you consider every single PC of an nmethod.

Document how to run perf-map-agent against Docker containers

Hi team,

We have our application running inside a container. If thats the case the map-agent should be installed outside or inside the container. I presume it should be done inside the container because the jdk we use is referenced inside. Could you please review and let us know the ideal approach.

Sattish.

Improve support for JSVC

Update scripts to detect jsvc process in addition to java. I may propose a patch but this will take me a few days.

Improve reporting of inlined frames

I think PMA is in a unique position to break new grounds in Java profiling by reporting the Java stack while highlighting the separation between real frames and virtual frames(inlined methods). To do this we will need:

Report all frames when reporting inlined data: This can easily lead to VERY long strings. I'm not sure what the maximum 'method' name length supported by perf is.
Support from flame graphs in colouring and displaying the frames.

run perf script fail

hello!
i have a question about perf
i ran the command as below

perf script|./stackcollapse-perf.pl |./flamegraph.pl --color=java --hash > flamegraph1.svg

system show me

Failed to open /usr/lib64/libc-2.17.so;56dd2155, continuing without symbols
Failed to open /usr/lib64/libpthread-2.17.so;56dd2155, continuing without symbols

os:CentOS Linux release 7.2.1511 (Core)
cmake:cmake version 2.8.11

Add a verbose mode for easier troubleshooting and log error messages

Currently PMA fails or succeeds silently. When things are good this is a non-issue, when something doesn't work you probably want to see a bit more output.
I've played around with the idea and would like to log:

Any JVMTI failure
Agent method processing time
At debug level add a per method printout of some form?

Ideas and feedback before I turn this into a PR?

Generating mappings for transient java processes / agentlib mode

Hi all,
I'm trying to profile systems that consist of lots of java processes being started and stopped over time. Does anyone know of a way to ensure that I can get a symbol mapping of each java process that gets run on my system? Just running once won't capture these transient processes, so I'd ideally like something that automatically makes a symbol dump, maybe on process termination? Any thoughts would be appreciate.

Not sure if this is the right place to ask, feel free to point me somewhere else!
Thanks a lot for this tool.

bump up to new version for new release?

The released pre-built binary got slightly old. https://github.com/jvm-profiling-tools/perf-map-agent/releases
For automated usecases, pre-built binary from official repository is helpful (especially in non-tech meaning).

Build is passing, so making new release should be easy.

unfold reports inlined methods off-by-one

As discovered by @nitsanw. Proposed fix in #21.

option to disable full method signatures

Would like an option to disable full method signatures. Eg, "nomsig". So this:

7feca143ba60 164c0 Lio/netty/handler/codec/http/DefaultHttpHeaders;.set(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio

would become:

7feca143ba60 164c0 Lio/netty/handler/codec/http/DefaultHttpHeaders;.set

Some C stack frames with Java ancestor not correctly shown

HI guys!

I have produced a flame graph with:

PERF_COLLAPSE_OPTS="--kernel --tid" PERF_RECORD_FREQ=99 PERF_RECORD_SECONDS=10 PERF_MAP_OPTIONS=unfoldall ~/perf-map-agent/bin/perf-java-flames <pid>

But seems that (some) C stack frames are not correctly shown as children of the related Java calls.
Am I missing anything in the configuration?

Thanks,
Franz

Document or improve scripts to make perf-java-flames work with docker

I'm trying to create a Java process FlameGraph with perf-java-flames. It seems to run successfully, but I can't find the resulting svg file.

$ ./perf-java-flames 161991 -F 99 -a -g -- sleep 30
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
Warning:
PID/TID switch overriding SYSTEM
$

CentOS 7
3.10.0-327.36.3.el7.x86_64
cmake version 2.8.12.2

Up-to-date versions of perf-map-agent and the FlameGraph repo.

Lack of perf PMU event

Hi,

It's impossible to record plain PMU events like mispredicted branches with perf-java-recors-stack. This script supports profiling with specified frequency only (-F argument).

I'd like to propose either add one more script (like perf-java-record-stacks) which is able to record PMU events or modify existing perf-java-record-stack script in order to let user to specify PMU event to profile. If you are ok for these changes, I would implement it.

--Sergey

Unable to load agent sence Premain-Class manifest attribute is missing

When I try to launch my jar with the agentpath set to attach-main.jar, I get

Failed to find Premain-Class manifest attribute in /perf-map/out/attach-main.jar

Any ideas?

perf-map-agent as jvmti agent on start

Hi,

I tried to use perf-map-agent as pure JVMTI agent via -agentlib argument. But the current implementation doesn't have Agent_OnLoad method.

So, does it worth to implement Agen_OnLoad function (I'm ready to do it)? If so, should I handle a situation when compiled method was abandoned (is it possible to handle these situations?)?

--Sergey

cannot print method defined by user program

Really a newbie question

When I use the agent on a perf record command line and then run perf report, I just see some built-in methods ，like System.out.prinln, but it does not print other methods ,such as , "main, print_rectangle（a user-defining method which just prints some "*"）". I follow the steps on your your blog Java in Flames ( 1. install perf-map-agent , 2. profiling and flame graph generation:).

When I run perf record -F 49 -a -g -- sleep 30; ./FlameGraph/jmaps , I see the perf-5084.map file.
The perf report shows some unknown info:
Samples: 331 of event 'cycles', Event count (approx.): 3811883239
Children Self Command Shared Object Symbol ◆
-- 69.94% 0.00% java [unknown] [.] 0xb3f180e200026c80 ▒
-- 0xb3f180e200026c80 ▒
81.85% 0 ▒
18.15% 0x8000268d8 ▒
-- 69.13% 1.37% java perf-7341.map [.] Ljava/io/FileOutputSt▒
-- Ljava/io/FileOutputStream;::writeBytes ▒
++ 81.75% Ljava/io/PrintStream;::print ▒
++ 18.25% Lsun/nio/cs/StreamEncoder;::flushBuffer ▒
++ 67.76% 0.00% java libjava.so [.] Java_java_io_FileOutp▒
++ 66.79% 0.00% java libpthread-2.17.so [.] 0xffff80d8d849543d ▒
++ 63.03% 6.52% java perf-7341.map [.] Ljava/io/PrintStream;▒
++ 62.09% 0.00% java libjava.so [.] 0xffff80d8dabfe1f0 ▒
++ 56.92% 0.00% java [unknown] [.] 0000000000000000 ▒
++ 56.66% 0.69% java [kernel.kallsyms] [k] sys_write ▒
++ 56.30% 0.00% java [kernel.kallsyms] [k] system_call_fastpath ▒
++ 51.99% 1.85% java [kernel.kallsyms] [k] vfs_write

My java version is 1.8.0.131. And up to date perf-map-agent repo~

add troubleshooting section

There is a known issues section, but it is reasonably easy to not generate the symbols for a number of reasons.
These problems should be fairly easy to deal with if documented in troubleshooting steps/procedure.
Sure they will be documented in the issues but not everyone reads the issues and then if needed wastes a roundtrip as we ask for diagnostic information.

'jvmticmlr.h' file not found

➜  perf-map-agent git:(master) ✗ make
-- JNI_INCLUDE_DIRS=/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers
-- JNI_LIBRARIES=/System/Library/Frameworks/JavaVM.framework;/System/Library/Frameworks/JavaVM.framework
-- JAVA_INCLUDE_PATH=/System/Library/Frameworks/JavaVM.framework/Headers
-- JAVA_INCLUDE_PATH2=/System/Library/Frameworks/JavaVM.framework/Headers
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   perfmap

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/xiezhaodong/Downloads/perf-profile/perf-map-agent
[ 16%] Building Java objects for attach-main.jar
[ 33%] Generating out/CMakeFiles/attach-main.dir/java_class_filelist
[ 50%] Creating Java archive attach-main.jar
[ 50%] Built target attach-main
[ 66%] Building C object CMakeFiles/perfmap.dir/src/c/perf-map-agent.c.o
/Users/xiezhaodong/Downloads/perf-profile/perf-map-agent/src/c/perf-map-agent.c:30:10: fatal error: 'jvmticmlr.h' file not found
#include <jvmticmlr.h>
         ^
1 error generated.
make[2]: *** [CMakeFiles/perfmap.dir/src/c/perf-map-agent.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2

I change CMakelist.text line set(CMAKE_JAVA_INCLUDE_PATH /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/lib/tools.jar)

./bin/dtrace-java-record-stack is not working

Ran ./bin/dtrace-java-record-stack but the command does not work. I think the dtrace command inside the ./bin/dtrace-java-record-stack is not used properly.

cat /tmp/dtrace-25840.data
Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I]] -s File.d [-o ]

incorrect method resolution?

Hi,

I'm seeing some methods being unexpectedly listed high on perf top, which are called very infrequently. I have cross-checked with a JMC flight recording (which I assume to be quite accurate), and these paths were not showing up there at all.

Is there a chance that the map is somehow inaccurate, perhaps over time with hotspot recompiling / deoptimizing / moving stuff around?

Btw, this agent is quite useful, keep up the good work :)

Regards,
Viktor

Could not find or load main class net.virtualvoid.perf.AttachOnce

# ./bin/perf-java-top 19835
Error: Could not find or load main class net.virtualvoid.perf.AttachOnce

scripts expects attach-main.jar to be found in out dir, while it is in the project root dir.

Failed to load agent library

Hi guys,

I used Java(TM) SE Runtime Environment (build 1.8.0_60-ea-b25)
and run ./create-java-perf-map.sh $pid

got:

Exception in thread "main" com.sun.tools.attach.AgentLoadException: Failed to load agent library
at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentPath(HotSpotVirtualMachine.java:88)
at net.virtualvoid.perf.AttachOnce.loadAgent(AttachOnce.java:39)
at net.virtualvoid.perf.AttachOnce.main(AttachOnce.java:33)

Any idea?

Thanks so much.

time to generate map file

Hello, Authors,
Thank you to open-source this projects. The agent and handy scripts are extremely helpful for me.

I have a problem about JIT-ed code. I end up with a lot of PCs which point to nowhere but fall in the range of map files. I interpret it in this way: the symbols in the map file are not accurate.

I have a question about this line. It seems that two statements happen in sequence.

perf-map-agent/bin/perf-java-record-stack

Line 25 in d9843a0

 sudo perf record -F $PERF_RECORD_FREQ -o $PERF_DATA_FILE -g -p $* -- sleep $PERF_RECORD_SECONDS 

That is to say, the script generates map file right after 'perf record' has done. Why not make them run in parallel? In current model, will agent has a skewed view? or I have to make sure my profiling program enter a stable state ?

Resolve `perf-map-agent` directory better

Thanks for your small but very useful tool. I've been working with the latest commits from master on the tool and I hit some issues mainly related to how the paths are solved and the main JAR file is executed. Here is a suggest to improve a bit for simpler usage:

diff --git a/bin/create-java-perf-map.sh b/bin/create-java-perf-map.sh
index 604fa44..b83f342 100755
--- a/bin/create-java-perf-map.sh
+++ b/bin/create-java-perf-map.sh
@@ -6,8 +6,7 @@ CUR_DIR=`pwd`
 PID=$1
 OPTIONS=$2
 ATTACH_JAR=attach-main.jar
-PERF_MAP_DIR=$(dirname $(readlink -f $0))/..
-ATTACH_JAR_PATH=$PERF_MAP_DIR/out/$ATTACH_JAR
+PERF_MAP_DIR=$(realpath $(dirname $(readlink -f $0))/..)
 PERF_MAP_FILE=/tmp/perf-$PID.map

 if [ -z $JAVA_HOME ]; then
@@ -18,5 +17,5 @@ fi
 [ -d $JAVA_HOME ] || (echo "JAVA_HOME directory at '$JAVA_HOME' does not exist." && false)

 sudo rm $PERF_MAP_FILE -f
-(cd $PERF_MAP_DIR && java -cp $ATTACH_JAR_PATH:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID $OPTIONS)
+(cd $PERF_MAP_DIR/out && java -cp $ATTACH_JAR:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID $OPTIONS)
 sudo chown root:root $PERF_MAP_FILE
diff --git a/bin/perf-java-flames b/bin/perf-java-flames
index eb7c235..4a3d4ee 100755
--- a/bin/perf-java-flames
+++ b/bin/perf-java-flames
@@ -26,7 +26,7 @@ if [ -z $PERF_DATA_FILE ]; then
 fi

 if [ -z $PERF_FLAME_OUTPUT ]; then
-  PERF_FLAME_OUTPUT=flamegraph-$PID.svg
+  PERF_FLAME_OUTPUT=$PERF_JAVA_TMP/flamegraph-$PID.svg
 fi

 $PERF_MAP_DIR/bin/perf-java-record-stack $*

Summary:

Remove ATTACH_JAR_PATH.
Use realpath for more correctly resolve the base directory of perf-map-agent.
Default PERF_FLAME_OUTPUT to $PERF_JAVA_TMP/flamegraph-$PID.svg if not already set.

Cannot build on OS X El Capitan

Hi,

I can't seem to build most of the targets on El Capitan.

$ cmake --version
cmake version 3.5.20160411-g9ac11

CMake suite maintained and supported by Kitware (kitware.com/cmake)

$ cmake .
-- JNI_INCLUDE_DIRS=/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers
-- JNI_LIBRARIES=/System/Library/Frameworks/JavaVM.framework;/System/Library/Frameworks/JavaVM.framework
-- JAVA_INCLUDE_PATH=/System/Library/Frameworks/JavaVM.framework/Headers
-- JAVA_INCLUDE_PATH2=/System/Library/Frameworks/JavaVM.framework/Headers
-- LIBS: /Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin/jar  incl: /System/Library/Frameworks/JavaVM.framework/Headers/../lib/tools.jar output_dir: /Users/username/code/perf-map-agent/out
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   perfmap

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/username/code/perf-map-agent

$ make --debug
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `default_target' does not exist.
   File `all' does not exist.
     File `cmake_check_build_system' does not exist.
    Must remake target `cmake_check_build_system'.
    Successfully remade target file `cmake_check_build_system'.
  Must remake target `all'.
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `all' does not exist.
   File `CMakeFiles/attach-main.dir/all' does not exist.
  Must remake target `CMakeFiles/attach-main.dir/all'.
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `CMakeFiles/attach-main.dir/depend' does not exist.
Must remake target `CMakeFiles/attach-main.dir/depend'.
Successfully remade target file `CMakeFiles/attach-main.dir/depend'.
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `CMakeFiles/attach-main.dir/build' does not exist.
   File `attach-main' does not exist.
     File `CMakeFiles/attach-main' does not exist.
       File `out/attach-main.jar' does not exist.
           File `out/CMakeFiles/attach-main.dir/java_compiled_attach-main' does not exist.
          Must remake target `out/CMakeFiles/attach-main.dir/java_compiled_attach-main'.
[ 16%] Building Java objects for attach-main.jar
src/java/AttachOnce.java:25: error: package com.sun.tools.attach does not exist
import com.sun.tools.attach.VirtualMachine;
                           ^
src/java/AttachOnce.java:37: error: cannot find symbol
        VirtualMachine vm = VirtualMachine.attach(pid);
        ^
  symbol:   class VirtualMachine
  location: class AttachOnce
src/java/AttachOnce.java:37: error: cannot find symbol
        VirtualMachine vm = VirtualMachine.attach(pid);
                            ^
  symbol:   variable VirtualMachine
  location: class AttachOnce
src/java/AttachOnce.java:46: error: package com.sun.tools.attach does not exist
        } catch(com.sun.tools.attach.AgentInitializationException e) {
                                    ^
4 errors
make[2]: *** [out/CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 1
make[1]: *** [CMakeFiles/attach-main.dir/all] Error 2
make: *** [all] Error 2

annotate a JITted method

Possible newbie question...

When I use the agent on a perf record command line and then run perf report, I do indeed see the JITted methods listed with event percentages, etc. But when I hit RETURN and try to drill down into a JITted method, no code is shown.

(still) unable to resolve JIT symbols

Hi Johannes,

I'm trying to profile a Java program (SPECjbb2015) using linux perf + your perf-map-agent.

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
perf-map-agent is compiled with JAVA_HOME set to the directory of
the above java (i.e. java is ${JAVA_HOME}/bin/java )
Lunux version is 4.4.0-45 (Ubuntu)
-XX:+PreserveFramePointer is used with java invocation
/opt/perf-map-agent-master/bin/create-java-perf-map.sh $pid is invoked right
after Java invocation where $pid is the pid of java
the symbols from Java runtime seem to be OK, as I can see lines like follows:
6.44% 201751 java libjvm.so [.] PSPromotionManager::copy_to_survivor_space
Most unresolved symbols are from perf-$pid-map, but still few came from libXXX (e.g. libc)
or vdso
Profiling 'cycles' events, the aggregated number of samples for unresolved symbols
take about 55% to 60% of the total samples.
Since JRE symbols are resolved, unresolved symbols seem to come from SPECjbb' jar file.
I recompiled it with the following options to pom.xml

-verbose -g -Xlint:all,-options,-path which inflated the jar size from 1.8MB to 9.1MB However, the results looked the same ( was already true in the original pom.xml).

Do you think of any (possible) mistake I'm making ?
Or any options I should try ?
Thank you very much
Hitoshi

Perf not listing more than 1 line in JVM stack

In one of my cluster, I can attach perf-map agent and get the symbol dump without issue.

I can record "perf record -F 99 -ag" and generate script via "perf script". However, it lists only one line for the JVM stack (the top frame in the stack). JVM is started with "-XX:-OmitStackTraceInFastThrow".

Has anyone observed something like this?

Partial Stacks

This is probably a JVM issue, but I'll file it here anyway as it's something anyone may run into when using perf-map-agent.

Many stacks that perf reports look partial/incomplete, showing the top frame only. Eg, the following output is from "perf report --stdio":

     0.78%     java  perf-22919.map      [.] Lio/netty/handler/codec/http/DefaultHttpHeaders;.add(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio/netty/handler/codec/
               |
               --- Lio/netty/handler/codec/http/DefaultHttpHeaders;.add(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio/netty/handler/codec/http/HttpHeaders;

     0.78%     java  perf-22919.map      [.] Lrx/subjects/PublishSubject;.onNext(Ljava/lang/Object;)V
               |
               --- Lrx/subjects/PublishSubject;.onNext(Ljava/lang/Object;)V

     0.78%     java  perf-22919.map      [.] Lio/netty/channel/AbstractChannel$AbstractUnsafe;.write(Ljava/lang/Object;Lio/netty/channel/ChannelPromise;)V
               |
               --- Lio/netty/channel/AbstractChannel$AbstractUnsafe;.write(Ljava/lang/Object;Lio/netty/channel/ChannelPromise;)V

It looks a lot like omit-frame-pointer optimization, however, I believe this is from hotspot. If there is a hotspot equivalent option for -fno-omit-frame-pointer, I haven't found it yet.

I am using:

-XX:-OmitStackTraceInFastThrow -XX:+UnlockDiagnosticVMOptions -XX:+ShowHiddenFrames

Interpreter frames

I often see "Interpreter" frames. I think I was expecting more symbols.

   100.00%     java  perf-20844.map  [.] Interpreter
               |
               --- Interpreter
                   Interpreter
                   Interpreter
                   Interpreter
                   call_stub
                   JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)
                   jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
                   jni_CallStaticVoidMethod
                   JavaMain
                   start_thread

Research using uprobes on JIT'd methods

Challenges:

uprobes only works with VMA's backed by files while the code cache of the (Hotspot) JVM uses an anonymous region
uprobes apparently doesn't allow the code memory region to be writable
dynamic unloading of JIT code requires some process to unregister the probe in time
it would be nice to access variables in a probe but then you need to know the mapping between Java / Bytecode variables and native registers / stack, the JVM tracks this information for various purposes, but it's probably not exposed in JVMTI

java 9 support

http://openjdk.java.net/jeps/220

some changes needed not limited to:

the tools.jar has gone. change create-java-perf-map.sh to find the classes via add-modules

(cd perf-map-agent/out/ && java --add-modules jdk.attach -jar ./attach-main.jar $(pidof java ) )

unfold reports wrong last method (or may even crash)

As discovered by @nitsanw. Proposed fix in #21.

Allow attaching to running JVMs

properly handle attaching
add user program to control attaching
(maybe fix the permission issue this way: if the program controlling the attachment is run as root it may be possible to create the file and share a descriptor for that file with enough permissions with the agent)

High CPU time in vtable chunks

During our code profiling tests on 16-core systems running 8-16 worker threads, we are observing a significant CPU time (25%) in "vtable chunks" based on the generated symbol map. I think this symbol is listed multiple times as well in the symbol map. This overhead also increases as the number of running threads is gradually raised from 8 to 16.

Is this an overhead of perf-map-agent or something else ? Would like to also confirm if its related to virtual methods in our code.

Another unrelated issue is that perf top always displays "unable to open vsyscalls". Is there a way to resolve it. Does this mean that perf is unable to display system calls and their CPU time ? All processes are being executed by the "root" user.