jvm-profiling-tools / perf-map-agent Goto Github PK
View Code? Open in Web Editor NEWA java agent to generate method mappings to use with the linux `perf` tool
License: GNU General Public License v2.0
A java agent to generate method mappings to use with the linux `perf` tool
License: GNU General Public License v2.0
jdk: OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
cmake: version 2.8.12.2
perf: version 3.10.0-693.11.1.el7.x86_64.debug
OS: centOS 7
After using perf-java-record-stack <pid>
get output in \tmp\perf-<pid>.data
, but using perf script -i /tmp/perf-<pid>.data -d
will result in Misordered timestamps: 0
. If perf script
not run in debug mod, nothing happens and got an empty stack file.
perf-java-top <pid>
and perf-java-report-stack <pid>
works as expected, but sometimes fail with empty sample
, I guess that is normal.
I try this in several machine I've got, all the same.
Attached my .data
and .map
file.
perf-11484.zip
My old version of MacOS does not support the readlink -f
option. I do not know if newer versions of MacOS support this option.
$ bash -xv bin/create-links-in /usr/local/bin
#!/bin/sh
set -e
+ set -e
#set -x
TARGET_DIR=$1
+ TARGET_DIR=/usr/local/bin
BIN_DIR=`dirname $0`
dirname $0
++ dirname bin/create-links-in
+ BIN_DIR=bin
if [ -z "$TARGET_DIR" ]; then
echo "Please specify a target directory."
exit
fi
+ '[' -z /usr/local/bin ']'
ln -s $(readlink -f $BIN_DIR/perf-java-top) $TARGET_DIR
readlink -f $BIN_DIR/perf-java-top
++ readlink -f bin/perf-java-top
readlink: illegal option -- f
usage: readlink [-n] [file ...]
+ ln -s /usr/local/bin
ln: ./bin: File exists
I'm filing it here in lack of a better place so that I don't forget about it.
E.g. if a frame couldn't be inlined because it hit MaxInlineDepth
, that info could be somehow annotated graphically in the flame graph. Similar for other inlining decisions.
The tool needed would take the output of -XX:+LogCompilation
to produce annotations. Flamegraph would need to be able to merge those annotations into the flamegraph.
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-8-oracle/include;/usr/lib/jvm/java-8-oracle/include/linux;/usr/lib/jvm/java-8-oracle/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ set -o vi
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make
Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:14:6: warning: conflicting types for ‘open_file’ [enabled by default]
void open_file() {
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:11:9: note: previous implicit declaration of ‘open_file’ was here
open_file();
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbCompiledMethodLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:46:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
fprintf(method_file, "%lx %x %s.%s%s\n", code_addr, code_size, csig, name, msig);
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbDynamicCodeGenerated’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:65:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
fprintf(method_file, "%lx %x %s\n", address, length, name);
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘Agent_OnLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:93:11: warning: incompatible implicit declaration of built-in function ‘memset’ [enabled by default]
(void)memset(&capabilities,0, sizeof(capabilities));
^
Linking C shared library libperfmap.so
[100%] Built target perfmap
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ ls
CMakeCache.txt cmake_install.cmake libperfmap.so perf-map-agent.c
CMakeFiles CMakeLists.txt Makefile README.md
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake . -- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:14:6: warning: conflicting types for ‘open_file’ [enabled by default]
void open_file() {
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:11:9: note: previous implicit declaration of ‘open_file’ was here
open_file();
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbCompiledMethodLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:46:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
fprintf(method_file, "%lx %x %s.%s%s\n", code_addr, code_size, csig, name, msig);
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbDynamicCodeGenerated’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:65:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
fprintf(method_file, "%lx %x %s\n", address, length, name);
^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘Agent_OnLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:93:11: warning: incompatible implicit declaration of built-in function ‘memset’ [enabled by default]
(void)memset(&capabilities,0, sizeof(capabilities));
^
Linking C shared library libperfmap.so
[100%] Built target perfmap
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake . -- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-8-oracle/include;/usr/lib/jvm/java-8-oracle/include/linux;/usr/lib/jvm/java-8-oracle/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
In file included from /home/posix4e/projects/perf-map-agent/perf-map-agent.c:1:0:
/usr/lib/jvm/java-8-oracle/include/jni.h:45:20: fatal error: jni_md.h: No such file or directory
#include "jni_md.h"
^
compilation terminated.
make[2]: *** [CMakeFiles/perfmap.dir/perf-map-agent.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ ls9390
I am trying to build flamegraph, flamegraph is getting created successfully, but it is having a lot of methods without name / Generic symbol.
Out put which i get is below.
[aemauthor@local-aem62-test bin]$ ./perf-java-flames 17715
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.696 MB /tmp/perf-17715.data (618 samples) ]
Failed to open 3E, continuing without symbols
Flame graph SVG written to PERF_FLAME_OUTPUT='/home/aemauthor/perf-map-agent/bin/flamegraph-17715.svg'.
Perf data, method map and Flamegraph are attached.
Hi
I have ran cmake .
but show me as below
do you know what it is happened?
finished or unfinished?
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at CMakeLists.txt:24 (include):
include could not find load file:
UseJava
CMake Error at CMakeLists.txt:30 (add_jar):
Unknown CMake command "add_jar".
-- Configuring incomplete, errors occurred!
The perf map should be created before or after the per record/top commands?
The repo as a whole is marked as GPLv2, though the C and java files have an explicit header indicating they're GPL2, but the shell scripts don't.
Are the shell scripts also GPL2?
I'm looking at using these for another JIT (i.e. not using the JIT, just the dtrace + perf map stuff), and considering packaging an 'easy profiling' dev-dependency - but we'd want to use that with projects licensed under licenses such as MIT (e.g. https://github.com/hhvm/hhast/blob/master/LICENSE) or more obscure combinations such as PHP+Zend (e.g. https://github.com/facebook/hhvm/ - parts are taken from PHP itself so we can not use other licenses there)
If they are GPL, we'll likely document how to use them, but avoid distributing them or adding any dependencies on them.
$ make
[ 50%] Built target attach-main
[ 66%] Building C object CMakeFiles/perfmap.dir/src/c/perf-map-file.c.o
/Users/justinthomas/tools/perf-map-agent/src/c/perf-map-file.c:24:10: fatal error: 'error.h' file not found
#include <error.h>
^
1 error generated.
make[2]: *** [CMakeFiles/perfmap.dir/src/c/perf-map-file.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2
i installed cmake, perf, and cmake version is 2.8.12.2, but failed
[root@test perf-map-agent]# cmake .
-- Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
JAVA_INCLUDE_PATH (ADVANCED)
used as include directory in directory /root/perf-map-agent
used as include directory in directory /root/perf-map-agent
JAVA_INCLUDE_PATH2 (ADVANCED)
used as include directory in directory /root/perf-map-agent
used as include directory in directory /root/perf-map-agent
-- Configuring incomplete, errors occurred!
See also "/root/perf-map-agent/CMakeFiles/CMakeOutput.log".
See also "/root/perf-map-agent/CMakeFiles/CMakeError.log".
CMakeError.log
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags:
The output was:
No such file or directory
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags: -c
The output was:
No such file or directory
Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags:
The output was:
No such file or directory
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags: -c
The output was:
No such file or directory
Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":
When running make I receive the following error:
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
^
Full output:
$ JAVA_HOME=/usr/java/jdk1.8.0_60/ cmake .
-- JNI_INCLUDE_DIRS=/usr/java/jdk1.8/include;/usr/java/jdk1.8/include/linux;/usr/java/jdk1.8/include
-- JNI_LIBRARIES=/usr/java/jdk1.8/jre/lib/amd64/libjawt.so;/usr/java/jdk1.8/jre/lib/amd64/server/libjvm.so
-- JAVA_INCLUDE_PATH=/usr/java/jdk1.8/include
-- JAVA_INCLUDE_PATH2=/usr/java/jdk1.8/include/linux
-- Configuring done
-- Generating done
-- Build files have been written to: /root/perf-map-agent
$ make
[ 20%] Building Java objects for attach-main.jar
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
^
make[2]: *** [out/CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 1
make[1]: *** [CMakeFiles/attach-main.dir/all] Error 2
make: *** [all] Error 2
$
I run a strace -fe file make
and it is finding tools.jar
file:
[pid 2128] open("/usr/java/jdk1.8/include/../lib/tools.jar", O_RDONLY) = 5
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
^
[pid 2154] +++ exited with 0 +++
I've found that perf-map-agent compiled for OracleJDK can crash OpenJDK (usually on the second invocation). I'm changing my tooling so that we have builds for each. This ticket is to suggest that build info is included in the out directory, so that other tooling can double check what it was built against. Eg, just a:
echo JAVA_HOME=$JAVA_HOME > out/build.txt
From the build process would be handy.
For reference, here's stack trace from one of the crashes:
Core was generated by `java -Xms24952m -Xmx24952m -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemory'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe7253fb700 (LWP 5078))]
Installing openjdk unwinder
(gdb) bt
#0 0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fe7a6d0e02a in __GI_abort () at abort.c:89
#2 0x00007fe7a6012db9 in os::abort(bool) (dump_core=<optimized out>) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
#3 0x00007fe7a61c6897 in VMError::report_and_die() (this=this@entry=0x7fe7253f7e20)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
#4 0x00007fe7a61c6fb7 in crash_handler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8070, ucVoid=0x7fe7253f7f40)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/vmError_linux.cpp:106
#5 0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fe7a6e1a287 in __GI__dl_addr (symbolp=0x0, mapp=0x0, info=0x7fe7253f83f0, match=0x7fe6f0001800, addr=140630735942332) at dl-addr.c:61
#7 0x00007fe7a6e1a287 in __GI__dl_addr (address=0x7fe7250f72bc, info=0x7fe7253f83f0, mapp=0x0, symbolp=0x0) at dl-addr.c:137
#8 0x00007fe7a6018b49 in os::find(unsigned char*, outputStream*) (addr=addr@entry=0x7fe7250f72bc "\205W\255Ox\361\030\233\067O\200\262\266\067\071_\321\357\335+\204\360\344_\223_", st=st@entry=0x7fe7253f8b90) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:5166
#9 0x00007fe7a600d61d in os::print_location(outputStream*, long, bool) (st=0x7fe7253f8b90, x=140630735942332, verbose=<optimized out>)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:1030
#10 0x00007fe7a601cfbb in os::print_register_info(outputStream*, void*) (st=0x7fe7253f8b90, context=0x7fe7253f8e00)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:852
#11 0x00007fe7a61c557a in VMError::report(outputStream*) (this=0x7fe7253f8c90, st=st@entry=0x7fe7253f8b90)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:542
#12 0x00007fe7a61c63eb in VMError::report_and_die() (this=this@entry=0x7fe7253f8c90)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:971
#13 0x00007fe7a601c29f in JVM_handle_linux_signal(int, siginfo_t*, void*, int) (sig=sig@entry=11, info=info@entry=0x7fe7253f8f30, ucVoid=ucVoid@entry=0x7fe7253f8e00, abort_if_unrecognized=abort_if_unrecognized@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
#14 0x00007fe7a600fef8 in signalHandler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8f30, uc=0x7fe7253f8e00)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4346
#15 0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#16 0x00007fe7a72b9cda in check_match (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", ref=ref@entry=0x0, version=version@entry=0x0, flags=flags@entry=2, type_class=type_class@entry=0, sym=sym@entry=0x6f8, symidx=44, strtab=0x7a0 <error: Cannot access memory at address 0x7a0>, map=0x7fe6f0001800, versioned_sym=0x7fe7253f9338, num_versions=0x7fe7253f9334) at dl-lookup.c:92
#17 0x00007fe7a72ba64b in do_lookup_x (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", new_hash=new_hash@entry=1336760197, old_hash=old_hash@entry=0x7fe7253f9400, ref=0x0, result=result@entry=0x7fe7253f9410, scope=<optimized out>, i=0, version=0x0, flags=2, skip=0x0, type_class=0, undef_map=0x7fe6f0001800) at dl-lookup.c:423
#18 0x00007fe7a72bab1f in _dl_lookup_symbol_x (undef_name=0x7fe6f00076b0 "Agent_OnAttach", undef_map=0x7fe6f0001800, ref=0x7fe7253f94e8, symbol_scope=0x7fe6f0001b88, version=0x0, type_class=0, flags=2, skip_map=0x0) at dl-lookup.c:829
#19 0x00007fe7a6e1a873 in do_sym (handle=0x7fe6f0001800, name=0x7fe6f00076b0 "Agent_OnAttach", who=<optimized out>, vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:168
#20 0x00007fe7a6e1ad1d in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at dl-sym.c:273
#21 0x00007fe7a68ba034 in dlsym_doit (a=a@entry=0x7fe7253f96f0) at dlsym.c:50
#22 0x00007fe7a72c0564 in _dl_catch_error (objname=0x7fe6f00017c0, errstring=0x7fe6f00017c8, mallocedp=0x7fe6f00017b8, operate=0x7fe7a68ba020 <dlsym_doit>, args=0x7fe7253f96f0) at dl-error.c:187
#23 0x00007fe7a68ba571 in _dlerror_run (operate=operate@entry=0x7fe7a68ba020 <dlsym_doit>, args=args@entry=0x7fe7253f96f0) at dlerror.c:163
#24 0x00007fe7a68ba088 in __dlsym (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach") at dlsym.c:70
#25 0x00007fe7a60138f4 in os::dll_lookup(void*, char const*) (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach")
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:2052
#26 0x00007fe7a600c34c in os::find_agent_function(AgentLibrary*, bool, char const**, unsigned long) (agent_lib=agent_lib@entry=0x7fe6f00079a0, check_lib=check_lib@entry=false,---Type <return> to continue, or q <return> to quit---
syms=syms@entry=0x7fe7253f9860, syms_len=syms_len@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:490
#27 0x00007fe7a5e8f5f3 in JvmtiExport::load_agent_library(AttachOperation*, outputStream*) (op=0x7fe6f0000a30, st=0x7fe7253fad20)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/prims/jvmtiExport.cpp:2252
#28 0x00007fe7a5a43db1 in attach_listener_thread_entry(JavaThread*, Thread*) (thread=<optimized out>, __the_thread__=<optimized out>)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/services/attachListener.cpp:453
#29 0x00007fe7a616aa07 in JavaThread::thread_main_inner() (this=this@entry=0x7fe708001000)
at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1699
#30 0x00007fe7a616aecc in JavaThread::run() (this=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1679
#31 0x00007fe7a6011d82 in java_start(Thread*) (thread=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:790
#32 0x00007fe7a66a36ba in start_thread (arg=0x7fe7253fb700) at pthread_create.c:333
#33 0x00007fe7a6dde3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Hello,
After succesfully creating flamegraphs a few days ago I tried it again today and only get this error:
Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/attach/AgentInitializationException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.AgentInitializationException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more
I haven't changed anything on the machine. Is there a solution to this or a way to get more error information?
Hi
I'm trying to use your project on a fedora 24 using openjdk 1.8.0.101 . The problem can be seen using perf-java-top and flame graphs as well . The create-java-perf-map seems to work well but it did saw how could get its output with perf to plot with frame. Jstack works as well (but as you mention it would be a terrible poor performance and quality).
I thought it would be something with preserving the framepointer but the jdk version is compatible and I'm using the following JVM options:
-XX:+PreserveFramePointer -XX:+StartAttachListener -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:-OmitStackTraceInFastThrow -XX:+ShowHiddenFrames
Do you have any suggestions about how to solve this problem?
Thanks in advance
Luan
Hi
I have ran cmake. Some error happened. Do you have any idea?
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- JAVA_INCLUDE_PATH=/usr/lib/jvm/java-7-openjdk-amd64/include
-- JAVA_INCLUDE_PATH2=/usr/lib/jvm/java-7-openjdk-amd64/include
CMake Error at /usr/share/cmake-2.8/Modules/UseJava.cmake:269 (string):
string sub-command LENGTH requires two arguments.
Call Stack (most recent call first):
CMakeLists.txt:30 (add_jar)
-- Configuring incomplete, errors occurred!
For some methods there can be some chunks of code for which proper inline information seems to be missing (compared to PrintAssembly
output).
I don't think there can be done anything about that realistically inside of perf-map-agent but if we find some clues why that would be the case it could be reported to openjdk (maybe it is but I couldn't find it).
For reference, here are the code locations where this info is gathered:
Through the agent/JVMTI interface:
In contrast, for PrintAssembly
:
From here:
to finally:
From my quick investigation, the issue might be the way that scopes_pcs_begin
reports interesting (= scoped) code regions, in contrast to what scopes are reported if you consider every single PC of an nmethod.
Hi team,
We have our application running inside a container. If thats the case the map-agent should be installed outside or inside the container. I presume it should be done inside the container because the jdk we use is referenced inside. Could you please review and let us know the ideal approach.
Sattish.
Update scripts to detect jsvc
process in addition to java
. I may propose a patch but this will take me a few days.
I think PMA is in a unique position to break new grounds in Java profiling by reporting the Java stack while highlighting the separation between real frames and virtual frames(inlined methods). To do this we will need:
hello!
i have a question about perf
i ran the command as below
perf script|./stackcollapse-perf.pl |./flamegraph.pl --color=java --hash > flamegraph1.svg
system show me
Failed to open /usr/lib64/libc-2.17.so;56dd2155, continuing without symbols
Failed to open /usr/lib64/libpthread-2.17.so;56dd2155, continuing without symbols
os:CentOS Linux release 7.2.1511 (Core)
cmake:cmake version 2.8.11
Currently PMA fails or succeeds silently. When things are good this is a non-issue, when something doesn't work you probably want to see a bit more output.
I've played around with the idea and would like to log:
Ideas and feedback before I turn this into a PR?
Hi all,
I'm trying to profile systems that consist of lots of java processes being started and stopped over time. Does anyone know of a way to ensure that I can get a symbol mapping of each java process that gets run on my system? Just running once won't capture these transient processes, so I'd ideally like something that automatically makes a symbol dump, maybe on process termination? Any thoughts would be appreciate.
Not sure if this is the right place to ask, feel free to point me somewhere else!
Thanks a lot for this tool.
The released pre-built binary got slightly old. https://github.com/jvm-profiling-tools/perf-map-agent/releases
For automated usecases, pre-built binary from official repository is helpful (especially in non-tech meaning).
Build is passing, so making new release should be easy.
Would like an option to disable full method signatures. Eg, "nomsig". So this:
7feca143ba60 164c0 Lio/netty/handler/codec/http/DefaultHttpHeaders;.set(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio
would become:
7feca143ba60 164c0 Lio/netty/handler/codec/http/DefaultHttpHeaders;.set
HI guys!
I have produced a flame graph with:
PERF_COLLAPSE_OPTS="--kernel --tid" PERF_RECORD_FREQ=99 PERF_RECORD_SECONDS=10 PERF_MAP_OPTIONS=unfoldall ~/perf-map-agent/bin/perf-java-flames <pid>
But seems that (some) C stack frames are not correctly shown as children of the related Java calls.
Am I missing anything in the configuration?
Thanks,
Franz
I'm trying to create a Java process FlameGraph with perf-java-flames
. It seems to run successfully, but I can't find the resulting svg file.
$ ./perf-java-flames 161991 -F 99 -a -g -- sleep 30
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
Warning:
PID/TID switch overriding SYSTEM
$
CentOS 7
3.10.0-327.36.3.el7.x86_64
cmake version 2.8.12.2
Up-to-date versions of perf-map-agent and the FlameGraph repo.
Hi,
It's impossible to record plain PMU events like mispredicted branches with perf-java-recors-stack. This script supports profiling with specified frequency only (-F argument).
I'd like to propose either add one more script (like perf-java-record-stacks) which is able to record PMU events or modify existing perf-java-record-stack script in order to let user to specify PMU event to profile. If you are ok for these changes, I would implement it.
--Sergey
When I try to launch my jar with the agentpath set to attach-main.jar, I get
Failed to find Premain-Class manifest attribute in /perf-map/out/attach-main.jar
Any ideas?
Hi,
I tried to use perf-map-agent as pure JVMTI agent via -agentlib argument. But the current implementation doesn't have Agent_OnLoad method.
So, does it worth to implement Agen_OnLoad function (I'm ready to do it)? If so, should I handle a situation when compiled method was abandoned (is it possible to handle these situations?)?
--Sergey
Really a newbie question
When I use the agent on a perf record command line and then run perf report, I just see some built-in methods ,like System.out.prinln, but it does not print other methods ,such as , "main, print_rectangle(a user-defining method which just prints some "*")". I follow the steps on your your blog Java in Flames ( 1. install perf-map-agent , 2. profiling and flame graph generation:).
When I run perf record -F 49 -a -g -- sleep 30; ./FlameGraph/jmaps , I see the perf-5084.map file.
The perf report shows some unknown info:
Samples: 331 of event 'cycles', Event count (approx.): 3811883239
Children Self Command Shared Object Symbol ◆
-- 69.94% 0.00% java [unknown] [.] 0xb3f180e200026c80 ▒
-- 0xb3f180e200026c80 ▒
81.85% 0 ▒
18.15% 0x8000268d8 ▒
-- 69.13% 1.37% java perf-7341.map [.] Ljava/io/FileOutputSt▒
-- Ljava/io/FileOutputStream;::writeBytes ▒
++ 81.75% Ljava/io/PrintStream;::print ▒
++ 18.25% Lsun/nio/cs/StreamEncoder;::flushBuffer ▒
++ 67.76% 0.00% java libjava.so [.] Java_java_io_FileOutp▒
++ 66.79% 0.00% java libpthread-2.17.so [.] 0xffff80d8d849543d ▒
++ 63.03% 6.52% java perf-7341.map [.] Ljava/io/PrintStream;▒
++ 62.09% 0.00% java libjava.so [.] 0xffff80d8dabfe1f0 ▒
++ 56.92% 0.00% java [unknown] [.] 0000000000000000 ▒
++ 56.66% 0.69% java [kernel.kallsyms] [k] sys_write ▒
++ 56.30% 0.00% java [kernel.kallsyms] [k] system_call_fastpath ▒
++ 51.99% 1.85% java [kernel.kallsyms] [k] vfs_write
My java version is 1.8.0.131. And up to date perf-map-agent repo~
There is a known issues section, but it is reasonably easy to not generate the symbols for a number of reasons.
These problems should be fairly easy to deal with if documented in troubleshooting steps/procedure.
Sure they will be documented in the issues but not everyone reads the issues and then if needed wastes a roundtrip as we ask for diagnostic information.
➜ perf-map-agent git:(master) ✗ make
-- JNI_INCLUDE_DIRS=/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers
-- JNI_LIBRARIES=/System/Library/Frameworks/JavaVM.framework;/System/Library/Frameworks/JavaVM.framework
-- JAVA_INCLUDE_PATH=/System/Library/Frameworks/JavaVM.framework/Headers
-- JAVA_INCLUDE_PATH2=/System/Library/Frameworks/JavaVM.framework/Headers
-- Configuring done
CMake Warning (dev):
Policy CMP0042 is not set: MACOSX_RPATH is enabled by default. Run "cmake
--help-policy CMP0042" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.
MACOSX_RPATH is not specified for the following targets:
perfmap
This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done
-- Build files have been written to: /Users/xiezhaodong/Downloads/perf-profile/perf-map-agent
[ 16%] Building Java objects for attach-main.jar
[ 33%] Generating out/CMakeFiles/attach-main.dir/java_class_filelist
[ 50%] Creating Java archive attach-main.jar
[ 50%] Built target attach-main
[ 66%] Building C object CMakeFiles/perfmap.dir/src/c/perf-map-agent.c.o
/Users/xiezhaodong/Downloads/perf-profile/perf-map-agent/src/c/perf-map-agent.c:30:10: fatal error: 'jvmticmlr.h' file not found
#include <jvmticmlr.h>
^
1 error generated.
make[2]: *** [CMakeFiles/perfmap.dir/src/c/perf-map-agent.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2
I change CMakelist.text line set(CMAKE_JAVA_INCLUDE_PATH /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/lib/tools.jar)
Ran ./bin/dtrace-java-record-stack but the command does not work. I think the dtrace command inside the ./bin/dtrace-java-record-stack is not used properly.
cat /tmp/dtrace-25840.data
Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I]] -s File.d [-o ]
Hi,
I'm seeing some methods being unexpectedly listed high on perf top, which are called very infrequently. I have cross-checked with a JMC flight recording (which I assume to be quite accurate), and these paths were not showing up there at all.
Is there a chance that the map is somehow inaccurate, perhaps over time with hotspot recompiling / deoptimizing / moving stuff around?
Btw, this agent is quite useful, keep up the good work :)
Regards,
Viktor
# ./bin/perf-java-top 19835
Error: Could not find or load main class net.virtualvoid.perf.AttachOnce
scripts expects attach-main.jar
to be found in out
dir, while it is in the project root dir.
Hi guys,
I used Java(TM) SE Runtime Environment (build 1.8.0_60-ea-b25)
and run ./create-java-perf-map.sh $pid
got:
Exception in thread "main" com.sun.tools.attach.AgentLoadException: Failed to load agent library
at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentPath(HotSpotVirtualMachine.java:88)
at net.virtualvoid.perf.AttachOnce.loadAgent(AttachOnce.java:39)
at net.virtualvoid.perf.AttachOnce.main(AttachOnce.java:33)
Any idea?
Thanks so much.
Hello, Authors,
Thank you to open-source this projects. The agent and handy scripts are extremely helpful for me.
I have a problem about JIT-ed code. I end up with a lot of PCs which point to nowhere but fall in the range of map files. I interpret it in this way: the symbols in the map file are not accurate.
I have a question about this line. It seems that two statements happen in sequence.
perf-map-agent/bin/perf-java-record-stack
Line 25 in d9843a0
That is to say, the script generates map file right after 'perf record' has done. Why not make them run in parallel? In current model, will agent has a skewed view? or I have to make sure my profiling program enter a stable state ?
Thanks for your small but very useful tool. I've been working with the latest commits from master on the tool and I hit some issues mainly related to how the paths are solved and the main JAR file is executed. Here is a suggest to improve a bit for simpler usage:
diff --git a/bin/create-java-perf-map.sh b/bin/create-java-perf-map.sh
index 604fa44..b83f342 100755
--- a/bin/create-java-perf-map.sh
+++ b/bin/create-java-perf-map.sh
@@ -6,8 +6,7 @@ CUR_DIR=`pwd`
PID=$1
OPTIONS=$2
ATTACH_JAR=attach-main.jar
-PERF_MAP_DIR=$(dirname $(readlink -f $0))/..
-ATTACH_JAR_PATH=$PERF_MAP_DIR/out/$ATTACH_JAR
+PERF_MAP_DIR=$(realpath $(dirname $(readlink -f $0))/..)
PERF_MAP_FILE=/tmp/perf-$PID.map
if [ -z $JAVA_HOME ]; then
@@ -18,5 +17,5 @@ fi
[ -d $JAVA_HOME ] || (echo "JAVA_HOME directory at '$JAVA_HOME' does not exist." && false)
sudo rm $PERF_MAP_FILE -f
-(cd $PERF_MAP_DIR && java -cp $ATTACH_JAR_PATH:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID $OPTIONS)
+(cd $PERF_MAP_DIR/out && java -cp $ATTACH_JAR:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID $OPTIONS)
sudo chown root:root $PERF_MAP_FILE
diff --git a/bin/perf-java-flames b/bin/perf-java-flames
index eb7c235..4a3d4ee 100755
--- a/bin/perf-java-flames
+++ b/bin/perf-java-flames
@@ -26,7 +26,7 @@ if [ -z $PERF_DATA_FILE ]; then
fi
if [ -z $PERF_FLAME_OUTPUT ]; then
- PERF_FLAME_OUTPUT=flamegraph-$PID.svg
+ PERF_FLAME_OUTPUT=$PERF_JAVA_TMP/flamegraph-$PID.svg
fi
$PERF_MAP_DIR/bin/perf-java-record-stack $*
Summary:
ATTACH_JAR_PATH
.realpath
for more correctly resolve the base directory of perf-map-agent
.PERF_FLAME_OUTPUT
to $PERF_JAVA_TMP/flamegraph-$PID.svg
if not already set.Hi,
I can't seem to build most of the targets on El Capitan.
$ cmake --version
cmake version 3.5.20160411-g9ac11
CMake suite maintained and supported by Kitware (kitware.com/cmake)
$ cmake .
-- JNI_INCLUDE_DIRS=/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers
-- JNI_LIBRARIES=/System/Library/Frameworks/JavaVM.framework;/System/Library/Frameworks/JavaVM.framework
-- JAVA_INCLUDE_PATH=/System/Library/Frameworks/JavaVM.framework/Headers
-- JAVA_INCLUDE_PATH2=/System/Library/Frameworks/JavaVM.framework/Headers
-- LIBS: /Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin/jar incl: /System/Library/Frameworks/JavaVM.framework/Headers/../lib/tools.jar output_dir: /Users/username/code/perf-map-agent/out
-- Configuring done
CMake Warning (dev):
Policy CMP0042 is not set: MACOSX_RPATH is enabled by default. Run "cmake
--help-policy CMP0042" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.
MACOSX_RPATH is not specified for the following targets:
perfmap
This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done
-- Build files have been written to: /Users/username/code/perf-map-agent
$ make --debug
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
File `default_target' does not exist.
File `all' does not exist.
File `cmake_check_build_system' does not exist.
Must remake target `cmake_check_build_system'.
Successfully remade target file `cmake_check_build_system'.
Must remake target `all'.
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
File `all' does not exist.
File `CMakeFiles/attach-main.dir/all' does not exist.
Must remake target `CMakeFiles/attach-main.dir/all'.
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
File `CMakeFiles/attach-main.dir/depend' does not exist.
Must remake target `CMakeFiles/attach-main.dir/depend'.
Successfully remade target file `CMakeFiles/attach-main.dir/depend'.
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
File `CMakeFiles/attach-main.dir/build' does not exist.
File `attach-main' does not exist.
File `CMakeFiles/attach-main' does not exist.
File `out/attach-main.jar' does not exist.
File `out/CMakeFiles/attach-main.dir/java_compiled_attach-main' does not exist.
Must remake target `out/CMakeFiles/attach-main.dir/java_compiled_attach-main'.
[ 16%] Building Java objects for attach-main.jar
src/java/AttachOnce.java:25: error: package com.sun.tools.attach does not exist
import com.sun.tools.attach.VirtualMachine;
^
src/java/AttachOnce.java:37: error: cannot find symbol
VirtualMachine vm = VirtualMachine.attach(pid);
^
symbol: class VirtualMachine
location: class AttachOnce
src/java/AttachOnce.java:37: error: cannot find symbol
VirtualMachine vm = VirtualMachine.attach(pid);
^
symbol: variable VirtualMachine
location: class AttachOnce
src/java/AttachOnce.java:46: error: package com.sun.tools.attach does not exist
} catch(com.sun.tools.attach.AgentInitializationException e) {
^
4 errors
make[2]: *** [out/CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 1
make[1]: *** [CMakeFiles/attach-main.dir/all] Error 2
make: *** [all] Error 2
Possible newbie question...
When I use the agent on a perf record command line and then run perf report, I do indeed see the JITted methods listed with event percentages, etc. But when I hit RETURN and try to drill down into a JITted method, no code is shown.
Hi Johannes,
I'm trying to profile a Java program (SPECjbb2015) using linux perf + your perf-map-agent.
Do you think of any (possible) mistake I'm making ?
Or any options I should try ?
Thank you very much
Hitoshi
In one of my cluster, I can attach perf-map agent and get the symbol dump without issue.
I can record "perf record -F 99 -ag" and generate script via "perf script". However, it lists only one line for the JVM stack (the top frame in the stack). JVM is started with "-XX:-OmitStackTraceInFastThrow".
Has anyone observed something like this?
This is probably a JVM issue, but I'll file it here anyway as it's something anyone may run into when using perf-map-agent.
Many stacks that perf reports look partial/incomplete, showing the top frame only. Eg, the following output is from "perf report --stdio":
0.78% java perf-22919.map [.] Lio/netty/handler/codec/http/DefaultHttpHeaders;.add(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio/netty/handler/codec/
|
--- Lio/netty/handler/codec/http/DefaultHttpHeaders;.add(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio/netty/handler/codec/http/HttpHeaders;
0.78% java perf-22919.map [.] Lrx/subjects/PublishSubject;.onNext(Ljava/lang/Object;)V
|
--- Lrx/subjects/PublishSubject;.onNext(Ljava/lang/Object;)V
0.78% java perf-22919.map [.] Lio/netty/channel/AbstractChannel$AbstractUnsafe;.write(Ljava/lang/Object;Lio/netty/channel/ChannelPromise;)V
|
--- Lio/netty/channel/AbstractChannel$AbstractUnsafe;.write(Ljava/lang/Object;Lio/netty/channel/ChannelPromise;)V
It looks a lot like omit-frame-pointer optimization, however, I believe this is from hotspot. If there is a hotspot equivalent option for -fno-omit-frame-pointer, I haven't found it yet.
I am using:
-XX:-OmitStackTraceInFastThrow -XX:+UnlockDiagnosticVMOptions -XX:+ShowHiddenFrames
I often see "Interpreter" frames. I think I was expecting more symbols.
100.00% java perf-20844.map [.] Interpreter
|
--- Interpreter
Interpreter
Interpreter
Interpreter
call_stub
JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)
jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
jni_CallStaticVoidMethod
JavaMain
start_thread
Challenges:
http://openjdk.java.net/jeps/220
some changes needed not limited to:
the tools.jar has gone. change create-java-perf-map.sh to find the classes via add-modules
(cd perf-map-agent/out/ && java --add-modules jdk.attach -jar ./attach-main.jar $(pidof java ) )
During our code profiling tests on 16-core systems running 8-16 worker threads, we are observing a significant CPU time (25%) in "vtable chunks" based on the generated symbol map. I think this symbol is listed multiple times as well in the symbol map. This overhead also increases as the number of running threads is gradually raised from 8 to 16.
Is this an overhead of perf-map-agent or something else ? Would like to also confirm if its related to virtual methods in our code.
Another unrelated issue is that perf top always displays "unable to open vsyscalls". Is there a way to resolve it. Does this mean that perf is unable to display system calls and their CPU time ? All processes are being executed by the "root" user.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.