Comments (10)
psychocoderHPC
commented on July 28, 2024
Found why /opt/rocm/rocprofiler is shown. rocprof -> /opt/rocm/rocprofiler/bin/rpl_run.sh is only a link to /opt/rocm...
Never the less the error message saying nothing to me :-(
from rocprofiler.
psychocoderHPC
commented on July 28, 2024
hip error 1011 means that a nullptr is passsed to an api function: hipRegisterApiCallback(0)
Now I understand what the error is saying but I still not know where this call is coming from. It must be something within rocprof.
Is it possible for a user to trace the error e.g. enable debug logs?
from rocprofiler.
eshcherb
commented on July 28, 2024
Do you use the standard HIP version from 2.10 release?
Could you send me output of the following command:
$ hipcc --version
You can use 'rocprof' with absolute path '/shared/opt/rocm/2.10/bin/rocprof'.
The 1011 HIP error is from 2.10 release:
hipErrorInvalidValue = 1011 (https://github.com/ROCm-Developer-Tools/HIP/blob/roc-2.10.0/include/hip/hip_runtime_api.h#L234)
And actually the error tells that on an attempt to register a callback for op-id '0', HIP returned error 'hipErrorInvalidValue' which looks quite strange which usually happens when HIP is built without tracing support.
Registering of NULL callback would cause the callback disabling.
from rocprofiler.
psychocoderHPC
commented on July 28, 2024
@eshcherb thanks for the explanation of error.
The hip version is
hipcc --version
HIP version: 2.10.19436-815a601
clang version 10.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
I will ask the guy who compiled the HIP environment for me. The strange thing is that it is working for me on the same system for a small example https://github.com/ComputationalRadiationPhysics/cupla/tree/dev/example/CUDASamples/matrixMul
I will provide today the steps to reproduce it and the system information via mail to Malaya, Nicholas from AMD via mail. I think the information will than somehow be push internally to you.
Since I work on an CRAY system where I can not publish all information to the public I hope that I can provide here the solution after the issue is solved so that other developers in the community can find it here.
from rocprofiler.
eshcherb
commented on July 28, 2024
Hi @psychocoderHPC , do I understand right that you have standard ROCm setup in '/opt/rocm' and non-standard in '/shared/opt/rocm/2.10'?
'rocprof' by default sets HIP_PATH to '/opt/rocm/hip' and the standard HIP version from ROCm release should have tracing enabled.
Could you try to set env HIP_PATH to custom location and to rerun your small example?
export HIP_PATH=/shared/opt/rocm/2.10/hip
To build HIP with enabled tracing you should use the following CMAKE parameters:
-DUSE_PROF_API=1 -DPROF_API_HEADER_PATH=<ROCm path>/roctracer/include/ext
And please check the CMAKE messages, it should print:
"Profiling API: <path to found prof_protocol.h header>"
from rocprofiler.
eshcherb
commented on July 28, 2024
Also from your previous message:
rocprof -> /opt/rocm/rocprofiler/bin/rpl_run.sh is only a link to /opt/rocm...
Actually '<ROCm path>/bin/rocprof' should be a relative link:
/opt/rocm/bin/rocprof -> ../rocprofiler/bin/rpl_run.sh
So profiler will be used from the same ROCm setup the 'rocprof' was called from.
I will fix the script to not set HIP_PATH and will check what is ROCm policy for choosing ROCm setup because the issue seems be related for two ROCm setups interference.
BTW you can use 'strace' to check which exactly HIP library was used:
$ rocprof strace <cmd-line>
from rocprofiler.
eshcherb
commented on July 28, 2024
Another way is to use LD_DEBUG env var.
I would appreciate if you can help to resolve the confusion with different behavior of your two tests.
Could you use LD_DEBUG env var to check which actual HIP library 'libhip_hcc.so' is used?:
$ LD_DEBUG=libs rocprof <cmdline> 2>&1 | grep hip_hcc
from rocprofiler.
psychocoderHPC
commented on July 28, 2024
matrix multiplication (profiling is working)
export LD_DEBUG=libs
mpiexec -n 1 /opt/rocm/bin/rocprof --hip-trace --timestamp on ./matrixMul 2>&1 | grep hip_hcc
...
20358: find library=libhip_hcc.so [0]; searching
20358: trying file=/opt/rocm/hcc/bin/../lib/libhip_hcc.so
20358: trying file=/opt/rocm//hip/lib/libhip_hcc.so
...
20358: search path=tls/x86_64:tls:x86_64: (RPATH from file /opt/rocm//hip/lib/libhip_hcc.so)
20358: calling init: /opt/rocm//hip/lib/libhip_hcc.so
20358: calling fini: /opt/rocm//hip/lib/libhip_hcc.so [0]
...
PIConGPU (profiling is not working)
export LD_DEBUG=libs
/opt/rocm/bin/rocprof --hip-trace --timestamp on ./bin/picongpu -d 1 1 1 -g 128 128 128 -s 100 --periodic 0 0 0 2>&1 | grep hip_hcc
...
38637: find library=libhip_hcc.so [0]; searching
38637: trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/tls/x86_64/libhip_hcc.so
38637: trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/tls/libhip_hcc.so
38637: trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/x86_64/libhip_hcc.so
38637: trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/tls/x86_64/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/tls/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/x86_64/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/cce/tls/x86_64/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/cce/tls/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/cce/x86_64/libhip_hcc.so
38637: trying file=/opt/cray/pe/lib64/cce/libhip_hcc.so
38637: trying file=/opt/cray/lib64/tls/x86_64/libhip_hcc.so
38637: trying file=/opt/cray/lib64/tls/libhip_hcc.so
38637: trying file=/opt/cray/lib64/x86_64/libhip_hcc.so
38637: trying file=/opt/cray/lib64/libhip_hcc.so
38637: trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/tls/x86_64/libhip_hcc.so
38637: trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/tls/libhip_hcc.so
38637: trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/x86_64/libhip_hcc.so
38637: trying file=/home/users/XXX/opt/install/boost_1_71_0/lib/libhip_hcc.so
38637: trying file=/usr/mpi/gcc/openmpi-4.0.0rc5/lib64/libhip_hcc.so
38637: trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/tls/x86_64/libhip_hcc.so
38637: trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/tls/libhip_hcc.so
38637: trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/x86_64/libhip_hcc.so
38637: trying file=/home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so
38637: search path=/opt/rocm/hcc/bin/../lib/tls/x86_64:/opt/rocm/hcc/bin/../lib/tls:/opt/rocm/hcc/bin/../lib/x86_64:/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: calling init: /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
38637: search path=/opt/rocm/hcc/bin/../lib (RPATH from file /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so)
from rocprofiler.
psychocoderHPC
commented on July 28, 2024
@eshcherb Thanks for your help. The problem was that libhip_hcc.so was used from /home/users/XXX/c-hip-clang-build/HIP/install/lib/libhip_hcc.so (thats the compiled HIP-clang location).
By setting LD_LIBRARY_PATH to /opt/rocm//hcc/lib:$LD_LIBRARY_PATH I was able get rid of the error hipRegisterApiCallback(0)
from rocprofiler.
psychocoderHPC
commented on July 28, 2024
To build HIP with enabled tracing you should use the following CMAKE parameters:
-DUSE_PROF_API=1 -DPROF_API_HEADER_PATH=/roctracer/include/ext
I am now able to profile my application.
This is very important, I missed it. I compiled now all by myself and added this flags.
from rocprofiler.
Related Issues (20)
- rocprofv2 session name is not recognized when used with mpiexec
- Is rocprofv2 part of ROCm? HOT 1
- Please add gfx1100 support for aqlprofie HOT 1
- /opt/rocm/libexec/rocprofiler/tblextr.py: ROCTX range begin not found, pid()
- Overlapping kernel in profiler trace HOT 1
- Do I need --parallel-kernels option for multi-kernel on multi-device scenario?
- Question: what is the recommended way to profile multi-gpu code?
- spack error building `[email protected]` HOT 3
- ROCr error when running papi_command_line with specific event
- Tool for reading CSV files? HOT 1
- Build rocprof source code error
- [Feature]: controlling the profiling scope
- [Feature]: add cmake imported target
- [Feature]: Ignore unsupported architectures HOT 1
- [Issue]: Unable to compile, receive error: ld.lld: error: relocation R_X86_64_32 HOT 1
- [Issue]: rocprofile doesn't work with dlopen hip
- [Issue]: what's correct way to use --plugins att auto? "Could not find att output kernel: ./*_kernel.txt" HOT 1
- [Issue]: rocprofv2 --hip-trace --roctx-trace --plugin perfetto, no ROCTX in Perfetto UI viewer
- rocprof bash sets PATH to the wrong location, then fails to find rocminfo
- [Issue]: rocprofv2 --plugin file --plugin-version 1 --kernel-trace -o file_name output csv header columns number doesn't match the number of data columns
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rocprofiler.