When execute tensorflow minist train task, occur the problem, 'Check failed: cublas_handle'.

It caused by dlopen, the complete command is dlopen("/usr/local/cuda/targets/x86_64-linux/lib/libcublas.so", RTLD_NOW | RTLD_LOCAL). And error throwed by dlopen is 'Failed to open /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so: /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so: symbol free_gemm_select version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference'.

According to ldd and nm, libcublas.so depend on libcublasLt.so.11, which linked to '/home/chenqian/Code/cuda_hook/output/lib64/libcublasLt.so.11'. And, there is no symbol free_gemm_select in both '/home/chenqian/Code/cuda_hook/output/lib64/libcublasLt.so.11' and '/usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.11'.

Moreover, if without cuda hook, the train task can complete.

Hello, I am learning cuda interception recently, can you tell me your contact information for convenience

I want to intercept more CUDA APIs through hooks, so I would like to use the generate_code.py in your repository to generate hook function source code for all .h files under the cuda/include directory.

It seems that it works with CUDA 11.6 and fails to work with CUDA 11.8 with Driver Version 525.85.12. Could you please fix it?

The following is the error message.
$ more log/sample/nvml/nvml_example.log
[CUDA-HOOK 71169:71169 hook.h:33 GetNVMLSymbol] check failed: nvml_handle

$ more matrix_mul.log
/home/ps/cuda_hook_retry/output/sample/cuda/matrix_mul: /home/ps/cuda_hook_retry/output/lib64/libcudart.so.11.0: no version information available (required by /home/ps/cuda
_hook_retry/output/sample/cuda/matrix_mul)
[CUDA-HOOK 71167:71167 hook.h:29 GetCUDASymbol] check failed: cuda_handle
[Matrix Multiply Using CUDA] - Starting...

Hi @Bruce-Lee-LY , Where or How to get the cuda_subset.h?

There is a version issue.
If I run nvidia-smi in the cuda_hook/output/lib64 directory, it returns the following error.
"Mismatch in versions between nvidia-smi and NVML.
Are you sure you are using nvidia-smi provided with the driver?"

In addition, the bandwidth_test shows the following error: cuda_hook/output/lib64/libcudart.so.11.0: no version information available (required by bandwidth_test)

After run "./build.sh -t Release -s ON -b OFF", I use "ldd" then find there is libcudart but no libcuda. In fact, hook libcuda is real work, how is this done when the libcuda not dynamic load in runtime. I find the related problem in stack overflow https://stackoverflow.com/questions/49971717/hooking-into-cuda-driver-api-calls

Description

I found that code_generate has bug when parse parameter with array type like "int x[]".

Reproduction

You can reproduce the bug using header file below:

void test_with_brackets_1(int x[]);

void test_with_brackets_2(int x[][5]);

The output of code_generate.py would be"

// auto generate 2 apis

#include "test_subset.h"
#include "hook.h"
#include "macro_common.h"
#include "trace_profile.h"

HOOK_C_API HOOK_DECL_EXPORT void test_with_brackets_1(int x) {
    HOOK_TRACE_PROFILE("test_with_brackets_1");
    using func_ptr = void (*)(int);
    static auto func_entry = reinterpret_cast<func_ptr>(HOOK_TEST_SYMBOL("test_with_brackets_1"));
    HOOK_CHECK(func_entry);
    return func_entry(x);
}

HOOK_C_API HOOK_DECL_EXPORT void test_with_brackets_2(int x) {
    HOOK_TRACE_PROFILE("test_with_brackets_2");
    using func_ptr = void (*)(int);
    static auto func_entry = reinterpret_cast<func_ptr>(HOOK_TEST_SYMBOL("test_with_brackets_2"));
    HOOK_CHECK(func_entry);
    return func_entry(x);
}

Recently, I have been studying this project and noticed that you manually removed the interception of cuGetProcAddress. May I ask why?

Hi,

after code_generation for 12.3
build.sh

I got matex_mul:

CUDA error at /opt/cuda_hook/sample/cuda/include/helper_cuda.h:739 code=36(cudaErrorCallRequiresNewerDriver) "cudaGetDeviceCount(&device_count)"

[root@xxxxbuild]# strings ./* | grep -i versi |grep 12
strings: Warning: './CMakeFiles' is a directory
CUDA_VERSION:STRING=12.3

Is this problem relate to cuda_subset.h ?