Conflicts with other profiling tools about dcgm HOT 7 CLOSED

@FindHao,

If you are using DCP metrics (1001,1002...) then it's impossible to collect metrics at the same time other profiling tools are enabled. DCGM is able to temporarily pause and resume the DCP metrics collection via dcgmi profile pause/resume command to help is such situations.
Profiling and DCGM both lock the same primitive in the driver, so that does not depend on which GPU you run profiling or the DCGM - it's a system-wide lock.

from dcgm.

@FindHao,

If you are using DCP metrics (1001,1002...) then it's impossible to collect metrics at the same time other profiling tools are enabled. DCGM is able to temporarily pause and resume the DCP metrics collection via dcgmi profile pause/resume command to help is such situations. Profiling and DCGM both lock the same primitive in the driver, so that does not depend on which GPU you run profiling or the DCGM - it's a system-wide lock.

I see. Thanks for your explanation.

But if someone uses only one GPUs on the 8 GPUs cluster, other users can't profile other GPUs at the same time. It's not a smart policy I think. Do you have any suggestions?

from dcgm.

@FindHao,

Unfortunately, for pre-Hopper GPUs, there are few options to mitigate this. You could use virtualization with GPU-passthrough to avoid the lock (each VM would use its driver). Starting from H100, we introduced new hardware counters that allow monitoring and profiling simultaneously without competing for resources.

from dcgm.

@FindHao,

Unfortunately, for pre-Hopper GPUs, there are few options to mitigate this. You could use virtualization with GPU-passthrough to avoid the lock (each VM would use its driver). Starting from H100, we introduced new hardware counters that allow monitoring and profiling simultaneously without competing for resources.

Interesting! Glad to know H100 will solve this issue!
Thanks for your reply.

from dcgm.

I'd like to reopen this issue since I tested some new cases.

I tried to run a dcgmi command on a 8xA100 server. Someone else is using GPU0 with dcgm client. But when I run dcgmi dmon -e 1007 -i 5, the error is the same with the previous discussion. the third-party profiling module returned an unrecoverable error.

But the field id 155 works well. 155 is the field id for power usage and 1007 is the field id for FP32 active.

Does that means if one client use profile field ids others will can't run dcgm clients with profile field ids?

from dcgm.

@FindHao,

Which DCGM version are you using? Multiple dcgm dmon clients for DCP fields (1001...1014) were implemented sometime before version 2.1.0. An exception - there should be only one nv-hostengine process in the system, including embedded engines.

For such an error, we need to see the debug logs from the nv-hostengine to understand the reason. There are several subsystems involved. The fields other than 1001...1014 do not use profiling facilities to get values and thus are unaffected.

from dcgm.

I'm using 2.4.6.
I'll try to get the debug log from the server admin. Thanks for your explanations!

from dcgm.