CUDA: Getting zero gradients about enzyme HOT 7 OPEN

Performance note, tou can (and should) remove the -fno-vectorize -fno-unroll-loops when using ClangEnzyme.

Regardless investigating.

Please update the guide https://enzyme.mit.edu/getting_started/CUDAGuide/.

from enzyme.

Indeed and in progress, see EnzymeAD/www#14

from enzyme.

Performance note, tou can (and should) remove the -fno-vectorize -fno-unroll-loops when using ClangEnzyme.

Regardless investigating.

from enzyme.

This does appear to be a memory error. Specifically because it thinks your input and output could overlap, it is caching each variable -- taking gigabytes per thread -- and thus clearly oom'ing.

Marking the two function inputs as restrict removes the caching, but something weird is still going on.

from enzyme.

This does appear to be a memory error. Specifically because it thinks your input and output could overlap, it is caching each variable -- taking gigabytes per thread -- and thus clearly oom'ing.

I also suspect there could be OOM, so I checked the execution with cuda-memcheck, and found nothing. (And just to confirm, I think Enzyme would also check for failed cuda memory allocation, right?) So I think there could be other errors happen even before the OOM. For example, if you are using 32-bit integers, the address or size can overflow for a gigabyte-sized buffer.

from enzyme.

Enzyme will assume allocation functions will succeed or assert (which apparently is not the case here). I'm guessing what could be happening is that the allocation itself crashes the kernel before any load/stores and perhaps cuda-memcheck doesn't check that?

But perhaps looking at an error code after the kernel launch could?

from enzyme.

Of course it is the best practice to check the error code after each kernel launch. You can check invalid kernel launch arguments by calling cudaGetLastError after the kernel launches, and check runtime error inside the kernel by checking the error code returned from a following synchronization call. In my practice, both are needed.

But AFAIK, cuda-memcheck not only checks each load/store, but also checks the error code from each kernel launch or CUDA API call.

from enzyme.