Changing num_warps results in different numerics about triton HOT 4 CLOSED

Hi Tom!

After having looked at it, this is actually not strictly a bug. The backward pass modifies neg_logprobs in-place, so the auto-tuner will keep modifying the buffer over and over again as it tries different configurations to run. If you set num_warps=[2] or num_warps=[4], it will also work. I would suggest you disable auto-tuning for now. We should leave this issue open until a solution is implemented

Probably the solution will look something like:
triton.kernel( ... , autotune_clone = 'neg_logprobs'), then the auto-tuner will just clone automatically neg_logprobs before running.

I hope at some point pointers modified in-place can be determined automatically from static analysis.

from triton.

This is an old issue but still not addressed actually. I remember having a patch ready at some point so I'll dig in to see if I can find it somewhere. Now that the JIT API is back to taking pure torch.tensor, it shouldn't be too hard to clone them before the auto-tuner runs.

from triton.

Awesome!

from triton.

Should have been fixed in #205 . Now you can provide reset_to_zero(list_of_arg_names) to the auto-tuner, and it will reset all arguments of provided name to zero before each benchmark run. Not ideal, but should fix this issue

from triton.