DifferentiationInterface testing about lux.jl HOT 4 OPEN

• https://github.com/LuxDL/Lux.jl/blob/main/test/enzyme_tests.jl should be a good starting point. There are other AD tests as well but they are sprinkled in other general numeric tests, so no point unnecessarily dabbling with those.
  • Once these work, I can try enabling DI in LuxTestUtils
• FluxML/Flux.jl#2469 (comment) the testing code here should work, just do it on the parameters instead of the model

from lux.jl.

Thanks! In the file you linked, the source of truth seems to be Zygote? What would you use to validate the Zygote gradients themselves?

from lux.jl.

Currently I compute with Zygote and then test against other backends based on the device

1. For CPU
   1. Tracker
   2. ReverseDiff
   3. ForwardDiff (if the array sizes are < 100)
   4. FiniteDifferences
   5. Enzyme (currently only tested in that file, but testing is being increased more here)
2. For GPU
   1. Tracker
   2. ForwardDiff in certain situations depending on the problem

Tracker and Zygote hit very different code paths in LuxLib (Zygote is the optimized one with often handwritten rules). In case of conflict/mismatch, the general assumption is that Tracker (on GPU) or FiniteDifferences (on CPU) is the source of truth.

from lux.jl.

I have been meaning to try out FiniteDiff to validate the GPU gradients but haven't had the time to set it up.

from lux.jl.