This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Couldn't think of a better place to raise this, so 🤷‍♂ .

Proposal

A long time ago Jarrett made DiffTests.jl. That package is useful for testing whether your AD tool works well on problems that are tricky to differentiate. What we don't currently have is a package to test whether your package is compatible with the API that ChainRules specifies because, when DiffTests was written, the Julia AD world only really believed in numbers and arrays of numbers.

In short, I would like a package that tells me: "does my AD tool work with this weird type input and it's differential.", for increasingly complicated set of types. Bonus points for producing a nice human-readable report at the end.

IIRC e.g. ForwardDiff, ReverseDiff, Tracker etc only believe in numbers and dense arrays of numbers. ForwardDiff2 is a bit more flexible, and Zygote is more flexible still. FiniteDifferencing should really be almost as flexible as Zygote on this front, but it's not clear that it currently is, and it's hard to know what it should look like for this to be the case. FiniteDiff also only believes in numbers and arrays of numbers. This range of capabilities provides strong motivation for this package.

To summarise, this package would help

• everyone to better understand the landscape,
• package authors can get a feel for where they sit,
• any time we find a type that feels hard to work with, we have a package that we can encode this in and test everything else against.

Benchmarking AD tools has come up a lot recently, and this seems like a good place to implement some benchmarks, in addition to "correctness" testing.

I was thinking that they should be micro-benchmarks, and the benchmarks themselves shouldn't depend on any functionality outside of Base and the standard libraries, with the possible exception of things that are needed to test supports for accelerators eg. CuArrays.jl. Equally these could be supported by typing things sufficiently abstractly 🤷

The first thing to do is figure out what it people actually care about the performance of. For example, I really care about broadcasting and operations involving linear algebra, but not so much about control flow, but I know that the Turing team has a different set of priorities. So perhaps if everyone could solicit what sorts of things they're interested in benchmarking, we can start to think about how to chop up tests. For example, there's a distinction between control-flow that depends on values and control-flow that doesn't from the perspective of reverse-mode AD, so we should probably be testing that kind of thing.

cc @vchuravy @yebai @oxinabox

It seems DiffTests 0.1.2, or more concretely #11, broke the ReverseDiff tests: JuliaDiff/ReverseDiff.jl#207

Ref: https://discourse.julialang.org/t/package-compatibility-caps/15301