Comments (4)
Yes I think this can be considered along with #35.
from abstractdifferentiation.jl.
It seems annoying that the pushforward and pullback function accept tuples of co-tangents instead of multiple arguments. Is there a compelling reason for doing so or was this a design decision that could be changed?
While Julia functions may take multiple inputs, no Julia function returns multiple outputs. Instead, they might return a tuple of outputs. The (co)tangent of a tuple is like a tuple itself. FWIW, this is consistent with how ChainRules behaves, hence why in ChainRules, it would be represented as a Tangent
, and here it would be represented as a tuple. One could make the case that since AD.jl supports only functions whose inputs and outputs are arrays, then if such a function returns a tuple it can only be interpreted as multiple outputs, but that would be inconsistent at least with ChainRules and Zygote.
it also makes it impossible to work with actual single-argument functions that take a tuple as only argument but maybe this is not needed anyway
I don't think function with tuple inputs would be supported anyways.
from abstractdifferentiation.jl.
While Julia functions may take multiple inputs, no Julia function returns multiple outputs. Instead, they might return a tuple of outputs. The (co)tangent of a tuple is like a tuple itself. FWIW, this is consistent with how ChainRules behaves, hence why in ChainRules, it would be represented as a Tangent, and here it would be represented as a tuple. One could make the case that since AD.jl supports only functions whose inputs and outputs are arrays, then if such a function returns a tuple it can only be interpreted as multiple outputs, but that would be inconsistent at least with ChainRules and Zygote.
Sure, multiple outputs are in fact just a tuple of outputs. But it does not necessarily mean that we have to use a tuple as input to the pullback and pushforward function.
The current design is also not completely consistent with ChainRules: In ChainRules one does not have to consider tuples of co-tangents of length 1 - the pullback function of a function with a single output just takes a single co-tangent without wrapping it as a tuple. Neglecting/not supporting tuples of length 1 would already solve the special case in #51, even if we stick with tuples in case of multiple outputs.
from abstractdifferentiation.jl.
I think in general AD.jl has a funny relationship with inputs and outputs. Like gradient
for a single input returns a tuple, and hessian
only supports single inputs and yet still returns a tuple. IMO this should be changed.
The pushforward of a function (talking about the actual pushforward, not the fusion of the pushforward and primal that frule
encodes) should be structured the same as the primal in terms of inputs and outputs. The pullback is the adjoint of the pushforward and vice versa, so a useful check of consistency is whether the rules we choose are symmetric.
i.e., these rules would maintain this symmetry, and perhaps they make sense:
- The adjoint of a single-argument function returns a single output (not a tuple)
- The adjoint of a multi-argument function returns a tuple of outputs
- The adjoint of a single-output (non-tuple) function takes a single input
- The adjoint of a multi-output (tuple) function takes multiple inputs
This is almost consistent with ChainRules, the key differences being that 1) in ChainRules, the function is treated as an argument, so there are no single-argument functions (or at least, I don't know of any examples where a rule is defined for a 0-argument function), hence all pullbacks return tuples and 2) a function might actually return a tuple directly, so it's not safe to interpret a tuple return value as being multiple outputs.
from abstractdifferentiation.jl.
Related Issues (20)
- `AD.jacobian` much slower than `Zygote.jacobian` HOT 3
- JET.jl reports possible errors on `AD.gradient` that do not appear with `Zygote.gradient` HOT 3
- Code inside function with rrule should not run HOT 2
- Using AbstractDifferentiation and Zygote as package dependencies errors: `UndefVarError: ZygoteBackend not defined` HOT 1
- Jacobians for functions beyond vector-to-vector?
- Whats the reason for the derivative / gradient difference?
- AD failure where Zygote succeeds HOT 3
- ForwardDiff is broken for differently sized inputs
- API for user code to detect if it's being differentiated HOT 11
- Feature request: option to turn off ForwardDiff tagging
- Zygote context cache incorrectly(?) persists between AD calls
- Missing `value_derivative_and_second_derivative`?
- Replace Requires.jl with conditional dependencies for Julia 1.9 HOT 2
- LoadError: UndefVarError: StaticArrays not defined HOT 2
- add `ForwardRuleConfigBackend` (to match `ReverseRuleConfigBackend`) HOT 7
- Lazy Jacobian multiplication HOT 7
- ERROR: MethodError: no method matching ZygoteBackend() HOT 1
- `lazy_jacobian` is less generic than `pullback_function`
- Enzyme support HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abstractdifferentiation.jl.