Comments (8)
Yes, it will be a drop-in replacement for Flux's current AD, and may eventually become the default.
from zygote.jl.
So the idea with Zygote is to play around with source-to-source AD techniques which are currently not in Capstan's focus. Jarrett is focused on a fairly different (and currently incompatible) implementation strategy, mainly because he has concerns about Zygote's ability to scale to handle larger programs. Right now, Zygote and Capstan will have to exist as alternatives, but Capstan is intended to become more pluggable so that it'll become possible to integrate whatever parts of Zygote work well in future.
@jrevels might wish to say if that's a fair summary, or elaborate on his concerns.
from zygote.jl.
On flux master you can do params(m)
and pass this to gradient
following Zygote's API. e.g.
m = Chain(...)
gradient(params(m)) do
m(input)
end
I haven't documented this yet because it doesn't have great coverage of Flux yet. But feel free to try it and open issues for anything you run into, and that'll help me prioritise fixes.
from zygote.jl.
Got it, thanks.
I assume you've also seen Google's Tangent? It also does source to source AD.
from zygote.jl.
Yup, Tangent is a good mental model for how Zygote is working as well. The main difficulty with doing this in Python is that you have to pretend you're working with a sufficiently-static subset of Python where you can resolve what everything actually does – otherwise you'd have to look up gradient definitions at runtime. Hence Tangent's various limitations with respect to closures, classes etc.
Zygote is fully dynamic, which works because we can actually look up gradients dynamically (via dispatch) and have Julia resolve that at compile time where possible (as usual).
Another difference is that Zygote works on SSA form IR rather than Julia source code. We don't aim for readable output in any way, but are able to support a lot of the language in a uniform way (e.g. arbitrary control flow including @goto
).
from zygote.jl.
@MikeInnes is Zygote going to be an alternative AD backend for Flux just like what you expect for capstan and cassette?
from zygote.jl.
How far are we along this path? I would like to use Zygote instead of Tracker in Flux but so far I haven't found a way of collecting parameters from more complex, i.e., chained models. But maybe I'm missing something obvious..
from zygote.jl.
Given it's been almost 2 years since Zygote became the default AD in Flux, this should be safe to close :)
from zygote.jl.
Related Issues (20)
- `repeat(X; outer, inner)` triggers scalar indexing error with CUDA HOT 1
- Missing support for muladd in case of brodcasting with a complex argument HOT 1
- `nothing` in output of a `pullback` HOT 2
- Assignment to multiple arrays is not differentiable on GPU since Zygote.jl 0.6.67 HOT 5
- Spurious "Output is complex, so the gradient is not defined" error HOT 2
- NaN in gradient of abs() on complex 0 HOT 1
- Pullback on mean() gives illegal memory access code 700 HOT 31
- test
- Type unstable gradients (@code_warntype) HOT 1
- Type unstable gradients HOT 1
- Zygote gradients different from ForwardDiff/ReverseDiff on Julia 1.10-rc2 HOT 3
- try/catch is not supported when attempting to use `remake` with Zygote HOT 1
- gradient of SVD not working for complex input HOT 1
- `Zygote` doesn't properly work with `Metal.jl` and half precision. HOT 4
- `gradient` broken for `(*)(::Diagonal{Real}, ::Matrix{Complex}, ::Diagonal{Real})` when updating Julia 1.8 -> 1.9 HOT 6
- Method ambiguities reported by Aqua
- slow/high allocation gradient with mapreduce and iterators HOT 11
- error in summation of product iterator HOT 2
- `sort(x; rev=true)` is not supported HOT 1
- Incorrect gradients for `plan_rfft(x) * x` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zygote.jl.