fluxml / zygote.jl Goto Github PK

View Code? Open in Web Editor NEW

1.4K 52.0 206.0 3.79 MB

21st century AD

Home Page: https://fluxml.ai/Zygote.jl/

License: Other

Julia 100.00%

julia julia-compiler control-flow gradient machine-learning automatic-differentiation

zygote.jl's Introduction

] add Zygote

Zygote provides source-to-source automatic differentiation (AD) in Julia, and is the next-gen AD system for the Flux differentiable programming framework. For more details and benchmarks of Zygote's technique, see our paper. You may want to check out Flux for more interesting examples of Zygote usage; the documentation here focuses on internals and advanced AD usage.

Zygote supports Julia 1.6 onwards, but we highly recommend using Julia 1.8 or later.

julia> using Zygote

julia> f(x) = 5x + 3

julia> f(10), f'(10)
(53, 5.0)

julia> @code_llvm f'(10)
define i64 @"julia_#625_38792"(i64) {
top:
  ret i64 5
}

"Source-to-source" means that Zygote hooks into Julia's compiler, and generates the backwards pass for you – as if you had written it by hand.

Zygote supports the flexibility and dynamism of the Julia language, including control flow, recursion, closures, structs, dictionaries, and more. Mutation and exception handling are currently not supported.

julia> fs = Dict("sin" => sin, "cos" => cos, "tan" => tan);

julia> gradient(x -> fs[readline()](x), 1)
sin
0.5403023058681398

Zygote benefits from using the ChainRules.jl ruleset. Custom gradients can be defined by extending the ChainRulesCore.jl's rrule:

julia> using ChainRulesCore

julia> add(a, b) = a + b

julia> function ChainRulesCore.rrule(::typeof(add), a, b)
           add_pb(dy) = (NoTangent(), dy, dy)
           return add(a, b), add_pb
       end

To support large machine learning models with many parameters, Zygote can differentiate implicitly-used parameters, as opposed to just function arguments.

julia> W, b = rand(2, 3), rand(2);

julia> predict(x) = W*x .+ b;

julia> g = gradient(Params([W, b])) do
         sum(predict([1,2,3]))
       end
Grads(...)

julia> g[W], g[b]
([1.0 2.0 3.0; 1.0 2.0 3.0], [1.0, 1.0])

zygote.jl's People

Contributors

Stargazers

Watchers

Forkers

radovankavicky gapdata jekbradbury baggepinnen c-p-murphy carreau adamnemecek lcy-seso babatundeladega shivamprasad99 roberthoenig mcabbott jessebett doktormike batmanabcdefg roger-luo antoine-levitt giggleliu ssfrr afqueiruga yingboma avik-pal staticfloat chrisrackauckas sipposip under-peter tokazama thebhatman dsweber2 hyzcn jumerckx stjordanis longhua8800w oxinabox yuehhua kraftpunk97-zz mohamed82008 stamp1993 unrealwill chengchingwen sinsixx asylumcorp tkf rxwei simeonschaub sethaxen wegamekinglc mythlee bliang26 dhairyalgandhi rohitmazumder findmyway rdyro kshyatt jonniedie torfjelde satya758 yuxguo azamatb xukai92 arhik wookay drchainsaw andrewning sleort afternone tanhevg astrozot mcognetta cdsousa aterenin pilcru devmotion visr apprenticearnab stev47 carlolucibello haampie marius311 philipvinc hessammehr kareemothman thomascherickal ararslan mschauer nirmal-suthar ven-k lhnguyen-vn ianbutterworth kristofferc aminya cossio appleparan astupidbear gartangh cjdoris tomzhang vishalbelsare hamletwanttocode fagan2888

zygote.jl's Issues

Taking the gradient of `x->sum(cumsum(x))` triggers segmentation fault

MWE:

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_PKG3_PRECOMPILE = 1

julia> using Zygote: gradient
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
┌ Warning: error processing module Zygote signature expressions Expr[:(Tuple{Core.Typeof(typeinf_code3), Method, Any, SimpleVector, Bool, Params})] from :(@eval Core.Compiler function typeinf_code3(method::Method, @nospecialize(atypes), sparams::SimpleVector, run_optimizer::Bool, params::Params)
│           code = code_for_method(method, atypes, sparams, params.world)
│           code === nothing && return (nothing, Any)
│           ccall(:jl_typeinf_begin, Cvoid, ())
│           result = InferenceResult(code)
│           frame = InferenceState(result, false, params)
│           frame === nothing && return (nothing, Any)
│           if typeinf(frame) && run_optimizer
│               opt = OptimizationState(frame)
│               optimize(opt, result.result)
│               (opt.src).inferred = true
│           end
│           ccall(:jl_typeinf_end, Cvoid, ())
│           frame.inferred || return (nothing, Any)
│           return frame
│       end)
└ @ Revise ~/.julia/packages/Revise/51oQc/src/exprutils.jl:94

julia> gradient(x->sum(cumsum(x)), zeros(5))

signal (11): Segmentation fault
in expression starting at no file:0
CreateConstInBoundsGEP1_32 at /buildworker/worker/package_linux64/build/usr/include/llvm/IR/IRBuilder.h:1292 [inlined]
emit_sparam at /buildworker/worker/package_linux64/build/src/codegen.cpp:3262
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:3923
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3615
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:3801 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6254
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1159
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1794
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
cumsum at ./accumulate.jl:116 [inlined]
_forward at /home/scheme/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
#3 at ./REPL[3]:1 [inlined]
_forward at /home/scheme/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
_forward at /home/scheme/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:21
forward at /home/scheme/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:27
gradient at /home/scheme/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:32
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:428
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7fc57b60581f)
unknown function (ip: 0x7)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:787
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
run_backend at /home/scheme/.julia/packages/Revise/51oQc/src/Revise.jl:766
#58 at ./task.jl:259
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1536 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 35859323 (Pool: 35853151; Big: 6172); GC: 79
[1]    22985 segmentation fault (core dumped)  $HOME/julia-1.0.0/bin/julia

Broken example in README.md?

Hi, I'm trying out Zygote and I can't evaluate the first example in README.md.

julia> using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

julia> f(x) = 5x + 3
f (generic function with 1 method)

julia> f(10), f'(10)
ERROR: MethodError: no method matching adjoint(::typeof(f))
Closest candidates are:
  adjoint(::Missing) at missing.jl:79
  adjoint(::Number) at number.jl:193
  adjoint(::LinearAlgebra.Adjoint{#s177,#s176} where #s176<:Union{StaticArray{Tuple{N},T,1} where T where N, StaticArray{Tuple{N,M},T,2} where T where M where N} where #s177) at /Users/mason/.julia/packages/StaticArrays/Ze5H3/src/linalg.jl:78
  ...
Stacktrace:
 [1] top-level scope at none:0

Interestingly, every time I try to load Zygote, it precompiles. Not just the first time.

Strange performance with closure

Please see the example code:

using BenchmarkTools, Zygote

# Wraps a binary function `op` and two unary functions `a` and `b`
struct Foo{Top, Ta, Tb}
    op::Top
    a::Ta
    b::Tb
end

# foo(x) = op(a(x), b(x))
(foo::Foo)(x) = foo.op(foo.a(x), foo.b(x))

@benchmark Foo(*, x->4, cos)(5.0) # fast :)
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     5.847 ns (0.00% GC)
  median time:      6.660 ns (0.00% GC)
  mean time:        6.782 ns (0.00% GC)
  maximum time:     22.986 ns (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     1000

@benchmark Zygote.gradient(Foo(*, x->4, cos), 5.0) # fast :)
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     11.898 ns (0.00% GC)
  median time:      11.923 ns (0.00% GC)
  mean time:        12.564 ns (0.00% GC)
  maximum time:     29.765 ns (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     999


produce_a_thing(a) = Foo(*, x->a, cos)

@benchmark produce_a_thing(4)(5.0) # fast :)
BenchmarkTools.Trial: 
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     6.077 ns (0.00% GC)
  median time:      6.895 ns (0.00% GC)
  mean time:        6.983 ns (0.00% GC)
  maximum time:     23.263 ns (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     1000

@benchmark Zygote.gradient(produce_a_thing(4), 5.0) # slow :(
BenchmarkTools.Trial: 
  memory estimate:  992 bytes
  allocs estimate:  28
  --------------
  minimum time:     3.691 μs (0.00% GC)
  median time:      3.856 μs (0.00% GC)
  mean time:        4.090 μs (4.16% GC)
  maximum time:     1.532 ms (99.30% GC)
  --------------
  samples:          10000
  evals/sample:     8

Results obtained on julia-1.0.3 and Zygote#master.

The only difference between the first and second example is the produce_a_thing function being used. The results of @code_llvm differ slightly between Foo(*, x->4, cos)(5.0) and produce_a_thing(4)(5.0), but the @code_lowered is identical. In short, I have no clue what's going on.

Strange `getproperty` behaviour when

MWE:

using Zygote, LinearAlgebra

struct Baz{T}
    X::T
end

# Comment this out to remove the error.
Zygote.@adjoint function LinearAlgebra.Symmetric(A::AbstractMatrix)
    println("Attemping to Symmetrise")
    @show size(A)
    return Symmetric(A), Δ->(symmetric_back(Δ),)
end

foo(X) = Baz(X).X

Zygote.forward(foo, randn(10, 10))

yields the error

Stacktrace:
 [1] getproperty(::Any, ::Symbol) at ./sysimg.jl:18
 [2] macro expansion at /home/wct23/.julia/packages/Zygote/T7POm/src/lib/lib.jl:116 [inlined]
 [3] adjoint at /home/wct23/.julia/packages/Zygote/T7POm/src/lib/grad.jl:33 [inlined]
 [4] _forward at /home/wct23/.julia/packages/Zygote/T7POm/src/lib/grad.jl:35 [inlined]
 [5] foo at ./REPL[9]:1 [inlined]
 [6] _forward(::Zygote.Context, ::typeof(foo), ::Array{Float64,2}) at /home/wct23/.julia/packages/Zygote/T7POm/src/compiler/interface2.jl:0
 [7] _forward(::Function, ::Array{Float64,2}) at /home/wct23/.julia/packages/Zygote/T7POm/src/compiler/interface.jl:31
 [8] forward(::Function, ::Array{Float64,2}) at /home/wct23/.julia/packages/Zygote/T7POm/src/compiler/interface.jl:37
 [9] top-level scope at none:0

This uses Zygote and IRTools master, and Julia v1.0.2.

(This is a @MikeInnes - sanctioned issue)

Basic Usage Fails

Hello, I am having trouble with basic usage of Zygote out of the box.

using Zygote
f(x) = 2*x + 3
f'(2)

produces:

ERROR: MethodError: no method matching adjoint(::typeof(f))
Closest candidates are:
  adjoint(::Missing) at missing.jl:79
  adjoint(::Number) at number.jl:193
  adjoint(::LinearAlgebra.Adjoint{#s177,#s176} where #s176<:Union{StaticArray{Tuple{N},T,1} where T where N, StaticArray{Tuple{N,M},T,2} where T where M where N} where #s177) at /home/tim/.julia/packages/StaticArrays/WmJnA/src/linalg.jl:78
  ...
Stacktrace:
 [1] top-level scope at none:0

Other Zyogte functionality doesn't seem right either. I got a stack trace error when trying to use gradient.

Are there perhaps install instructions I am missing?

Thank you!

Support for rational operations

While trying to debug TPU ResNet, I found that Zygote is incompatible with the rationals used in FluxML/Flux.jl#538

MWE:

julia> derivative(x -> sum(x * 1 // 4), x)
ERROR: MethodError: no method matching (::Zygote.Jnew{Rational{Int64},Nothing})(::Float32)
Closest candidates are:
  Jnew(::Union{Nothing, NamedTuple}) where {T, G} at /home/staticfloat/.julia/packages/Zygote/8pJmb/src/lib/lib.jl:175
Stacktrace:
 [1] (::getfield(Zygote, Symbol("##261#back#159")){Zygote.Jnew{Rational{Int64},Nothing}})(::Float32) at /home/staticfloat/.julia/packages/Zygote/8pJmb/src/lib/grad.jl:37
 [2] Type at ./rational.jl:15 [inlined]
 [3] (::Zygote.Pullback{Tuple{Type{Rational{Int64}},Int64,Int64},Tuple{getfield(Zygote, Symbol("##82#92")),Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1}}})(::Float32) at /home/staticfloat/.julia/packages/Zygote/8pJmb/src/compiler/interface2.jl:0
 [4] Type at ./rational.jl:20 [inlined]
 [5] // at ./rational.jl:43 [inlined]
 [6] #25 at ./REPL[17]:1 [inlined]
 [7] (::getfield(Zygote, Symbol("##66#67")){Zygote.Pullback{Tuple{getfield(Main, Symbol("##25#26")),Array{Float32,4}},Tuple{Zygote.Pullback{Tuple{typeof(//),Int64,Int64},Tuple{Zygote.Pullback{Tuple{Type{Rational},Int64,Int64},Tuple{Zygote.Pullback{Tuple{Type{Rational{Int64}},Int64,Int64},Tuple{getfield(Zygote, Symbol("##82#92")),Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1}}}}}}},Zygote.Pullback{Tuple{typeof(*),Array{Float32,4},Rational{Int64}},Tuple{Zygote.Pullback{Tuple{typeof(broadcast),typeof(*),Array{Float32,4},Rational{Int64}},Tuple{getfield(Zygote, Symbol("##166#back#114")){typeof(identity)},getfield(Zygote, Symbol("##214#back#136")){getfield(Zygote, Symbol("##134#135")){getfield(Zygote, Symbol("##2207#back#885")){getfield(Zygote, Symbol("##883#884"))},Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}}}},getfield(Zygote, Symbol("##2219#back#889")){getfield(Zygote, Symbol("##887#888")){Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{4},Nothing,typeof(*),Tuple{Array{Float32,4},Rational{Int64}}},getfield(Zygote, Symbol("#back#877")){Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{4},NTuple{4,Base.OneTo{Int64}},getfield(Zygote, Symbol("##862#865")){typeof(*)},Tuple{Array{Float32,4},Rational{Int64}}},Tuple{Array{Float32,4},Array{Float32,4}}}}}}}}},getfield(Zygote, Symbol("##1890#back#734")){getfield(Zygote, Symbol("##730#732")){Array{Float32,4}}}}}})(::Int8) at /home/staticfloat/.julia/packages/Zygote/8pJmb/src/compiler/interface.jl:38
 [8] gradient(::Function, ::Array{Float32,4}) at /home/staticfloat/.julia/packages/Zygote/8pJmb/src/compiler/interface.jl:44
 [9] derivative(::getfield(Main, Symbol("##25#26")), ::Array{Float32,4}) at /home/staticfloat/.julia/packages/Zygote/8pJmb/src/compiler/interface.jl:47
 [10] top-level scope at none:0

Relation to capstan and cassette.jl

How does this relate to capstan and cassette?

Is this an alternative, replacement or stopgap?

Some simple optimisations

Following on from #2; I should really be paper-writing, but we're dangerously close to matching the hand-written gradient for logsumexp, so here we are. Poking at it I found a few general optimisations that will help close the gap.

The most egregious issue is that ForwardDiff's method for exp actually calculates exp(x) twice, presumably due to a failure of CSE. But that's easy to fix and will shave off a bunch of time.

The hand-written version takes advantage of the fact that the gradient of A (1 - sema/sema) cancels out. I don't think we can elide this completely -- we're honour-bound to evaluate the chain rule everywhere -- but through optimisations to maximum and broadcast we can at least avoid any extra allocations for it; then the scalar ops should be inconsequential.

On broadcast, it occurred to me that we can and may want to do optimisations on the broadcasted struct before evaluating. The most obvious thing is to pick up on cases like this, where we don't need to actually use dual numbers but can just use a function of the output. Not sure if that would win over a properly-optimised dual number implementation, but it's worth exploring.

It'd also be good to avoid explicitly instantiating the gradient of sum, instead using broadcasting to expand it. We can get the same effect with a FillArray.

Crash on 0.7.0-rc3.0

Attached jl produces attached log when run with
julia .\zg-logsumexp.jl

Apologies, haven't tested with 1.0. Zygote pkg is added from master in this repo.

zg.log

using Test
using Printf
using Zygote
using BenchmarkTools

# logsumexp()
function logsumexp(x::Array{Float64,1})
  A = maximum(x);
  ema = exp.(x .- A);
  sema = sum(ema);
  log(sema) + A;
end

r = rand(5);
@test logsumexp(r) ≈ log(sum(exp.(r))) atol=1.0e-8
@printf("logsumexp: %f = %f\n", logsumexp(r), log(sum(exp.(r))));

# logsumexp()
function logsumexp_both(x::Array{Float64,1})
  A = maximum(x);
  ema = exp.(x .- A);
  sema = sum(ema);
  l = log(sema) + A;
  Jacobian = ema ./ sema;
  return (l, Jacobian)
end

x = rand(100)

"Timing logsumexp_both" |> println
@btime logsumexp_both(x)

"Timing zygote" |> println
@btime Zygote.gradient(logsumexp, x)

julia> derivative(x -> fs[readline()](x), 1) running, won't stop

https://gist.github.com/hpoit/dc0b2db9f4dffb77403497f673b7e26d

Make Zygote handle invoke

Flux.jl now uses invoke in its dense layer, which Zygote doesn't handle, breaking VGG. We should fix that by supporting invoke in Zygote.

gradient(f,args...) not update when rewriting function

julia> using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

julia> f(x) = 2x + 3
f (generic function with 1 method)

julia> Zygote.gradient(f,0)
(2,)

julia> f(x) = 0
f (generic function with 1 method)

julia> Zygote.gradient(f,0) # should be 0
(2,)

Backward performance of `tr(x1 * x2)`

Hi Mike,

I tried to implement a very straight forward AD (YAAD.jl, only took me a few hours to write the core) for fun (writing a blog post about how to implement a AD in a day with Julia), but it somehow have a much better performance (than Zygote) on this expression:

import Zygote: @grad
import Zygote
using BenchmarkTools

Zygote.@grad LinearAlgebra.tr(x) = LinearAlgebra.tr(x), Δ-> (Δ * Matrix(I, size(x)), )

function bench_tr_mul_zygote(x1, x2)
    g = Zygote.gradient(()->tr(x1 * x2), Zygote.Params([x1, x2]))
    g[x1], g[x2]
end

xv, yv = rand(30, 30), rand(30, 30)
@benchmark bench_tr_mul_zygote(xv, yv)

BTW, it is just similar to AutoGrad.jl

julia> @benchmark bench_tr_mul_autograd(autograd_x, autograd_y)
BenchmarkTools.Trial:
  memory estimate:  33.20 KiB
  allocs estimate:  82
  --------------
  minimum time:     50.218 μs (0.00% GC)
  median time:      62.364 μs (0.00% GC)
  mean time:        90.422 μs (9.86% GC)
  maximum time:     55.386 ms (99.86% GC)
  --------------
  samples:          10000
  evals/sample:     1

julia> @benchmark bench_tr_mul_yaad(yaad_x, yaad_y)
BenchmarkTools.Trial:
  memory estimate:  51.50 KiB
  allocs estimate:  16
  --------------
  minimum time:     10.387 μs (0.00% GC)
  median time:      13.429 μs (0.00% GC)
  mean time:        24.273 μs (45.13% GC)
  maximum time:     55.963 ms (99.96% GC)
  --------------
  samples:          10000
  evals/sample:     1

julia> @benchmark bench_tr_mul_zygote(xv, yv)
BenchmarkTools.Trial:
  memory estimate:  32.45 KiB
  allocs estimate:  60
  --------------
  minimum time:     56.645 μs (0.00% GC)
  median time:      61.555 μs (0.00% GC)
  mean time:        73.006 μs (11.52% GC)
  maximum time:     51.025 ms (99.72% GC)
  --------------
  samples:          10000
  evals/sample:     1

The benchmark codes are in https://github.com/Roger-luo/YAAD.jl/blob/master/benchmark/tr_mul.jl

I thought I won't be faster than Zygote, cuz what I did was just dispatching functions based on types (it is not source to source). Is there anything I did wrong with benchmarking Zygote? Or there is an issue for the compiler? Or there is some problem with my machine (someone can help reproduce it? maybe?)

I did compare the gradient, and I implemented a gradcheck which will compare the Jacobian of each function, and my implementation has the same result with AutoGrad.jl and Zygote.jl, thus I guess there is some issue here, but I tried to read Zygote and had no clue. I'm not sure this is related to control flow? but I didn't use control flow..

version info (I build it from source):

julia> versioninfo()
Julia Version 1.0.1
Commit 0d713926f8 (2018-09-29 19:05 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.0.0)
  CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libimf
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

Bests,
Roger

Gradients for QR Decomposition

Would be nice to have gradients for e.g. QRPackedQ, similar to what we have for Cholesky.

Unsupported Control Flow with custom adjoint

I'm looking at implementing a couple of custom adjoints. With this MWE:

using Zygote, Random
using Zygote: @adjoint
import LinearAlgebra: \

@adjoint function \(A::AbstractMatrix, B::AbstractVector)
    Y = A \ B
    return Y, function(Ȳ)
        B̄ = A' \ Ȳ
        return (-B̄ * Y', B̄)
    end
end

rng, P, Q = MersenneTwister(123456), 10, 9
X, Y = randn(rng, P, P), randn(rng, P, Q)

f = (X, Y)->sum(X \ Y)
Zygote.gradient(f, X, Y)

I get this error:

ERROR: Compiling Tuple{typeof(\),Array{Float64,2},Array{Float64,2}}: Unsupported control flow
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] merge_returns(::Core.Compiler.IRCode) at /home/wct23/.julia/dev/Zygote/src/compiler/reverse.jl:29
 [3] #Primal#39(::Nothing, ::Type, ::Core.Compiler.IRCode) at /home/wct23/.julia/dev/Zygote/src/compiler/reverse.jl:193
 [4] Type at ./none:0 [inlined]
 [5] #Adjoint#65 at /home/wct23/.julia/dev/Zygote/src/compiler/reverse.jl:392 [inlined]
 [6] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at ./none:0
 [7] _lookup_grad(::Type) at /home/wct23/.julia/dev/Zygote/src/compiler/emit.jl:121
 [8] #s54#851 at /home/wct23/.julia/dev/Zygote/src/compiler/interface2.jl:17 [inlined]
 [9] #s54#851(::Any, ::Any, ::Any) at ./none:0
 [10] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:506
 [11] #7 at ./REPL[7]:1 [inlined]
 [12] (::Zygote.J{Tuple{getfield(Main, Symbol("##7#8")),Array{Float64,2},Array{Float64,2}},Tuple{getfield(Main, Symbol("##7#8")),Array{Float64,2},Array{Float64,2},getfield(Zygote, Symbol("##1884#back#734")){getfield(Zygote, Symbol("##730#732")){Array{Float64,2}}},Zygote.J{Tuple{typeof(\),Array{Float64,2},Array{Float64,2}},Tuple{typeof(\)}}}})(::Int8) at /home/wct23/.julia/dev/Zygote/src/compiler/interface2.jl:0
 [13] (::getfield(Zygote, Symbol("##66#67")){Zygote.J{Tuple{getfield(Main, Symbol("##7#8")),Array{Float64,2},Array{Float64,2}},Tuple{getfield(Main, Symbol("##7#8")),Array{Float64,2},Array{Float64,2},getfield(Zygote, Symbol("##1884#back#734")){getfield(Zygote, Symbol("##730#732")){Array{Float64,2}}},Zygote.J{Tuple{typeof(\),Array{Float64,2},Array{Float64,2}},Tuple{typeof(\)}}}}})(::Int8) at /home/wct23/.julia/dev/Zygote/src/compiler/interface.jl:38
 [14] gradient(::Function, ::Array{Float64,2}, ::Vararg{Array{Float64,2},N} where N) at /home/wct23/.julia/dev/Zygote/src/compiler/interface.jl:44
 [15] top-level scope at none:0

Any thoughts? Am I doing something wrong, or is this an issue on Zygote's end?

(I'm running on Zygote and IRTools master, and Julia 1.0.2)

Compiler crashes when calling 'gradient' for a lambda containing a Higher-order function

using Zygote: derivative

derivative(x -> sum(map(x -> x + 3, x)), [1, 2, 3])

Compiler error info:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x43a31f94a -- dyn_cast<llvm::Constant, llvm::Value> at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/usr/include/llvm/Support\Casting.h:334 [inlined]
CreateConstInBoundsGEP1_32 at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/usr/include/llvm/IR\IRBuilder.h:1294 [inlined]
emit_sparam at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3262
in expression starting at no file:0
CreateConstInBoundsGEP1_32 at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/usr/include/llvm/IR\IRBuilder.h:1292 [inlined]
emit_sparam at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3262
emit_expr at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3923
emit_ssaval_assign at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3615
emit_stmtpos at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3801 [inlined]
emit_function at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:6262
jl_compile_linfo at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:1159
emit_invoke at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3094
emit_expr at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3893
emit_ssaval_assign at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3615
emit_stmtpos at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:3801 [inlined]
emit_function at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:6262
jl_compile_linfo at /home/Administrator/buildbot/worker/package_win64/build/src/home/Administrator/buildbot/worker/package_win64/build/src\codegen.cpp:1159
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1796
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1537 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
axes at .\tuple.jl:162 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1831
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
axes at .\generator.jl:52 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
_similar_for at .\array.jl:533 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1831
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
_collect at .\array.jl:637 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1831
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1537 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
collect_similar at .\array.jl:561 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1831
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
map at .\abstractarray.jl:1987 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1831
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1537 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
f at .\REPL[4]:1 [inlined]
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface2.jl:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1831
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
_forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface.jl:21
forward at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface.jl:27
gradient at C:\Users\zyeric\.julia\packages\Zygote\g8WMA\src\compiler\interface.jl:32
grad at .\REPL[2]:1
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
do_call at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:430
eval_stmt_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:363 [inlined]
eval_body at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:678
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:806
unknown function (ip: FFFFFFFFFFFFFFFE)
unknown function (ip: 0000000007E99C7F)
unknown function (ip: 0000000000000000)
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:805
jl_toplevel_eval_in at /home/Administrator/buildbot/worker/package_win64/build/src\builtins.c:622
eval at .\boot.jl:319
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
eval_user_input at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\REPL\src\REPL.jl:85
macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\REPL\src\REPL.jl:117 [inlined]
#28 at .\task.jl:259
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2184
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1537 [inlined]
start_task at /home/Administrator/buildbot/worker/package_win64/build/src\task.c:268
Allocations: 35355225 (Pool: 35349792; Big: 5433); GC: 78

Is it possible to calculate array assignments? How?

Is it possible to make this work?

function foo(x)
     z = zero(x)
     for i in eachindex(x)
           z[i] = x[i]
      end
       z
end

gradient(x->sum(foo(x)), rand(10)) # this gives unsupported control flow error at the moment

I would love to help implement this if this is possible, but anyone can tell if this is possible? and how?

I know it would be quite hard for operator overloaded AD, but maybe this is much easier here?

Splatting

This currently fails silently:

using Zygote
y, back = Zygote.forward(x->(x...,), [1, 2, 3])
back((1, 1, 1))

Presumably the answer should just be ([1, 1, 1],), or even ((1, 1, 1),). It is currently (1,).

Incorrect gradient

This is example is a simplified version of something I'm doing with FillArrays and Zygote, and definitely computes the wrong result. Any thoughts?

using Zygote, Flux

# A vector of length 10
struct MyFill{T} <: AbstractVector{T}
    value::T
end
Base.getindex(m::MyFill, n::Int) = m.value
Base.size(m::MyFill) = (10,)

foo(x) = sum(MyFill(x))

# This yields the right answer :)
Flux.gradient(foo, 5.0)

# This yields the wrong answer :(
Zygote.gradient(foo, 5.0)

gradient of (::Vector, )->Real function

Perhaps I am not using this package idiomatically, but when I try the following, I get an error:

julia> import Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
[ Info: Precompiling IRTools [7869d1d1-7146-5819-86e3-90919afe41df]

julia> f(x) = -sum(abs2, x)
f (generic function with 1 method)

julia> x = ones(10)
10-element Array{Float64,1}:
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0

julia> Zygote.gradient(f, x)
ERROR: BoundsError: attempt to access 4-element Array{Int32,1} at index [5]
Stacktrace:
 [1] getindex at ./array.jl:731 [inlined]
 [2] reverse_ir(::Zygote.Primal) at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/reverse.jl:296
 [3] Zygote.Adjoint(::Zygote.Primal) at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/reverse.jl:378
 [4] #Adjoint#65 at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/reverse.jl:382 [inlined]
 [5] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at ./none:0
 [6] _lookup_grad(::Type) at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/emit.jl:121
 [7] #s18#630 at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/interface2.jl:17 [inlined]
 [8] #s18#630(::Any, ::Any, ::Any) at ./none:0
 [9] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:506
 [10] f at ./REPL[2]:1 [inlined]
 [11] (::Zygote.J{Tuple{typeof(f),Array{Float64,1}},Tuple{typeof(f),Array{Float64,1},getfield(Zygote, Symbol("##701#back2#402")){getfield(Zygote, Symbol("##400#401"))},Zygote.J{Tuple{typeof(sum),typeof(abs2),Array{Float64,1}},Tuple{typeof(sum)}}}})(::Int64) at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/interface2.jl:0
 [12] (::getfield(Zygote, Symbol("##66#67")){Zygote.J{Tuple{typeof(f),Array{Float64,1}},Tuple{typeof(f),Array{Float64,1},getfield(Zygote, Symbol("##701#back2#402")){getfield(Zygote, Symbol("##400#401"))},Zygote.J{Tuple{typeof(sum),typeof(abs2),Array{Float64,1}},Tuple{typeof(sum)}}}}})(::Int64) at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/interface.jl:28
 [13] gradient(::Function, ::Array{Float64,1}) at /home/tamas/.julia/packages/Zygote/tOamD/src/compiler/interface.jl:34
 [14] top-level scope at none:0

Julia v"1.1.0-DEV.322", Zygote#master.

Miscompilation of simple BatchNorm example

I'm trying to calculate sensitivities on the input of something that approximates a simple Batchnorm operation. MWE:

using Zygote, Statistics;
x = randn(7, 7, 1, 1);
loss, back = Zygote._forward(
    Zygote.Context{Nothing}(nothing),
    x -> begin
        mu = mean(x, dims=(1,2,4));
        return sum((x .- mu)./stddev(x .- mu));
    end,
    x,
)

While compiling the backward pass, I get the following error:

ERROR: UndefVarError: phi29 not defined
Stacktrace:
 [1] _mapreducedim! at ./reducedim.jl:271 [inlined]
 [2] _forward(::Zygote.Context{Nothing}, ::typeof(Base._mapreducedim!), ::typeof(identity), ::typeof(Base.add_sum), ::Array{Float64,4}, ::Array{Float64,4}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [3] mapreducedim! at ./reducedim.jl:274 [inlined]
 [4] #sum!#563 at ./reducedim.jl:670 [inlined]
 [5] #sum! at ./none:0 [inlined]
 [6] _forward(::Zygote.Context{Nothing}, ::getfield(Base, Symbol("#kw##sum!")), ::NamedTuple{(:init,),Tuple{Bool}}, ::typeof(sum!), ::typeof(identity), ::Array{Float64,4}, ::Array{Float64,4}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [7] #sum!#564 at ./reducedim.jl:672 [inlined]
 [8] #sum! at ./none:0 [inlined]
 [9] _forward(::Zygote.Context{Nothing}, ::getfield(Base, Symbol("#kw##sum!")), ::NamedTuple{(:init,),Tuple{Bool}}, ::typeof(sum!), ::Array{Float64,4}, ::Array{Float64,4}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [10] mean! at /home/staticfloat/tmp/julia-build/julia-master/usr/share/julia/stdlib/v1.2/Statistics/src/Statistics.jl:101 [inlined]
 [11] _forward(::Zygote.Context{Nothing}, ::typeof(mean!), ::Array{Float64,4}, ::Array{Float64,4}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [12] _mean at /home/staticfloat/tmp/julia-build/julia-master/usr/share/julia/stdlib/v1.2/Statistics/src/Statistics.jl:134 [inlined]
 [13] _forward(::Zygote.Context{Nothing}, ::typeof(Statistics._mean), ::Array{Float64,4}, ::Tuple{Int64,Int64,Int64}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [14] #mean#1 at /home/staticfloat/tmp/julia-build/julia-master/usr/share/julia/stdlib/v1.2/Statistics/src/Statistics.jl:132 [inlined]
 [15] _forward(::Zygote.Context{Nothing}, ::getfield(Statistics, Symbol("##mean#1")), ::Tuple{Int64,Int64,Int64}, ::typeof(mean), ::Array{Float64,4}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [16] _forward(::Zygote.Context{Nothing}, ::getfield(Statistics, Symbol("#kw##mean")), ::NamedTuple{(:dims,),Tuple{Tuple{Int64,Int64,Int64}}}, ::typeof(mean), ::Array{Float64,4}) at ./none:0
 [17] #3 at ./REPL[1]:1 [inlined]
 [18] _forward(::Zygote.Context{Nothing}, ::getfield(Main, Symbol("##3#4")), ::Array{Float64,4}) at /home/staticfloat/.julia/packages/Zygote/uy5LG/src/compiler/interface2.jl:0
 [19] top-level scope at none:0

Interface documentation

I am trying to understand the interface you are planning. Is there any documentation other than the examples in README? The basic questions for the interfaces you introduce (gradient, derivative, forward):

How do I mark variables I want gradients wrt?
How do I get those gradients out?
How do I run code with/without gradient calculation?

nested derivative does not work

julia> derivative(x->x*derivative(y->x+y,1),1)
ERROR: MethodError: no method matching exprtype(::Core.Compiler.IRCode, ::String)
Closest candidates are:
  exprtype(::Core.Compiler.IRCode, ::Expr) at C:\Users\han\.julia\packages\Zygote\5mNII\src\tools\ir.jl:54
  exprtype(::Core.Compiler.IRCode, ::QuoteNode) at C:\Users\han\.julia\packages\Zygote\5mNII\src\tools\ir.jl:51
  exprtype(::Core.Compiler.IRCode, ::GlobalRef) at C:\Users\han\.julia\packages\Zygote\5mNII\src\tools\ir.jl:50
  ...
Stacktrace:
 [1] _broadcast_getindex_evalf at .\broadcast.jl:574 [inlined]
 [2] _broadcast_getindex at .\broadcast.jl:547 [inlined]
 [3] getindex at .\broadcast.jl:507 [inlined]
 [4] copyto_nonleaf!(::Array{DataType,1}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Base.Broadcast.Extruded{Array{Any,1},Tuple{Bool},Tuple{Int64}}}}, ::Base.OneTo{Int64}, ::Int64, ::Int64) at .\broadcast.jl:923
 [5] copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Array{Any,1}}}) at .\broadcast.jl:786
 [6] materialize(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Array{Any,1}}}) at .\broadcast.jl:748
 [7] record!(::Core.Compiler.IRCode) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\reverse.jl:144
 [8] #Primal#39(::Int64, ::Type, ::Core.Compiler.IRCode) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\reverse.jl:189
 [9] Type at .\none:0 [inlined]
 [10] #Adjoint#65 at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\reverse.jl:387 [inlined]
 [11] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Int64}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at .\none:0
 [12] _lookup_grad(::Type) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\emit.jl:121
 [13] #s18#633 at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface2.jl:17 [inlined]
 [14] #s18#633(::Any, ::Any, ::Any) at .\none:0
 [15] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at .\boot.jl:506
 [16] derivative at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:37 [inlined]
 [17] (::Zygote.J{Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64,getfield(Zygote, Symbol("##148#back2#115")){getfield(Zygote, Symbol("##111#113")){1,Int64}},Zygote.J{Tuple{typeof(gradient),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(gradient)}}}})(::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface2.jl:0
 [18] #31 at .\REPL[11]:1 [inlined]
 [19] (::Zygote.J{Tuple{getfield(Main, Symbol("##31#33")),Int64},Tuple{getfield(Main, Symbol("##31#33")),Int64,getfield(Zygote, Symbol("##796#back2#450")){getfield(Zygote, Symbol("##448#449")){Int64,Int64}},Zygote.J{Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64,getfield(Zygote, Symbol("##148#back2#115")){getfield(Zygote, Symbol("##111#113")){1,Int64}},Zygote.J{Tuple{typeof(gradient),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(gradient)}}}},getfield(Zygote, Symbol("##194#back2#147")){Zygote.Jnew{getfield(Main, Symbol("##32#34")){Int64},Nothing}}}})(::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface2.jl:0
 [20] (::getfield(Zygote, Symbol("##66#67")){Zygote.J{Tuple{getfield(Main, Symbol("##31#33")),Int64},Tuple{getfield(Main, Symbol("##31#33")),Int64,getfield(Zygote, Symbol("##796#back2#450")){getfield(Zygote, Symbol("##448#449")){Int64,Int64}},Zygote.J{Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(derivative),getfield(Main, Symbol("##32#34")){Int64},Int64,getfield(Zygote, Symbol("##148#back2#115")){getfield(Zygote, Symbol("##111#113")){1,Int64}},Zygote.J{Tuple{typeof(gradient),getfield(Main, Symbol("##32#34")){Int64},Int64},Tuple{typeof(gradient)}}}},getfield(Zygote, Symbol("##194#back2#147")){Zygote.Jnew{getfield(Main, Symbol("##32#34")){Int64},Nothing}}}}})(::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:28
 [21] gradient(::Function, ::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:34
 [22] derivative(::Function, ::Int64) at C:\Users\han\.julia\packages\Zygote\5mNII\src\compiler\interface.jl:37
 [23] top-level scope at none:0

derivative of function involving sincos errors

julia> f(x) = sum(sincos(x))
f (generic function with 1 method)

julia> f'(pi)
ERROR: BoundsError: attempt to access 15-element Array{Core.Compiler.BasicBlock,1} at index [16]
Stacktrace:
 [1] getindex at ./array.jl:731 [inlined]
 [2] iterate(::Core.Compiler.IncrementalCompact, ::Tuple{Int64,Int64}) at ./compiler/ssair/ir.jl:1066
 [3] iterate at /Users/kristoffer/.julia/dev/IRTools/src/ir/wrap.jl:20 [inlined]
 [4] record_branches!(::Core.Compiler.IRCode) at /Users/kristoffer/.julia/dev/Zygote/src/compiler/reverse.jl:72
 [5] #Primal#39(::Nothing, ::Type, ::Core.Compiler.IRCode) at /Users/kristoffer/.julia/dev/Zygote/src/compiler/reverse.jl:189
 [6] Type at ./none:0 [inlined]
 [7] #Adjoint#65 at /Users/kristoffer/.julia/dev/Zygote/src/compiler/reverse.jl:387 [inlined]
 [8] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at ./none:0
 [9] _lookup_grad(::Type) at /Users/kristoffer/.julia/dev/Zygote/src/compiler/emit.jl:121
 [10] #s53#633 at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface2.jl:17 [inlined]
 [11] #s53#633(::Any, ::Any, ::Any) at ./none:0
 [12] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:505
 [13] _sincos at ./special/trig.jl:201 [inlined]
 [14] (::Zygote.J{Tuple{typeof(Base.Math._sincos),Float64},Tuple{typeof(Base.Math._sincos),Float64,Zygote.J{Tuple{typeof(sincos),Float64},Tuple{typeof(sincos)}}}})(::Tuple{Int64,Int64}) at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface2.jl:0
 [15] sincos at ./special/trig.jl:204 [inlined]
 [16] (::Zygote.J{Tuple{typeof(sincos),Irrational{:π}},Tuple{typeof(sincos),Irrational{:π},Zygote.J{Tuple{typeof(Base.Math._sincos),Float64},Tuple{typeof(Base.Math._sincos),Float64,Zygote.J{Tuple{typeof(sincos),Float64},Tuple{typeof(sincos)}}}},Zygote.J{Tuple{typeof(float),Irrational{:π}},Tuple{typeof(float),Irrational{:π},Zygote.J{Tuple{Type{AbstractFloat},Irrational{:π}},Tuple{DataType,Irrational{:π},Zygote.J{Tuple{Type{Float64},Irrational{:π}},Tuple{DataType,Irrational{:π}}}}}}}}})(::Tuple{Int64,Int64}) at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface2.jl:0
 [17] f at ./REPL[4]:1 [inlined]
 [18] #66 at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface.jl:28 [inlined]
 [19] gradient at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface.jl:34 [inlined]
 [20] derivative at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface.jl:37 [inlined]
 [21] (::getfield(Zygote, Symbol("##68#69")){typeof(f)})(::Irrational{:π}) at /Users/kristoffer/.julia/dev/Zygote/src/compiler/interface.jl:39
 [22] top-level scope at none:0

Gradient of function containing copyto! fails

Sometimes I would like to "intercept" a calculation and save some intermediate results in addition to calculate the gradient of the final value. Unfortunately, this does not work:

julia> using Zygote

julia> W, b = rand(2,3), rand(2);

julia> function f!(y,x)
           y .= W*x + b
           sum(y)
       end
f! (generic function with 1 method)

julia> y = zeros(2); x = [1,2,3];

julia> gradient(()->f!(y,x), Params([W,b]))
Internal error: encountered unexpected error in runtime:
BoundsError(a=Array{Any, (2,)}[SSAValue(5), Core.Compiler.Argument(n=2)], i=(18,))
rec_backtrace at /buildworker/worker/package_linux64/build/src/stackwalk.c:94
record_backtrace at /buildworker/worker/package_linux64/build/src/task.c:246
jl_throw at /buildworker/worker/package_linux64/build/src/task.c:577
jl_bounds_error_ints at /buildworker/worker/package_linux64/build/src/rtutils.c:187
getindex at ./array.jl:731 [inlined]
ssa_substitute_op! at ./compiler/ssair/inlining.jl:1087
ssa_substitute_op! at ./compiler/ssair/inlining.jl:1126
ssa_substitute! at ./compiler/ssair/inlining.jl:1081 [inlined]
ir_inline_item! at ./compiler/ssair/inlining.jl:332
batch_inline! at ./compiler/ssair/inlining.jl:515
ssa_inlining_pass! at ./compiler/ssair/inlining.jl:63 [inlined]
run_passes at ./compiler/ssair/driver.jl:118
optimize at ./compiler/optimize.jl:166
typeinf at ./compiler/typeinfer.jl:35
typeinf_edge at ./compiler/typeinfer.jl:492
abstract_call_method at ./compiler/abstractinterpretation.jl:342
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:81
abstract_eval_call at ./compiler/abstractinterpretation.jl:817
abstract_eval at ./compiler/abstractinterpretation.jl:904
typeinf_local at ./compiler/abstractinterpretation.jl:1128
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1184
typeinf at ./compiler/typeinfer.jl:15
typeinf_edge at ./compiler/typeinfer.jl:492
abstract_call_method at ./compiler/abstractinterpretation.jl:342
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:81
abstract_eval_call at ./compiler/abstractinterpretation.jl:817
abstract_eval at ./compiler/abstractinterpretation.jl:904
typeinf_local at ./compiler/abstractinterpretation.jl:1128
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1184
typeinf at ./compiler/typeinfer.jl:15
typeinf_edge at ./compiler/typeinfer.jl:492
abstract_call_method at ./compiler/abstractinterpretation.jl:342
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:81
abstract_eval_call at ./compiler/abstractinterpretation.jl:817
abstract_eval at ./compiler/abstractinterpretation.jl:904
typeinf_local at ./compiler/abstractinterpretation.jl:1128
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1184
typeinf at ./compiler/typeinfer.jl:15
typeinf_edge at ./compiler/typeinfer.jl:492
abstract_call_method at ./compiler/abstractinterpretation.jl:342
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:81
abstract_eval_call at ./compiler/abstractinterpretation.jl:817
abstract_eval at ./compiler/abstractinterpretation.jl:904
typeinf_local at ./compiler/abstractinterpretation.jl:1128
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1184
typeinf at ./compiler/typeinfer.jl:15
typeinf_ext at ./compiler/typeinfer.jl:567
typeinf_ext at ./compiler/typeinfer.jl:604
jfptr_typeinf_ext_1.clone_1 at /home/troels/packages/julias/julia-1.0.2/lib/julia/sys.so (unknown line)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
jl_apply_with_saved_exception_state at /buildworker/worker/package_linux64/build/src/rtutils.c:257
jl_type_infer at /buildworker/worker/package_linux64/build/src/gf.c:275
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1786 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1830
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
gradient at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface.jl:44
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1831
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:430
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:682
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:806
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f9b65adadaf)
unknown function (ip: 0x8)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:805
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
run_backend at /home/troels/.julia/packages/Revise/7ClGZ/src/Revise.jl:766
#58 at ./task.jl:259
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1831
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
ERROR: Compiling Tuple{typeof(copyto!),Array{Float64,1},Int64,Array{Float64,1},Int64,Int64}: Unsupported control flow
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] merge_returns(::Core.Compiler.IRCode) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/reverse.jl:29
 [3] #Primal#39(::Nothing, ::Type, ::Core.Compiler.IRCode) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/reverse.jl:193
 [4] Type at ./none:0 [inlined]
 [5] #Adjoint#65 at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/reverse.jl:392 [inlined]
 [6] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at ./none:0
 [7] _lookup_grad(::Type) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/emit.jl:121
 [8] #s54#851 at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:17 [inlined]
 [9] #s54#851(::Any, ::Any, ::Any) at ./none:0
 [10] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:506
 [11] copyto! at ./array.jl:277 [inlined]
 [12] (::Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Array{Float64,1}},Tuple{typeof(copyto!),Array{Float64,1},Array{Float64,1},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Int64,Array{Float64,1},Int64,Int64},Tuple{typeof(copyto!)}},getfield(Zygote, Symbol("##669#676"))}})(::Nothing) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:0
 [13] copyto! at ./broadcast.jl:833 [inlined]
 [14] (::Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},getfield(Zygote, Symbol("##82#92")),Zygote.J{Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}}}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Int8,1},Array{Int8,1}}})(::Nothing) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:0
 [15] copyto! at ./broadcast.jl:792 [inlined]
 [16] (::Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},getfield(Zygote, Symbol("##82#92")),Zygote.J{Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}}}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Int8,1},Array{Int8,1}}},Zygote.J{Tuple{typeof(convert),Type{Base.Broadcast.Broadcasted{Nothing,Axes,F,Args} where Args<:Tuple where F where Axes},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(convert),UnionAll,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}}}})(::Nothing) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:0
 [17] materialize! at ./broadcast.jl:751 [inlined]
 [18] (::Zygote.J{Tuple{typeof(Base.Broadcast.materialize!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(Base.Broadcast.materialize!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},getfield(Zygote, Symbol("##82#92")),Zygote.J{Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}}}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Int8,1},Array{Int8,1}}},Zygote.J{Tuple{typeof(convert),Type{Base.Broadcast.Broadcasted{Nothing,Axes,F,Args} where Args<:Tuple where F where Axes},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(convert),UnionAll,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}}}},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(isa),Tuple{Base.OneTo{Int64}},Type{Nothing}},Tuple{typeof(isa)}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Axes,F,Args} where Args<:Tuple where F where Axes},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{UnionAll,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},Zygote.J{Tuple{typeof(Core.Typeof),typeof(identity)},Tuple{typeof(Core.Typeof),typeof(identity),Zygote.J{Tuple{typeof(isa),typeof(identity),Type{Type}},Tuple{typeof(isa)}},Array{Int8,1}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:Typeof,Zygote.Context,Module,typeof(Core.Typeof)}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Module}},getfield(Zygote, Symbol("##79#89"))}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}})(::Nothing) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:0
 [19] f! at ./REPL[3]:2 [inlined]
 [20] (::Zygote.J{Tuple{typeof(f!),Array{Float64,1},Array{Int64,1}},Tuple{typeof(f!),Array{Float64,1},Array{Int64,1},getfield(Zygote, Symbol("##1884#back#734")){getfield(Zygote, Symbol("##730#732")){Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(Base.Broadcast.materialize!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},getfield(Zygote, Symbol("##82#92")),Zygote.J{Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}}}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Int8,1},Array{Int8,1}}},Zygote.J{Tuple{typeof(convert),Type{Base.Broadcast.Broadcasted{Nothing,Axes,F,Args} where Args<:Tuple where F where Axes},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(convert),UnionAll,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}}}},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(isa),Tuple{Base.OneTo{Int64}},Type{Nothing}},Tuple{typeof(isa)}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Axes,F,Args} where Args<:Tuple where F where Axes},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{UnionAll,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},Zygote.J{Tuple{typeof(Core.Typeof),typeof(identity)},Tuple{typeof(Core.Typeof),typeof(identity),Zygote.J{Tuple{typeof(isa),typeof(identity),Type{Type}},Tuple{typeof(isa)}},Array{Int8,1}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:Typeof,Zygote.Context,Module,typeof(Core.Typeof)}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Module}},getfield(Zygote, Symbol("##79#89"))}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}},getfield(Zygote, Symbol("##2068#back#842")){getfield(Zygote, Symbol("##840#841"))},getfield(Zygote, Symbol("##1624#back#620")){getfield(Zygote, Symbol("##616#618")){Tuple{Array{Float64,1},Array{Float64,1}}}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Array{Float64,1}}},getfield(Zygote, Symbol("##1932#back#759")){getfield(Zygote, Symbol("##757#758")){Array{Float64,2},Array{Int64,1}}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Array{Float64,2}}}}})(::Int8) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:0
 [21] #3 at ./REPL[8]:1 [inlined]
 [22] (::Zygote.J{Tuple{getfield(Main, Symbol("##3#4"))},Tuple{getfield(Main, Symbol("##3#4")),Zygote.J{Tuple{typeof(f!),Array{Float64,1},Array{Int64,1}},Tuple{typeof(f!),Array{Float64,1},Array{Int64,1},getfield(Zygote, Symbol("##1884#back#734")){getfield(Zygote, Symbol("##730#732")){Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(Base.Broadcast.materialize!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(copyto!),Array{Float64,1},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},getfield(Zygote, Symbol("##82#92")),Zygote.J{Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(Base.Broadcast._axes),Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}}}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1},Array{Int8,1},Array{Int8,1}}},Zygote.J{Tuple{typeof(convert),Type{Base.Broadcast.Broadcasted{Nothing,Axes,F,Args} where Args<:Tuple where F where Axes},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(convert),UnionAll,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}}}},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Zygote.J{Tuple{typeof(isa),Tuple{Base.OneTo{Int64}},Type{Nothing}},Tuple{typeof(isa)}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:axes,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Tuple{Base.OneTo{Int64}}}},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Any,1},Array{Int8,1}}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Axes,F,Args} where Args<:Tuple where F where Axes},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{UnionAll,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},Zygote.J{Tuple{Type{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}}},typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}}},Tuple{DataType,typeof(identity),Tuple{Array{Float64,1}},Tuple{Base.OneTo{Int64}},getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(identity),Tuple{Array{Float64,1}}},Nothing}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Base.OneTo{Int64}}},Tuple{Base.OneTo{Int64}}},Tuple{typeof(convert),DataType,Tuple{Base.OneTo{Int64}}}},Zygote.J{Tuple{typeof(convert),Type{Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}},Tuple{typeof(convert),DataType,Tuple{Array{Float64,1}}}},Zygote.J{Tuple{typeof(convert),Type{typeof(identity)},typeof(identity)},Tuple{typeof(convert),DataType,typeof(identity)}}}},Zygote.J{Tuple{typeof(Core.Typeof),typeof(identity)},Tuple{typeof(Core.Typeof),typeof(identity),Zygote.J{Tuple{typeof(isa),typeof(identity),Type{Type}},Tuple{typeof(isa)}},Array{Int8,1}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:Typeof,Zygote.Context,Module,typeof(Core.Typeof)}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Module}},getfield(Zygote, Symbol("##79#89"))}},Zygote.J{Tuple{typeof(axes),Array{Float64,1}},Tuple{typeof(axes),Array{Float64,1},Zygote.J{Tuple{typeof(map),Type{Base.OneTo},Tuple{Int64}},Tuple{typeof(map),UnionAll,Tuple{Int64},getfield(Zygote, Symbol("##160#back#114")){typeof(identity)},Zygote.J{Tuple{Type{Base.OneTo},Int64},Tuple{UnionAll,Int64,Zygote.J{Tuple{Type{Base.OneTo{Int64}},Int64},Tuple{DataType,Int64,getfield(Zygote, Symbol("##255#back#159")){Zygote.Jnew{Base.OneTo{Int64},Nothing}},getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))},getfield(Zygote, Symbol("##1480#back#570")){getfield(Zygote, Symbol("##568#569")){Int64,Int64}},Zygote.J{Tuple{typeof(zero),Type{Int64}},Tuple{typeof(zero),DataType,getfield(Zygote, Symbol("##1612#back#614")){getfield(Zygote, Symbol("##612#613"))}}}}}}},getfield(Zygote, Symbol("##172#back#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}},getfield(Zygote, Symbol("##668#675"))}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:args,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},Tuple{Array{Float64,1}}}},getfield(Zygote, Symbol("##233#back#147")){getfield(Zygote, Symbol("#back#146")){:f,Zygote.Context,Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(identity),Tuple{Array{Float64,1}}},typeof(identity)}}}},getfield(Zygote, Symbol("##2068#back#842")){getfield(Zygote, Symbol("##840#841"))},getfield(Zygote, Symbol("##1624#back#620")){getfield(Zygote, Symbol("##616#618")){Tuple{Array{Float64,1},Array{Float64,1}}}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Array{Float64,1}}},getfield(Zygote, Symbol("##1932#back#759")){getfield(Zygote, Symbol("##757#758")){Array{Float64,2},Array{Int64,1}}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Array{Float64,2}}}}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Array{Int64,1}}},getfield(Zygote, Symbol("##148#back#112")){getfield(Zygote, Symbol("##110#111")){Zygote.Context,Array{Float64,1}}}}})(::Int8) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface2.jl:0
 [23] (::getfield(Zygote, Symbol("##70#71")))(::Any) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface.jl:76
 [24] gradient(::Function, ::Params) at /home/troels/.julia/packages/Zygote/HvFbo/src/compiler/interface.jl:44
 [25] top-level scope at none:0

while the non-mutating function

julia> function f(x)
           y = W*x + b
           sum(y)
       end

works just fine.

julia> VERSION
v"1.0.2"

julia> Pkg.status()
    Status `~/.julia/environments/v1.0/Project.toml`
  ...
  [7869d1d1] IRTools v0.1.0 #master (https://github.com/MikeInnes/IRTools.jl.git)
  ...
  [e88e6eb3] Zygote v0.1.0+ #master (https://github.com/FluxML/Zygote.jl.git)

Error, cannot multiply two vectors

Julia v0.7 / Zygote v0.1

x        = randn(3)             # Input
v        = randn(3)             # Vector
H        = randn(3,3); H = H+H' # Hessian
f(x)     = 0.5*x'*(H*x)         
val,back = Zygote.forward(f,x)

julia> back(x)
ERROR: DimensionMismatch("Cannot multiply two vectors")
Stacktrace:
 [1] *(::Array{Float64,1}, ::Array{Float64,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/LinearAlgebra/src/deprecated.jl:566
 [2] (::getfield(Zygote, Symbol("##480#481")){LinearAlgebra.Adjoint{Float64,Array{Float64,1}},Array{Float64,1}})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/lib/array.jl:21
 [3] (::getfield(Zygote, Symbol("##844#back2#482")){getfield(Zygote, Symbol("##480#481")){LinearAlgebra.Adjoint{Float64,Array{Float64,1}},Array{Float64,1}}})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/lib/lib.jl:33
 [4] * at ./operators.jl:502 [inlined]
 [5] (::Zygote.J{Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}},Array{Float64,1}},Any})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [6] f at ./REPL[5]:1 [inlined]
 [7] (::Zygote.J{Tuple{typeof(f),Array{Float64,1}},Any})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [8] (::getfield(Zygote, Symbol("##51#52")){Zygote.J{Tuple{typeof(f),Array{Float64,1}},Any}})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:28
 [9] top-level scope at none:0

How to include functions Zygote doesn't know about?

If for instance I want ForwardDiff to be able to take derivatives of the trigonometric functions from SLEEF.jl, I would just give the relevant definition

julia> using ForwardDiff: derivative, Dual, value, partials

julia> using SLEEF

julia> SLEEF.sin(d::Dual{T,V,N}) where {T,V,N} = Dual{T,V,N}(SLEEF.sin(value(d)), SLEEF.cos(value(d)) * partials(d))

julia> derivative(SLEEF.sin, 3.14)
-0.9999987317275395

Is there an analogous way to do this with Zygote?

Updating zygote errors? `MethodError: no method matching getindex(::Nothing, ::String)`

(v1.0) pkg> up
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Updating git-repo `https://github.com/FluxML/Zygote.jl`
 Resolving package versions...
  Updating `~/.julia/environments/v1.0/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v1.0/Manifest.toml`
 [no changes]
ERROR: MethodError: no method matching getindex(::Nothing, ::String)
Stacktrace:
 [1] #build_versions#45(::Bool, ::Function, ::Pkg.Types.Context, ::Array{Base.UUID,1}) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1016
 [2] build_versions at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1005 [inlined]
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1209
 [4] #up#31(::Pkg.Types.UpgradeLevel, ::Pkg.Types.PackageMode, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:184
 [5] up at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:158 [inlined]
 [6] do_up!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:610
 [7] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:686
 [8] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:685
 [9] do_cmd!(::Pkg.REPLMode.PkgCommand, ::REPL.LineEditREPL) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:542
 [10] #do_cmd#30(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:507
 [11] do_cmd at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:503 [inlined]
 [12] (::getfield(Pkg.REPLMode, Symbol("##41#44")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:842
 [13] #invokelatest#1 at ./essentials.jl:686 [inlined]
 [14] invokelatest at ./essentials.jl:685 [inlined]
 [15] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/REPL/src/LineEdit.jl:2261
 [16] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:1029
 [17] run_repl(::REPL.AbstractREPL, ::Any) at /home/chris/Documents/prog/julia-dev/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:191
 [18] (::getfield(Base, Symbol("##720#722")){Bool,Bool,Bool,Bool})(::Module) at ./logging.jl:311
 [19] #invokelatest#1 at ./essentials.jl:686 [inlined]
 [20] invokelatest at ./essentials.jl:685 [inlined]
 [21] macro expansion at ./logging.jl:308 [inlined]
 [22] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:330
 [23] exec_options(::Base.JLOptions) at ./client.jl:242
 [24] _start() at ./client.jl:421

Crashing on functions involving array operations

This is probably due to a known unimplemented feature, but I'm having problems getting gradients with respect to functions involving arrays. Sometimes f'(p) works but derivative(f, p) does not. I want to use the latter because I need gradients with respect to parameter vectors, not scalars as in these minimal examples.

EXAMPLE 1:

function f1(p)
    x = [1.0, 1.0]
    x[2] = p * x[1]
    return x[2]
end

julia> f1'(1.5)
0.9999999999991082

julia> derivative(f1, 1.5)
ERROR: $(Expr(:boundscheck))
Stacktrace:
 [1] error(::Expr) at ./error.jl:42
 [2] exprtype(::Core.Compiler.IRCode, ::Expr) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/tools/ir.jl:54
 [3] _broadcast_getindex_evalf at ./broadcast.jl:574 [inlined]
 [4] _broadcast_getindex at ./broadcast.jl:547 [inlined]
 [5] getindex at ./broadcast.jl:507 [inlined]
 [6] copyto_nonleaf!(::Array{DataType,1}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Base.Broadcast.Extruded{Array{Any,1},Tuple{Bool},Tuple{Int64}}}}, ::Base.OneTo{Int64}, ::Int64, ::Int64) at ./broadcast.jl:923
 [7] copy(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Array{Any,1}}}) at ./broadcast.jl:786
 [8] materialize(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(Zygote.exprtype),Tuple{Base.RefValue{Core.Compiler.IRCode},Array{Any,1}}}) at ./broadcast.jl:748
 [9] record!(::Core.Compiler.IRCode) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/reverse.jl:142
 [10] #grad_ir#50(::Nothing, ::Function, ::Core.Compiler.IRCode) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/reverse.jl:319
 [11] (::getfield(Zygote, Symbol("#kw##grad_ir")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::typeof(Zygote.grad_ir), ::Core.Compiler.IRCode) at ./none:0
 [12] _lookup_grad(::Type) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/emit.jl:118
 [13] #s19#533 at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:18 [inlined]
 [14] #s19#533(::Any, ::Any, ::Any) at ./none:0
 [15] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:506
 [16] f1 at ./REPL[9]:3 [inlined]
 [17] (::Zygote.J{Tuple{typeof(f1),Float64},Any})(::Int64) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [18] (::getfield(Zygote, Symbol("##51#52")){Zygote.J{Tuple{typeof(f1),Float64},Any}})(::Int64) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:28
 [19] gradient(::Function, ::Float64) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:34
 [20] derivative(::Function, ::Float64) at /Users/nurban/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:37
 [21] top-level scope at none:0

The next example with a 'for' loop actually segfaults.

EXAMPLE 2:

function f2(p)
    x = [1.0, 1.0]
    for i in 1:1
        x[i+1] = p * x[i]
    end
    return x[2]
end

julia> f2'(1.5)
0.9999999999991082

julia> derivative(f2, 1.5)

signal (11): Segmentation fault: 11
in expression starting at no file:0

Clearly f'(p) and derivative(f, p) work differently, although I would have expected the former to be syntactic sugar for the latter. This is true even for simple working examples: f(x) = x^2 gives f'(1.5) == 2.9999999999973244 while derivative(f, 1.5) == 3.0.

Also, these problems seem to have to do with array operations, because it works if I write out the array operations by hand as variables:

function g2(p)
    x1 = 1.0
    x2 = 1.0
    x2 = p * x1
    return x2
end

julia> derivative(g2, 1.5)
1.0

Am I misunderstanding how derivative() is to be used?

ERROR: `throw` not supported

julia> using StaticArrays

julia> import Zygote: gradient

julia> const S = (@SMatrix randn(6,4)) |> x -> x' * x;

julia> g(x) = 0.5 * x' * S * x
g (generic function with 1 method)

julia> x = @SVector randn(4);

julia> g(x)
1.8359824145349797

julia> gradient(g, x)
ERROR: `throw` not supported
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] merge_returns(::Core.Compiler.IRCode) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/reverse.jl:15
 [3] #Primal#46(::Nothing, ::Type, ::Core.Compiler.IRCode) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/reverse.jl:176
 [4] Type at ./none:0 [inlined]
 [5] #Adjoint#72 at /home/chris/.julia/packages/Zygote/zd432/src/compiler/reverse.jl:375 [inlined]
 [6] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::Type{Zygote.Adjoint}, ::Core.Compiler.IRCode) at ./none:0
 [7] _lookup_grad(::Type) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/emit.jl:116
 [8] #s18#615 at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:19 [inlined]
 [9] #s18#615(::Any, ::Any, ::Any) at ./none:0
 [10] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:506
 [11] _broadcast_getindex at ./broadcast.jl:525 [inlined]
 [12] _getindex at ./broadcast.jl:571 [inlined]
 [13] (::Zygote.J{Tuple{typeof(Base.Broadcast._getindex),Tuple{Tuple{Float64,Float64}},Int64},Tuple{typeof(Base.Broadcast._getindex),Tuple{Tuple{Float64,Float64}},Int64,getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)},Zygote.J{Tuple{typeof(Base.Broadcast._broadcast_getindex),Tuple{Float64,Float64},Int64},Tuple{typeof(Base.Broadcast._broadcast_getindex),Tuple{Float64,Float64},Int64,getfield(Zygote, Symbol("##154#back2#120")){getfield(Zygote, Symbol("##116#118")){2,Int64}},Zygote.J{Tuple{typeof(getindex),Int64,Int64},Tuple{typeof(getindex)}}}},getfield(Zygote, Symbol("##154#back2#120")){getfield(Zygote, Symbol("##116#118")){1,Int64}}}})(::Tuple{Float64}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [14] _broadcast_getindex at ./broadcast.jl:546 [inlined]
 [15] (::Zygote.J{Tuple{typeof(Base.Broadcast._broadcast_getindex),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}},Int64},Any})(::Float64) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [16] #17 at ./broadcast.jl:922 [inlined]
 [17] (::Zygote.J{Tuple{getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Int64},Any})(::Float64) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [18] ntuple at ./tuple.jl:157 [inlined]
 [19] (::Zygote.J{Tuple{typeof(ntuple),getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Val{2}},Tuple{typeof(ntuple),getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Val{2},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)},Zygote.J{Tuple{getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Int64},Any},Zygote.J{Tuple{getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Int64},Any}}})(::Tuple{Float64,Float64}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [20] tuplebroadcast at ./broadcast.jl:922 [inlined]
 [21] (::Zygote.J{Tuple{typeof(Base.Broadcast.tuplebroadcast),Tuple{Float64,Float64},Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Any})(::Tuple{Float64,Float64}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [22] copy at ./broadcast.jl:920 [inlined]
 [23] (::Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Any})(::Tuple{Float64,Float64}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [24] materialize at ./broadcast.jl:724 [inlined]
 [25] (::Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}},Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(adjoint),Tuple{Tuple{Float64,Float64}}}}}}})(::Tuple{Float64,Float64}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [26] #5 at /home/chris/Documents/prog/zygote/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/adjtrans.jl:185 [inlined]
 [27] (::Zygote.J{Tuple{getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Float64,Float64},Any})(::Float64) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [28] macro expansion at /home/chris/.julia/packages/StaticArrays/Ze5H3/src/broadcast.jl:133 [inlined]
 [29] _broadcast at /home/chris/.julia/packages/StaticArrays/Ze5H3/src/broadcast.jl:94 [inlined]
 [30] (::Zygote.J{Tuple{typeof(StaticArrays._broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Size{(4,)},Tuple{Size{()},Size{(4,)}},Float64,SArray{Tuple{4},Float64,1,4}},Any})(::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [31] (::getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing,Nothing,Tuple{Nothing,Nothing}},Tuple{Nothing,Nothing}},Zygote.J{Tuple{typeof(StaticArrays._broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Size{(4,)},Tuple{Size{()},Size{(4,)}},Float64,SArray{Tuple{4},Float64,1,4}},Any}})(::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/lib/lib.jl:156
 [32] (::getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing,Nothing,Tuple{Nothing,Nothing}},Tuple{Nothing,Nothing}},Zygote.J{Tuple{typeof(StaticArrays._broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Size{(4,)},Tuple{Size{()},Size{(4,)}},Float64,SArray{Tuple{4},Float64,1,4}},Any}}})(::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/lib/lib.jl:36
 [33] copy at /home/chris/.julia/packages/StaticArrays/Ze5H3/src/broadcast.jl:24 [inlined]
 [34] (::Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Tuple{Base.OneTo{Int64}},getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Any})(::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [35] materialize at ./broadcast.jl:724 [inlined]
 [36] broadcast at ./broadcast.jl:702 [inlined]
 [37] (::Zygote.J{Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Float64,SArray{Tuple{4},Float64,1,4}},Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Tuple{Base.OneTo{Int64}},getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.instantiate)}}}},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},getfield(Zygote, Symbol("##1051#back2#594")){getfield(Zygote, Symbol("##592#593"))}}},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)}}})(::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [38] (::getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},Zygote.J{Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Float64,SArray{Tuple{4},Float64,1,4}},Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Tuple{Base.OneTo{Int64}},getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.instantiate)}}}},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},getfield(Zygote, Symbol("##1051#back2#594")){getfield(Zygote, Symbol("##592#593"))}}},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)}}}})(::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/lib/lib.jl:156
 [39] #166#back2 at /home/chris/.julia/packages/Zygote/zd432/src/lib/lib.jl:36 [inlined]
 [40] broadcast at /home/chris/Documents/prog/zygote/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/adjtrans.jl:185 [inlined]
 [41] (::Zygote.J{Tuple{typeof(broadcast),typeof(*),Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}},Tuple{typeof(broadcast),typeof(*),Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}},getfield(Zygote, Symbol("##1027#back2#575")){getfield(Zygote, Symbol("##573#574"))},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},Zygote.J{Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Float64,SArray{Tuple{4},Float64,1,4}},Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Tuple{Base.OneTo{Int64}},getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.instantiate)}}}},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},getfield(Zygote, Symbol("##1051#back2#594")){getfield(Zygote, Symbol("##592#593"))}}},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)}}}}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}},Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}}}}},getfield(Zygote, Symbol("##1051#back2#594")){getfield(Zygote, Symbol("##592#593"))},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)},getfield(Zygote, Symbol("##188#back2#145")){Zygote.Jnew{getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Nothing}}}})(::LinearAlgebra.Transpose{Float64,SArray{Tuple{4},Float64,1,4}}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [42] * at ./arraymath.jl:52 [inlined]
 [43] * at ./operators.jl:502 [inlined]
 [44] (::Zygote.J{Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},SArray{Tuple{4,4},Float64,2,16},SArray{Tuple{4},Float64,1,4}},Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},SArray{Tuple{4,4},Float64,2,16},Tuple{SArray{Tuple{4},Float64,1,4}},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing,Nothing},Tuple{Nothing}},Zygote.J{Tuple{typeof(Base.afoldl),typeof(*),LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},SArray{Tuple{4},Float64,1,4}},Tuple{typeof(Base.afoldl),typeof(*),LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},SArray{Tuple{4},Float64,1,4},getfield(Zygote, Symbol("##1015#back2#569")){getfield(Zygote, Symbol("##567#568")){LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},SArray{Tuple{4},Float64,1,4}}}}}}},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)},getfield(Zygote, Symbol("##1015#back2#569")){getfield(Zygote, Symbol("##567#568")){LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},SArray{Tuple{4,4},Float64,2,16}}},Zygote.J{Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}},Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}},Zygote.J{Tuple{typeof(broadcast),typeof(*),Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}},Tuple{typeof(broadcast),typeof(*),Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}},getfield(Zygote, Symbol("##1027#back2#575")){getfield(Zygote, Symbol("##573#574"))},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},Zygote.J{Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Float64,SArray{Tuple{4},Float64,1,4}},Tuple{typeof(broadcast),getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Tuple{Base.OneTo{Int64}},getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{StaticArrays.StaticArrayStyle{1},Nothing,getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Tuple{Float64,SArray{Tuple{4},Float64,1,4}}}},Tuple{typeof(Base.Broadcast.instantiate)}}}},getfield(Zygote, Symbol("##166#back2#128")){getfield(Zygote, Symbol("##126#127")){Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}},getfield(Zygote, Symbol("##1051#back2#594")){getfield(Zygote, Symbol("##592#593"))}}},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)}}}}},Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}},Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}},Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}},Any},Zygote.J{Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}},Tuple{typeof(Base.Broadcast.instantiate),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,SArray{Tuple{4},Float64,1,4}}}}}}}}},getfield(Zygote, Symbol("##1051#back2#594")){getfield(Zygote, Symbol("##592#593"))},getfield(Zygote, Symbol("##148#back2#115")){typeof(identity)},getfield(Zygote, Symbol("##188#back2#145")){Zygote.Jnew{getfield(LinearAlgebra, Symbol("##5#6")){typeof(*)},Nothing}}}}}}}})(::Int64) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [45] g at ./REPL[4]:1 [inlined]
 [46] (::Zygote.J{Tuple{typeof(g),SArray{Tuple{4},Float64,1,4}},Any})(::Int64) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface2.jl:0
 [47] (::getfield(Zygote, Symbol("##73#74")){Zygote.J{Tuple{typeof(g),SArray{Tuple{4},Float64,1,4}},Any}})(::Int64) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface.jl:28
 [48] gradient(::Function, ::SArray{Tuple{4},Float64,1,4}) at /home/chris/.julia/packages/Zygote/zd432/src/compiler/interface.jl:34 [49] top-level scope at none:0

julia> versioninfo() # latest commit from here https://github.com/JuliaLang/julia/tree/mji/zygote
Julia Version 1.0.0
Commit ece21e1ae5 (2018-08-29 14:15 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, znver1)

I am on the latest commit of this branch of Julia, and on Zygote master.

Simple derivative is very slow

I tried the most simple example

julia> f(x) = 3x^2 + 2x + 1
f (generic function with 1 method)

julia> f′(x) = derivative(f, x)
f′ (generic function with 1 method)

But the performance of the derivative is way too slow (this should be rather something like 2ns):

julia> @btime f′($5.0)
  8.095 μs (77 allocations: 1.92 KiB)
32.0

Version problems with IRTools

Zygote supports julia v0.7 whereas IRTools only support Julia > v1.0. This causes the package manager to cry when using julia v0.7

`@nograd rand` a bit overzealous

rand is currently flagged as a @nograd. While this is probably appropriate for the methods of rand one finds in Base, it's certainly not for things in Distributions. For example:

Zygote.gradient(x->rand(Normal(x, 1)), x)

Here the gradient w.r.t. x definitely shouldn't be zero in the general case.

I would suggest that @nograd should be extended slightly to allow it to be applied only to specific methods of function.

word choice in README

"Despite its performance, Zygote supports the full flexibility and dynamism of the Julia language."
Despite? maybe:

Zygote supports the full flexibility and dynamism of the Julia language.
Zygote supports the full flexibility, dynamism, and performance of the Julia language.
Zygote supports the full flexibility and dynamism of the Julia language without runtime overhead.
Zygote supports the full flexibility and dynamism of the Julia language. Runtimes are unchanged.
Zygote supports the full flexibility and dynamism of the Julia language. Runtime costs are negligible.

Assertion error in record_branches

Julia v0.7 / Zygote v0.1

using Zygote
x      = randn(3)             # Input
v      = randn(3)             # Vector
H      = randn(3,3); H = H+H' # Hessian
f(x)   = 0.5*x'*(H*x)     # i'H*i function to take hessian of
fp  = Zygote.gradient(f,x)
fp(x)

ERROR: AssertionError: length(preds) <= 2
Stacktrace:
 [1] record_branches!(::Core.Compiler.IRCode) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/reverse.jl:76
 [2] #grad_ir#50(::Nothing, ::Function, ::Core.Compiler.IRCode) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/reverse.jl:319
 [3] (::getfield(Zygote, Symbol("#kw##grad_ir")))(::NamedTuple{(:varargs,),Tuple{Nothing}}, ::typeof(Zygote.grad_ir), ::Core.Compiler.IRCode) at ./none:0
 [4] _lookup_grad(::Type) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/emit.jl:118
 [5] #s21#533 at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:18 [inlined]
 [6] #s21#533(::Any, ::Any, ::Any) at ./none:0
 [7] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:506
 [8] _broadcast_getindex at ./broadcast.jl:525 [inlined]
 [9] (::Zygote.J{Tuple{typeof(Base.Broadcast._broadcast_getindex),Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}},Int64},Any})(::NamedTuple{(:parent,),Tuple{Array{Float64,1}}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [10] _getindex at ./broadcast.jl:571 [inlined]
 [11] (::Zygote.J{Tuple{typeof(Base.Broadcast._getindex),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}},Int64},Any})(::Tuple{NamedTuple{(:parent,),Tuple{Array{Float64,1}}}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [12] _broadcast_getindex at ./broadcast.jl:546 [inlined]
 [13] (::Zygote.J{Tuple{typeof(Base.Broadcast._broadcast_getindex),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}}},Int64},Any})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [14] #17 at ./broadcast.jl:922 [inlined]
 [15] (::Zygote.J{Tuple{getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}}}},Int64},Any})(::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [16] ntuple at ./tuple.jl:157 [inlined]
 [17] (::Zygote.J{Tuple{typeof(ntuple),getfield(Base.Broadcast, Symbol("##17#18")){Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}}}},Val{2}},Any})(::Tuple{Float64,Array{Float64,1}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [18] tuplebroadcast at ./broadcast.jl:922 [inlined]
 [19] (::Zygote.J{Tuple{typeof(Base.Broadcast.tuplebroadcast),Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}},Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}}}},Any})(::Tuple{Float64,Array{Float64,1}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [20] copy at ./broadcast.jl:920 [inlined]
 [21] (::Zygote.J{Tuple{typeof(copy),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}}}},Any})(::Tuple{Float64,Array{Float64,1}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [22] materialize at ./broadcast.jl:724 [inlined]
 [23] (::Zygote.J{Tuple{typeof(Base.Broadcast.materialize),Base.Broadcast.Broadcasted{Base.Broadcast.Style{Tuple},Nothing,typeof(LinearAlgebra.quasiparenta),Tuple{Tuple{Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}}}}},Any})(::Tuple{Float64,Array{Float64,1}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [24] broadcast at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/LinearAlgebra/src/adjtrans.jl:185 [inlined]
 [25] (::Zygote.J{Tuple{typeof(broadcast),typeof(*),Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}},Any})(::LinearAlgebra.Transpose{Float64,Array{Float64,1}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [26] * at ./arraymath.jl:52 [inlined]
 [27] (::Zygote.J{Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}}},Any})(::LinearAlgebra.Transpose{Float64,Array{Float64,1}}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [28] * at ./operators.jl:502 [inlined]
 [29] (::Zygote.J{Tuple{typeof(*),Float64,LinearAlgebra.Adjoint{Float64,Array{Float64,1}},Array{Float64,1}},Any})(::Int64) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [30] f at ./REPL[5]:1 [inlined]
 [31] (::Zygote.J{Tuple{typeof(f),Array{Float64,1}},Any})(::Int64) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
 [32] (::getfield(Zygote, Symbol("##51#52")){Zygote.J{Tuple{typeof(f),Array{Float64,1}},Any}})(::Int64) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:28
 [33] gradient(::Function, ::Array{Float64,1}) at /home/fredrikb/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:34
 [34] top-level scope at none:0

Segmentation fault when calling copy

using Zygote, LinearAlgebra
x        = randn(3)             # Input
v        = randn(3)             # Vector
H        = randn(3,3); H = H+H' # Hessian
f(x)     = 0.5*(copy(x')*(H*x))
f'(x) # Segmentation fault

using Zygote, LinearAlgebra
x        = randn(3)             # Input
v        = randn(3)             # Vector
H        = randn(3,3); H = H+H' # Hessian
f(x)     = 0.5*(x'*(H*x))
f'(x) #  No segmentation fault

Seems to be #265ing

Maybe there is a catch that is not getting updated when a function is redefined?

Code:

using Zygote;
g(x) = 2x;
@show g'(10);
g(x) = 3x;
@show g'(10);

Expected output is 2, then 3
Actual output is 2, then 2 again.

Demo:

julia> using Zygote;

julia> g(x) = 2x;

julia> @show g'(10);
(g')(10) = 2

julia> g(x) = 3x;

julia> @show g'(10);
(g')(10) = 2

Slow AD for 1/sin(x)?

julia> s1(x) = sin(x)
s1 (generic function with 1 method)

julia> @btime Zygote.derivative(s1, pi/4)
  19.910 ns (0 allocations: 0 bytes)
0.7071067811865476

julia> s2(x) = 1/sin(x)
s2 (generic function with 1 method)

julia> @btime Zygote.derivative(s2, pi/4)
  1.485 μs (16 allocations: 560 bytes)
-1.4142135623730954

Automatic differenciation for complex numbers

I find Zygote supports complex number AD, I like it more now! e.g.

julia> gradient(imag, x)
((re = nothing, im = 1),)

But I have several questions,

Question 1:

Why it returns a NamedTuple rather than a complex number.

Question 2:

Why the following expression complaints not implemented error

gradient(x-> log(x) |> abs, x)

log is holomophic and complex gradient for abs is already defined in Zygote, there is no reason to break here. I suggest to create some traits for functions (is it possible?!) to help program know log is holomophic.

Question 3:

About convention, is the way of Zygote defining gradients for complex numbers proper?
We have two conventions here

zygote: gradient
partial derivative: https://giggleliu.github.io/2018/02/01/complex_bp.html

I think they are conjugate to each other.

I have implementation of the latter one here
https://github.com/GiggleLiu/poorman_nn/blob/44b54d4a86f689f528a301e3a4db6b05210bb16a/poornn/functions.py#L1034

I choose the latter convention because we want gradient(log, x) = log'(x) = 1/x rather than gradient(log, x) = log'(x)' = 1/x^*, it is more convenient for holomophic functions, right?

Unexpected output for identity function

julia> using Zygote

julia> f(x) = x
f (generic function with 1 method)

julia> f'(1.0)
true

I was not expecting true for f'(1.0).

Zygote fails to build

On a fresh julia 1.0.1, after ] add Zygote#master IRTools#master

julia> using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
ERROR: LoadError: LoadError: LoadError: Need a function definition
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] gradm(::Expr, ::Bool) at /home/antoine/.julia/packages/Zygote/NV7Nj/src/lib/grad.jl:19 (repeats 2 times)
 [3] @adjoint(::LineNumberNode, ::Module, ::Any) at /home/antoine/.julia/packages/Zygote/NV7Nj/src/lib/grad.jl:51
 [4] include at ./boot.jl:317 [inlined]
 [5] include_relative(::Module, ::String) at ./loading.jl:1041
 [6] include at ./sysimg.jl:29 [inlined]
 [7] include(::String) at /home/antoine/.julia/packages/Zygote/NV7Nj/src/Zygote.jl:1
 [8] top-level scope at none:0
 [9] include at ./boot.jl:317 [inlined]
 [10] include_relative(::Module, ::String) at ./loading.jl:1041
 [11] include(::Module, ::String) at ./sysimg.jl:29
 [12] top-level scope at none:2
 [13] eval at ./boot.jl:319 [inlined]
 [14] eval(::Expr) at ./client.jl:389
 [15] top-level scope at ./none:3
in expression starting at /home/antoine/.julia/packages/Zygote/NV7Nj/src/precompile.jl:15
in expression starting at /home/antoine/.julia/packages/Zygote/NV7Nj/src/precompile.jl:15
in expression starting at /home/antoine/.julia/packages/Zygote/NV7Nj/src/Zygote.jl:42
ERROR: Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to /home/antoine/.julia/compiled/v1.0/Zygote/4kbLI.ji.

How to calculate Jacobians (or just derivatives of vector functions) with Zygote?

Why does

Zygote.forward(a -> [2.,1.]*a, 1)[2](1)

return 3 instead of [2,1]?

Am I missing a big philosophical or computational reason for the omission of non-scalar functions? Zygote is doing a form of forward AD, so the derivative of a function that maps a scalar to a vector should be cheap, right?

Function with views error when taking gradient

julia> function rosenbrock(x)
           a = one(eltype(x))
           b = 100 * a
           result = zero(eltype(x))
           x2 = @view x[2:end]
           x1 = @view x[1:end-1]
           z = @. (a - x1)^2 + b * (x2 - x1^2)^2
           return sum(z)
       end
rosenbrock (generic function with 1 method)

julia> x = rand(2);

julia> Zygote.gradient(rosenbrock, x)
ERROR: MethodError: no method matching (::Zygote.Jnew{SubArray{Float64,1,Array{Float64,1},Tuple{UnitRange{Int64}},true},Nothing})(::Array{Float64,1})
Closest candidates are:
  Jnew(::Union{Nothing, NamedTuple}) where {T, G} at /Users/kristoffer/.julia/packages/Zygote/Ohw1K/src/lib/lib.jl:175
Stacktrace:
 [1] (::getfield(Zygote, Symbol("##256#back#159")){Zygote.Jnew{SubArray{Float64,1,Array{Float64,1},Tuple{UnitRange{Int64}},true},Nothing}})(::Array{Float64,1}) at /Users/kristoffer/.julia/packages/Zygote/Ohw1K/src/lib/grad.jl:37
 [2] Type at ./subarray.jl:16 [inlined]
 [3] Type at ./subarray.jl:32 [inlined]
 [4] Type at ./subarray.jl:22 [inlined]

Zygote DiffEq Integration (Pun Intended)

MWE:

using OrdinaryDiffEq, ParameterizedFunctions, ForwardDiff

f = @ode_def begin
  dx = a*x - b*x*y
  dy = -c*y + x*y
end a b c

p = [1.5,1.0,3.0]
prob = ODEProblem(f,[1.0;1.0],(0.0,10.0),p)
t = 0.0:0.5:10.0

function G(p)
  tmp_prob = remake(prob,u0=convert.(eltype(p),prob.u0),p=p)
  sol = solve(tmp_prob,Vern9(),abstol=1e-14,reltol=1e-14,saveat=t)
  A = convert(Array,sol)
  sum(((1 .- A).^2)./2)
end
G([1.5,1.0,3.0])
res2 = ForwardDiff.gradient(G,[1.5,1.0,3.0])

using Zygote
Zygote.derivative(G,[1.5,1.0,3.0])

Enable precompilation?

@MikeInnes Is there any plan to enable precompilation in the future?

no method matching adjoint(::typeof(f))

Hello. I heard about Zygote on the web and wanted to try it. It looks really great! Many thanks for writing the package.

I copied the line of the introduction on Github with macOS 10.14 and Julia1.0.1 (this the version of Julia I have for JuliaPro. I ran it on Julia 1.0.3 and got the same problem).

using Zygote
f(x) = 5x + 3
f(10), f'(10)

The error message is:

ERROR: LoadError: MethodError: no method matching adjoint(::typeof(f))
Closest candidates are:
  adjoint(::Missing) at missing.jl:79
  adjoint(::Number) at number.jl:193
  adjoint(::LinearAlgebra.Adjoint{#s177,#s176} where #s176<:Union{StaticArray{Tuple{N},T,1} where T where N, StaticArray{Tuple{N,M},T,2} where T where M where N} where #s177) at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/StaticArrays/3ENSR/src/linalg.jl:78
  ...
Stacktrace:
 [1] top-level scope at none:0
 [2] include_string(::Module, ::String, ::String) at ./loading.jl:1005
 [3] include_string(::Module, ::String, ::String, ::Int64) at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/CodeTools/hB4Hy/src/eval.jl:30
 [4] (::getfield(Atom, Symbol("##104#109")){String,Int64,String})() at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/eval.jl:62
 [5] withpath(::getfield(Atom, Symbol("##104#109")){String,Int64,String}, ::String) at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/CodeTools/hB4Hy/src/utils.jl:30
 [6] withpath at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/eval.jl:46 [inlined]
 [7] #103 at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/eval.jl:60 [inlined]
 [8] with_logstate(::getfield(Atom, Symbol("##103#108")){String,Int64,String}, ::Base.CoreLogging.LogState) at ./logging.jl:397
 [9] with_logger at ./logging.jl:493 [inlined]
 [10] #102 at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/eval.jl:59 [inlined]
 [11] hideprompt(::getfield(Atom, Symbol("##102#107")){String,Int64,String}) at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/repl.jl:85
 [12] macro expansion at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/eval.jl:58 [inlined]
 [13] macro expansion at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Media/ItEPc/src/dynamic.jl:24 [inlined]
 [14] (::getfield(Atom, Symbol("##101#106")))(::Dict{String,Any}) at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/eval.jl:53
 [15] handlemsg(::Dict{String,Any}, ::Dict{String,Any}) at /Applications/JuliaPro-1.0.1.1.app/Contents/Resources/pkgs-1.0.1.1/packages/Atom/v2iqN/src/comm.jl:164
 [16] (::getfield(Atom, Symbol("##19#21")){Array{Any,1}})() at ./task.jl:259

Grad of comprehensions

Doing this:

using Zygote
Zygote.gradient(v -> sum([x for x in v]), [1.1,2.2,3.3])

I get

ERROR: MethodError: no method matching (::Zygote.Jnew{Base.Generator{Array{Float64,1},getfield(Main, Symbol("##4#6"))},Nothing})(::Zygote.FillArray{Int8,1})
Closest candidates are:
  Jnew(::Union{Nothing, NamedTuple}) where {T, G} at \dev\julia\Zygote-MSFT\src\lib\lib.jl:178
Stacktrace:
 [1] (::getfield(Zygote, Symbol("##258#back#162")){Zygote.Jnew{Base.Generator{Array{Float64,1},getfield(Main, Symbol("##4#6"))},Nothing}})(::Zygote.FillArray{Int8,1}) at .\dev\julia\Zygote-MSFT\src\lib\grad.jl:41
 [2] Type at .\generator.jl:32 [inlined]
ec

segfault when calling `gradient`

Possibly related to #24 and #54, I've found an even shorter example of an invocation of gradient that segfaults:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.2 (2018-11-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

julia> Zygote.gradient((a, b) -> sum(a) + sum(b), [1 2], [3 4])
([1 1], [1 1])

julia> Zygote.gradient((a, b) -> sum(a + b), [1 2], [3 4])

signal (11): Segmentation fault
in expression starting at no file:0
CreateConstInBoundsGEP1_32 at /buildworker/worker/package_linux64/build/usr/include/llvm/IR/IRBuilder.h:1292 [inlined]
emit_sparam at /buildworker/worker/package_linux64/build/src/codegen.cpp:3262
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:3923
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3615
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:3801 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6254
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1159
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1796
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
Type at ./range.jl:298 [inlined]
_forward at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
map at ./tuple.jl:163 [inlined]
_forward at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
unknown function (ip: 0x7f948f4c161d)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1831
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
axes at ./abstractarray.jl:75 [inlined]
promote_shape at ./indices.jl:145 [inlined]
_forward at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
_forward at ./arraymath.jl:45
#5 at ./REPL[3]:1 [inlined]
_forward at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface2.jl:0
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1831
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
_forward at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:21
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
forward at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:27
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:556
gradient at /home/marks/.julia/packages/Zygote/g8WMA/src/compiler/interface.jl:32
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:430
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:682
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:806
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f94abf26a3f)
unknown function (ip: 0x9)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:805
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:117 [inlined]
#28 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 36388947 (Pool: 36383022; Big: 5925); GC: 79
fish: “julia” terminated by signal SIGSEGV (Address boundary error)
marks@localhost ~/>

deepcopy of Modules not supported

I am trying to add support for AD via Zygote for LogDensityProblems in this PR: tpapp/LogDensityProblems.jl#24, but this line (currently skipped) fails with

julia> @test logdensity(ValueGradient, ∇ℓ, x) ≅ ValueGradient(f(x), -6 .* x)
Error During Test at REPL[28]:1
  Test threw exception
  Expression: logdensity(ValueGradient, ∇ℓ, x) ≅ ValueGradient(f(x), -6 .* x)
  Compiling Tuple{typeof(logdensity),Type{Value},TransformedLogDensity{TransformVariables.ArrayTransform{TransformVariables.Identity,1},typeof(f)},Array{Float64,1}}: deepcopy of Modules not supported
  Stacktrace:
   [1] error(::String) at ./error.jl:33
   [2] deepcopy_internal(::Module, ::IdDict{Any,Any}) at ./deepcopy.jl:34
   [3] _deepcopy_array_t(::Any, ::Type, ::IdDict{Any,Any}) at ./deepcopy.jl:91
   [4] deepcopy_internal(::Array{Any,1}, ::IdDict{Any,Any}) at ./deepcopy.jl:78
   [5] deepcopy_internal(::Any, ::IdDict{Any,Any}) at ./deepcopy.jl:67
   [6] _deepcopy_array_t(::Any, ::Type, ::IdDict{Any,Any}) at ./deepcopy.jl:91
   [7] deepcopy_internal at ./deepcopy.jl:78 [inlined]
   [8] deepcopy at ./deepcopy.jl:28 [inlined]
   [9] Core.Compiler.IRCode(::IRTools.Meta) at /home/tamas/.julia/packages/IRTools/yAuDJ/src/ir/wrap.jl:104
   [10] _lookup_grad(::Type) at /home/tamas/.julia/packages/Zygote/Ohw1K/src/compiler/emit.jl:124
   [11] #s53#851 at /home/tamas/.julia/packages/Zygote/Ohw1K/src/compiler/interface2.jl:17 [inlined]
   [12] #s53#851(::Any, ::Any, ::Any) at ./none:0
   [13] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:522
   [14] #3 at /home/tamas/code/julia/LogDensityProblems/src/LogDensityProblems.jl:237 [inlined]
   [15] (::Zygote.Pullback{Tuple{getfield(LogDensityProblems, Symbol("##3#4")){TransformedLogDensity{TransformVariables.ArrayTransform{TransformVariables.Identity,1},typeof(f)}},Array{Float64,1}},Tuple{getfield(Zygote, Symbol("##222#back#144")){getfield(Zygote, Symbol("##142#143")){Zygote.Context,getfield(LogDensityProblems, Symbol("##3#4")){TransformedLogDensity{TransformVariables.ArrayTransform{TransformVariables.Identity,1},typeof(f)}},Symbol,TransformedLogDensity{TransformVariables.ArrayTransform{TransformVariables.Identity,1},typeof(f)}}},Zygote.Pullback{Tuple{typeof(logdensity),Type{Value},TransformedLogDensity{TransformVariables.ArrayTransform{TransformVariables.Identity,1},typeof(f)},Array{Float64,1}},Tuple{typeof(logdensity)}},getfield(Zygote, Symbol("##234#back#147")){getfield(Zygote, Symbol("#back#146")){:value,Zygote.Context,Value{Float64},Float64}}}})(::Int8) at /home/tamas/.julia/packages/Zygote/Ohw1K/src/compiler/interface2.jl:0
   [16] #66 at /home/tamas/.julia/packages/Zygote/Ohw1K/src/compiler/interface.jl:38 [inlined]
   [17] logdensity(::Type{ValueGradient}, ::LogDensityProblems.ZygoteGradientLogDensity{TransformedLogDensity{TransformVariables.ArrayTransform{TransformVariables.Identity,1},typeof(f)}}, ::Array{Float64,1}) at /home/tamas/code/julia/LogDensityProblems/src/AD_Zygote.jl:20
   [18] top-level scope at none:0
   [19] eval(::Module, ::Any) at ./boot.jl:328
   [20] eval_user_input(::Any, ::REPL.REPLBackend) at /home/tamas/src/julia-git/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:85
   [21] run_backend(::REPL.REPLBackend) at /home/tamas/.julia/packages/Revise/gStbk/src/Revise.jl:771
   [22] (::getfield(Revise, Symbol("##58#60")){REPL.REPLBackend})() at ./task.jl:259
ERROR: There was an error during testing

I realize the example is far from minimal, but if I am doing something obviously wrong, I would appreciate advice about it.

Using Zygote failed on Julia v1.0.0, MacOS

The following procedure works on Ubuntu 16.04, Julia v1.0.0.
But fails on MacOS 10.13.6, Julia v1.0.0

(v1.0) pkg> activate test
[ Info: new environment will be placed at /Users/jc/test

(test) pkg> add Zygote
 Resolving package versions...
 [ ......some packages added]

julia> using Zygote
ERROR: schedule: Task not runnable
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] enq_work at ./event.jl:88 [inlined]
 [3] #schedule#428 at ./event.jl:134 [inlined]
 [4] schedule at ./event.jl:129 [inlined]
 [5] notify(::Condition, ::Nothing, ::Bool, ::Bool) at ./event.jl:68
 [6] put!(::Channel{Any}, ::Tuple{Expr,Int64}) at ./channels.jl:261
 [7] send_to_backend(::Any, ::Any, ::Any) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:687
 [8] send_to_backend at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:683 [inlined]
 [9] (::getfield(REPL, Symbol("#do_respond#40")){Bool,getfield(REPL, Symbol("##50#59")){REPL.LineEditREPL,REPL.REPLHistoryProvider},REPL.LineEditREPL,REPL.LineEdit.Prompt})(::Any, ::Any, ::Any) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:702
 [10] #invokelatest#1 at ./essentials.jl:686 [inlined]
 [11] invokelatest at ./essentials.jl:685 [inlined]
 [12] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/LineEdit.jl:2261
 [13] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:1029
 [14] run_repl(::REPL.AbstractREPL, ::Any) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:191
 [15] (::getfield(Base, Symbol("##720#722")){Bool,Bool,Bool,Bool})(::Module) at ./logging.jl:311
 [16] #invokelatest#1 at ./essentials.jl:686 [inlined]
 [17] invokelatest at ./essentials.jl:685 [inlined]
 [18] macro expansion at ./logging.jl:308 [inlined]
 [19] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:330
 [20] exec_options(::Base.JLOptions) at ./client.jl:242
 [21] _start() at ./client.jl:421

fluxml / zygote.jl Goto Github PK

zygote.jl's Introduction

zygote.jl's People

Contributors

Stargazers

Watchers

Forkers

zygote.jl's Issues

Code:

Demo:

Question 1:

Question 2:

Question 3:

Recommend Projects

Recommend Topics

Recommend Org