sdobber / fluxarchitectures.jl Goto Github PK

View Code? Open in Web Editor NEW

119.0 119.0 16.0 45.36 MB

Complex neural network examples for Flux.jl

License: MIT License

Julia 100.00%

fluxarchitectures.jl's People

Contributors

Stargazers

Watchers

Forkers

hedgefair zeta1999 cbdavis mcreel wtwong316 acheryauka udemirezen bgladman cybervalient multidis matthew-whisenant luboshanus brittany-erickson emmanuel-r8 alba-intelligence

fluxarchitectures.jl's Issues

Doubt about Stacked LSTMs

Hi! First of all, thanks a lot for FluxArchitectures.jl! :)

More than an issue, this is a question because of my misunderstanding. While checking the documentation, I realized that I might have been using a wrong implementation of a Stacked LSTM in my codes. However, I’m confused with your current implementation.

I was comparing the description of the Stacked LSTM from your blog with your current implementation and I don’t understand why you are not using the HiddenRecur anymore. Now it seems like you are chaining the LSTMs just in the regular way, matching the inner dimensions but not feeding the inner cells with the previous hidden states and memories.

Also, in

function (m::StackedLSTMCell)(x)
	out = m.chain(x)
	m.state = out
	return out
end

you are saving only the output in the state of the StackedLSTMCell, so I don’t understand how this would be fixing the Flux issue that you describe in your blog (Flux.jl's standard setup only allows feeding the output of one cell as the new input) or is it the case that now Flux behavior has changed fixing this issue?

Size of Input in LeNet5 of LSTNet

Hi, I didn't find any other way to contact you.

Please, would you be willing to tell me how the network works in a way that you can input batches of different lengths without using reset!? Is it because the LeNet5() does some reshaping? I am not much familiar with convolutional networks. I'd like to forecast a basic AR model using just LSTM for example.

I am aware of many examples at Discourse, and it makes sense to use reset! on hidden states. But you do not in your function that is why I am asking.

Thanks!

DSAnet is broken

Upon training, all output values are NaN.

DA-RNN Multi-step Prediction

As DA-RNN is basically a encoder-decoder network, how would you do multi step prediction in the decoder phase?

Latest Zygote/Flux throws error in DARNN example file

In training loop:

ERROR: MethodError: no method matching JuliennedArrays.Slices(::ChainRulesCore.NoTangent, ::Int64)

`get_data` creates NaNs on the GPU

input, target = get_data(:exchange_rate, poollength, datalength, horizon) |> gpu works fine on the CPU, but creates some NaNs at the beginning of the dataset (at least on a Jetson Nano).

Register package

@JuliaRegistrator register

Tests fail on Julia 1.10

DARNN test fail in the backward pass with type Nothing has no field method. This is probably coming from FluxML/Zygote.jl#1410.

Unfortunately, Zygote cannot be updated currently, see #48

Horizon must be larger than poollength in examples

With horizon <= poollength, the RNN has direct access to the value is is supposed to predict.

Thanks @luboshanus for finding this issue.

Error running LSTnet - Copy-Paste Code example

Running Julia 1.8.0 on Windows 11, I copy-pasted the example code at the top of the examples page into a new Julia environment after adding FluxArchitectures and Plots packages. When it reached the line Flux.train!, it gave the following stack trace:

julia> Flux.train!(loss, Flux.params(model),Iterators.repeated((input, target), 20), ADAM(0.01), cb=cb)
ERROR: BoundsError: attempt to access Tuple{} at index [0]
Stacktrace:
  [1] getindex(t::Tuple, i::Int64)
    @ Base .\tuple.jl:29
  [2] last(a::Tuple{})
    @ Base .\abstractarray.jl:479
  [3] rrule(config::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::typeof(foldl), op::Base.var"#57#58"{typeof(Flux.reset!)}, x::Tuple{}; init::Nothing)
    @ ChainRules C:\Users\joel\.julia\packages\ChainRules\fgVxV\src\rulesets\Base\mapreduce.jl:448
  [4] chain_rrule_kw(::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::Function, ::NamedTuple{(:init,), Tuple{Nothing}}, ::Function, ::Function, ::Vararg{Any})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:230
  [5] macro expansion
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0 [inlined]
  [6] _pullback(::Zygote.Context{true}, ::Base.var"#foldl##kw", ::NamedTuple{(:init,), Tuple{Nothing}}, ::typeof(foldl), ::Base.var"#57#58"{typeof(Flux.reset!)}, ::Tuple{})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:9
  [7] _pullback
    @ .\tuple.jl:555 [inlined]
  [8] _pullback(::Zygote.Context{true}, ::typeof(foreach), ::typeof(Flux.reset!), ::Tuple{})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
  [9] _pullback
    @ C:\Users\joel\.julia\packages\Flux\EXOFx\src\layers\recurrent.jl:180 [inlined]
 [10] _pullback(ctx::Zygote.Context{true}, f::typeof(Flux.reset!), args::typeof(relu))
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [11] _pullback
    @ .\abstractarray.jl:2774 [inlined]
 [12] _pullback(::Zygote.Context{true}, ::typeof(foreach), ::typeof(Flux.reset!), ::NamedTuple{(:σ, :weight, :bias, :stride, :pad, :dilation, :groups), Tuple{typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Int64}, NTuple{4, Int64}, Tuple{Int64, Int64}, Int64}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [13] _pullback
    @ C:\Users\joel\.julia\packages\Flux\EXOFx\src\layers\recurrent.jl:180 [inlined]
 [14] _pullback
    @ .\tuple.jl:555 [inlined]
 [15] #rrule_via_ad#46
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:255 [inlined]
 [16] rrule_via_ad
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:243 [inlined]
 [17] #1678
    @ C:\Users\joel\.julia\packages\ChainRules\fgVxV\src\rulesets\Base\mapreduce.jl:444 [inlined]
 [18] BottomRF
    @ .\reduce.jl:81 [inlined]
 [19] #836
    @ .\accumulate.jl:291 [inlined]
 [20] afoldl(::Base.var"#836#837"{Base.BottomRF{ChainRules.var"#1678#1680"{Zygote.ZygoteRuleConfig{Zygote.Context{true}}, Base.var"#57#58"{typeof(Flux.reset!)}}}}, ::Tuple{Tuple{}, Tuple{Nothing, Nothing}}, ::Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, ::Function)
    @ Base .\operators.jl:548
 [21] accumulate(op::Function, xs::Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"}; init::Tuple{Nothing, Nothing})
    @ Base .\accumulate.jl:290
 [22] rrule(config::Zygote.ZygoteRuleConfig{Zygote.Context{true}}, ::typeof(foldl), op::Base.var"#57#58"{typeof(Flux.reset!)}, x::Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"}; init::Nothing)
    @ ChainRules C:\Users\joel\.julia\packages\ChainRules\fgVxV\src\rulesets\Base\mapreduce.jl:440
 [23] chain_rrule_kw
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:230 [inlined]
 [24] macro expansion
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0 [inlined]
 [25] _pullback(::Zygote.Context{true}, ::Base.var"#foldl##kw", ::NamedTuple{(:init,), Tuple{Nothing}}, ::typeof(foldl), ::Base.var"#57#58"{typeof(Flux.reset!)}, ::Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:9
 [26] _pullback
    @ .\tuple.jl:555 [inlined]
 [27] _pullback(::Zygote.Context{true}, ::typeof(foreach), ::typeof(Flux.reset!), ::Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [28] _pullback
    @ C:\Users\joel\.julia\packages\Flux\EXOFx\src\layers\recurrent.jl:180 [inlined]
 [29] _pullback(ctx::Zygote.Context{true}, f::typeof(Flux.reset!), args::Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [30] _pullback
    @ .\abstractarray.jl:2774 [inlined]
 [31] _pullback(::Zygote.Context{true}, ::typeof(foreach), ::typeof(Flux.reset!), ::NamedTuple{(:layers,), Tuple{Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"}}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [32] _pullback
    @ C:\Users\joel\.julia\packages\Flux\EXOFx\src\layers\recurrent.jl:180 [inlined]
 [33] _pullback
    @ .\tuple.jl:555 [inlined]
 [34] #rrule_via_ad#46
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:255 [inlined]
 [35] rrule_via_ad
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:243 [inlined]
 [36] #1678
    @ C:\Users\joel\.julia\packages\ChainRules\fgVxV\src\rulesets\Base\mapreduce.jl:444 [inlined]
 [37] BottomRF
    @ .\reduce.jl:81 [inlined]
 [38] #836
    @ .\accumulate.jl:291 [inlined]
 [39] afoldl
    @ .\operators.jl:548 [inlined]
 [40] #accumulate#835
    @ .\accumulate.jl:290 [inlined]
 [41] #rrule#1677
    @ C:\Users\joel\.julia\packages\ChainRules\fgVxV\src\rulesets\Base\mapreduce.jl:440 [inlined]
 [42] chain_rrule_kw
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\chainrules.jl:230 [inlined]
 [43] macro expansion
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0 [inlined]
 [44] _pullback
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:9 [inlined]
 [45] _pullback
    @ .\tuple.jl:555 [inlined]
 [46] _pullback(::Zygote.Context{true}, ::typeof(foreach), ::typeof(Flux.reset!), ::Tuple{Chain{Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"}}, FluxArchitectures.Seq{Flux.Recur{FluxArchitectures.ReluGRUCell{CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}}, FluxArchitectures.SeqSkip{FluxArchitectures.ReluGRUCell{CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}, Int64}, Chain{Tuple{Dense{typeof(identity), CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, Chain{Tuple{FluxArchitectures.var"#62#64", Dense{typeof(identity), CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [47] _pullback
    @ C:\Users\joel\.julia\packages\FluxArchitectures\9iPjE\src\LSTnet.jl:111 [inlined]
 [48] _pullback(ctx::Zygote.Context{true}, f::typeof(Flux.reset!), args::FluxArchitectures.LSTnetCell{Chain{Tuple{Conv{2, 4, typeof(relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, FluxArchitectures.var"#61#63"}}, FluxArchitectures.Seq{Flux.Recur{FluxArchitectures.ReluGRUCell{CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}}, FluxArchitectures.SeqSkip{FluxArchitectures.ReluGRUCell{CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}}, Int64}, Chain{Tuple{Dense{typeof(identity), CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, Chain{Tuple{FluxArchitectures.var"#62#64", Dense{typeof(identity), CUDA.CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [49] _pullback
    @ .\REPL[16]:2 [inlined]
 [50] _pullback(::Zygote.Context{true}, ::typeof(loss), ::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, ::CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [51] _apply
    @ .\boot.jl:816 [inlined]
 [52] adjoint
    @ C:\Users\joel\.julia\packages\Zygote\qGFGD\src\lib\lib.jl:203 [inlined]
 [53] _pullback
    @ C:\Users\joel\.julia\packages\ZygoteRules\AIbCs\src\adjoint.jl:65 [inlined]
 [54] _pullback
    @ C:\Users\joel\.julia\packages\Flux\EXOFx\src\optimise\train.jl:120 [inlined]
 [55] _pullback(::Zygote.Context{true}, ::Flux.Optimise.var"#37#40"{typeof(loss), Tuple{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface2.jl:0
 [56] pullback(f::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface.jl:373
 [57] gradient(f::Function, args::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}})
    @ Zygote C:\Users\joel\.julia\packages\Zygote\qGFGD\src\compiler\interface.jl:96
 [58] macro expansion
    @ C:\Users\joel\.julia\packages\Flux\EXOFx\src\optimise\train.jl:119 [inlined]
 [59] macro expansion
    @ C:\Users\joel\.julia\packages\ProgressLogging\6KXlp\src\ProgressLogging.jl:328 [inlined]
 [60] train!(loss::Function, ps::Zygote.Params{Zygote.Buffer{Any, Vector{Any}}}, data::Base.Iterators.Take{Base.Iterators.Repeated{Tuple{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, opt::Adam; cb::var"#1#2")
    @ Flux.Optimise C:\Users\joel\.julia\packages\Flux\EXOFx\src\optimise\train.jl:117
 [61] top-level scope
    @ REPL[20]:1

Issues training models on FluxBench

Ref https://buildkite.com/julialang/fluxbench-dot-jl/builds/84#627a51df-aa8d-4357-963d-877ff8fdef41

some models seem to fail. this is likely due to changes made to Zygote. Is it possible to do a bisect to find out which version of Zygote broke differentiability of certain models? We found this in Metalhead as well, but I would love to have a good idea of what caused this. It is likely aligned with when Zygote started wrapping ChainRules' outputs.

Fix recurrent network calls

Currently, the hidden state of recurrent network parts is not updated in the right way, as all data fed to them is considerered to be a single timestep. Instead, it needs to be fed as a sequence so that the hidden state gets updated after each timestep.

One way of doing this is by using Flux.unstack and Flux.stack. The drawback is that this makes computations extremely slow.

TPA-LSTM trains slower on GPU than on CPU

Apparently, the construction with Flux.unstack and Flux.stack is much slower than the 'slow' Zygote.Buffer. The latter cannot be used on the GPU due to missing support for array mutation.

How to run DARNN on GPU

I have tried to run DARNN model on GPU, But some error has occurred.
I modified some code below
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
encoder_lstm = Seq(HiddenRecur(Flux.LSTMCell(inp, encodersize) |> gpu))
decoder_lstm = Seq(HiddenRecur(Flux.LSTMCell(1, decodersize) |> gpu ))
#= @inbounds =# for t in 1:m.poollength # for gpu
input = input |> gpu
target = target |> gpu
model = model |> gpu
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Error message is

Warning: haskey(::TargetIterator, name::String) is deprecated, use Target(; name = name) !== nothing instead.
│ caller = llvm_compat(::VersionNumber) at compatibility.jl:176
└ @ CUDAnative ~/.julia/packages/CUDAnative/ierw8/src/compatibility.jl:176
[ Info: CUDA is on
[ Info: device = gpu
[ Info: Training
[ Info: epochs = 1
ERROR: LoadError: MethodError: no method matching (::var"#loss#30")(::CuArrays.CuArray{Float32,4,Nothing}, ::CuArrays.CuArray{Float32,1,Nothing})
Stacktrace:
[1] macro expansion at /home/jee/.julia/packages/Zygote/1GXzF/src/compiler/interface2.jl:0 [inlined]
[2] _pullback(::Zygote.Context, ::var"#loss#30", ::CuArrays.CuArray{Float32,4,Nothing}, ::CuArrays.CuArray{Float32,1,Nothing}) at /home/jee/.julia/packages/Zygote/1GXzF/src/compiler/interface2.jl:13
[3] loss at /home/jee/Projects/Julia/learn/FluxArch/DARNN/e2.jl:82 [inlined]
[4] _pullback(::Zygote.Context, ::var"#loss#30") at /home/jee/.julia/packages/Zygote/1GXzF/src/compiler/interface2.jl:0
[5] pullback(::Function, ::Zygote.Params) at /home/jee/.julia/packages/Zygote/1GXzF/src/compiler/interface.jl:172
[6] gradient(::Function, ::Zygote.Params) at /home/jee/.julia/packages/Zygote/1GXzF/src/compiler/interface.jl:53
[7] train2() at /home/jee/Projects/Julia/learn/FluxArch/DARNN/e2.jl:85
[8] top-level scope at /home/jee/Projects/Julia/learn/FluxArch/DARNN/e2.jl:92
[9] include(::Module, ::String) at ./Base.jl:377
[10] exec_options(::Base.JLOptions) at ./client.jl:288
[11] _start() at ./client.jl:484

Would you to help me to fix the model?

Better example data

Include better training data for the models. Could for example be based on the UCI Machine Learning Repository.

Implementing TPALSTM using own data

I am trying to implement TPALSTM using my own data [link](https://www.kaggle.com/datasets/shenba/time-series-datasets/discussion?select=sales-of-shampoo-over-a-three-ye.csv). How can I go about it?

SkipGRU not using correct hidden state

The SkipGRU layer does not use the correct previous hidden state, but only the current ones.

question, not an issue. How to get the predicted value ?

I'am testing with a test set and want the next value after the time series. Horizon is 1....
So after 800 values, how to fetch the predicted 801 ?

Compilation fails on Julia 1.7

Creating a system image with FluxArchitectures fails when using PackageCompiler 2.0.2 and Julia 1.7. It works fine with Julia 1.6.4.

See: JuliaLang/PackageCompiler.jl#643

Zygote v0.6.44 breaks tests

With Zygote v0.6.44, the tests break when doing a gradient calculation on a DARNN network. Might be related to FluxML/Zygote.jl#1111

Zygote must be pinned to versions lower than 0.6.44

Otherwise, the backward pass on DARNN fails.

Seems to be related to FluxML/Zygote.jl#1304

FluxArchitectures not working with Flux v0.13

Change from Diagonal to Scale messes things up.

Training fails on sample datasets.

Hello, first of all i want to thank you for this repo because it is the only one i found with some recent "complex" implementation in julia.

im trying the example of tpa-lstm but it seems it is not able to forecast local minima (after a while it turn into a line moving up and down) (I’m not sure if I should try with a larger dataset)
The DSANet Instead output no "pred" (nan32) instead.
any idea?

(julia 1.5.1 and i have all the pkg used updated)

question on how the examples work

Hello, i have a few questions regarding the exmaples:
1)from what i understood the model (lets say LSTnet) get as input a target and the data with features (same size of the target + target itself) then update params to make it fit the initial target and the result is an array with the length of the initial size/target
i dont get why the model already get as input most of the target (here m.ConvLayer(x)) (aside from the last few shifted target with the horizon)
so it could just output the input as target with random the last part of the horizon and it would still return an almost "perfect fit"?

assuming it actually forecast without already getting as input the part it should forecast:

3)is the forecast value at time x only last point of the horizon to forecast from time y = x - horizon for any point of the time series?

how is the loss calculated if the last few part in the plot (the horizon of the forecast) are outside the range of the input target (at least in the plot)?

im trying to recreate something like this: https://github.com/jingw2/demand_forecast using these models, and maybe it is the same thing the examples.jl here do when plotting the last part of the horizon outside the target range, but im not sure.

Allow Zygote updates past v0.5.5

Updating from Zygote v0.5.5 to v0.5.6 breaks training in the function a -> dropdims(a, dims = (findall(size(a) .== 1)...,)). Direct workaround would be replacement with a -> a[1,1,:,:]. See also issue #4.

Data for Electricity has 321 users not 370

Hi,

I would like to check if I see it correctly that in papers researcher refer to the data with 370 users and the data here have 321.
https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014

If so, do you think it is possible to update?

Thank you very much.

Luboš

Slices not defined

I try to run TPALSTM example on julia 1.9 but it throws error that 'Slices' not defined

exception =
│ UndefVarError: Slices not defined
│ Stacktrace:
│ [1] (::FluxArchitectures.Seq{FluxArchitectures.StackedLSTMCell{Chain{Tuple{Flux.Recur{Flux.LSTMCell{Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Tuple{Matrix{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Matrix{Float32}}}, Flux.Recur{Flux.LSTMCell{Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Tuple{Matrix{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Matrix{Float32}}}}}, Vector{Float32}}})(#unused#::Vector{Float32}, #unused#::FluxArchitectures.StackedLSTMCell{Chain{Tuple{Flux.Recur{Flux.LSTMCell{Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Tuple{Matrix{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Matrix{Float32}}}, Flux.Recur{Flux.LSTMCell{Matrix{Float32}, Matrix{Float32}, Vector{Float32}, Tuple{Matrix{Float32}, Matrix{Float32}}}, Tuple{Matrix{Float32}, Matrix{Float32}}}}}, Vector{Float32}}, x::Matrix{Float32})

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

LSTnet throws Method Error in training

ERROR: MethodError: no method matching iterate(::Nothing) in Flux.train!.

How can LSTnet output a 2 dims times Array?

Hi,
I'm trying to use LSTNet for weather elements forecast.They are 2 dims times mat for a weather station. When I use LSTNet, It out :

the ["PRS", "TEM", "RHU", "U", "V", "PRE_1h", "GST", "lat", "lon", "RHO", "ARED", "AGREEN", "ABLUE"] 1/3
┌ Warning: Size mismatch in loss function! In future this will be an error. In Flux <= 0.12 broadcasting accepts this, but may not give sensible results
│ summary(ŷ) = "1×25 Matrix{Float32}"
│ summary(y) = "13×25 Matrix{Float32}"

Could you give me some advice?
Thank you.

Upcoming Flux v0.12 will possibly break models

Recurrent neural network code will change in a future Flux release:

Type Recur gets field init removed.
GRUCell, LSTMCell etc. get field h for the initial hidden state removed.