timkimd / rnntools.jl Goto Github PK

A Julia package for training recurrent neural networks (RNNs), vanilla neural ordinary differential equations (nODEs) and gated neural ordinary differential equations (gnODEs).

License: MIT License

Julia 99.58% Shell 0.42%

neural-differential-equations

rnntools.jl's Introduction

RNNTools.jl

This repository contains code to accompany the paper: Trainability, Expressivity and Interpretability in Gated Neural ODEs, ICML 2023 (arXiv). RNNTools.jl is a Julia package for training recurrent neural networks (RNNs) and neural ordinary differential equations (nODEs).

Installation

The package was tested on Julia v1.6, but other versions may also work. To install RNNTools.jl, follow these steps:

$ module load julia/1.6.1
$ julia
julia> ]
(@v1.6) pkg> add Flux
(@v1.6) pkg> add <the address where you put the RNNTools repository>/RNNTools.jl

Getting Started

See src/RNNTools.jl and experiments/basic/basic.jl to get started on using the package.

N-Bit Flip-Flop Task

run_flipflop_array.sh: shell script used to run the flip-flop experiments.

Fixed-Amplitude 3-Bit Flip-Flop Task

n_bit_flipflop/three_bit/original/n_bit_flipflop_disc_time_array.jl: julia code used to train various networks on the fixed-amplitude 3-bit flip-flop task (Sussillo & Barak, 2013).

Variable-Amplitude 3-Bit Flip-Flop Task

n_bit_flipflop/three_bit/variable/n_bit_flipflop_varying_amp_disc_time_array_gnode.jl: julia code used to train gnODE on the variable-amplitude 3-bit flip-flop task.
n_bit_flipflop/three_bit/variable/n_bit_flipflop_varying_amp_disc_time_array_gru.jl: julia code used to train GRU on the variable-amplitude 3-bit flip-flop task.
n_bit_flipflop/three_bit/variable/n_bit_flipflop_varying_amp_disc_time_array_mgru.jl: julia code used to train mGRU on the variable-amplitude 3-bit flip-flop task.
n_bit_flipflop/three_bit/variable/n_bit_flipflop_varying_amp_disc_time_array_node.jl: julia code used to train nODE on the variable-amplitude 3-bit flip-flop task.
n_bit_flipflop/three_bit/variable/n_bit_flipflop_varying_amp_disc_time_array_rnn.jl: julia code used to train vanilla RNN on the variable-amplitude 3-bit flip-flop task.

Disk 2-Bit Flip-Flop Task

n_bit_flipflop/two_bit/disk/two_bit_flipflop_disk_disc_time_gnode.jl: julia code used to train gnODE on the disk 2-bit flip-flop task.

Fixed-Amplitude (4 Stable Fixed Points) 2-Bit Flip-Flop Task

n_bit_flipflop/two_bit/original/two_bit_flipflop_disc_time_gnode.jl: julia code used to train gnODE on the fixed-amplitude 2-bit flip-flop task (Sussillo & Barak, 2013).

Rectangle 2-Bit Flip-Flop Task

n_bit_flipflop/two_bit/rectangle/two_bit_flipflop_rectangle_disc_time_gnode.jl: julia code used to train gnODE on the rectangle 2-bit flip-flop task.

Ring 2-Bit Flip-Flop Task

n_bit_flipflop/two_bit/ring/two_bit_flipflop_ring_disc_time_gnode.jl: julia code used to train gnODE on the ring 2-bit flip-flop task.

Square 2-Bit Flip-Flop Task

n_bit_flipflop/two_bit/square/two_bit_flipflop_square_disc_time_gnode.jl: julia code used to train gnODE on the square 2-bit flip-flop task.
n_bit_flipflop/two_bit/square/two_bit_flipflop_square_disc_time_gru.jl: julia code used to train GRU on the square 2-bit flip-flop task.
n_bit_flipflop/two_bit/square/two_bit_flipflop_square_disc_time_mgru.jl: julia code used to train mGRU on the square 2-bit flip-flop task.
n_bit_flipflop/two_bit/square/two_bit_flipflop_square_disc_time_node.jl: julia code used to train nODE on the square 2-bit flip-flop task.
n_bit_flipflop/two_bit/square/two_bit_flipflop_square_disc_time_rnn.jl: julia code used to train vanilla RNN on the square 2-bit flip-flop task.

Citation

@article{kim2023gnode,
 title     = {{T}rainability, {E}xpressivity and {I}nterpretability in {G}ated {N}eural {ODE}s},
 author    = {Kim, Timothy Doyeon and Can, Tankut and Krishnamurthy, Kamesh}, 
 journal   = {Proceedings of the 40th International Conference on Machine Learning},
 year      = {2023}
}

rnntools.jl's People

Contributors

Watchers

rnntools.jl's Issues

install error

When I install the RNNTool.jl, julia return those error log:
Precompiling project...
✗ RNNTools
0 dependencies successfully precompiled in 46 seconds (238 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

RNNTools [26172e16-81c1-454e-824a-1f6790a1377f]

Failed to precompile RNNTools [26172e16-81c1-454e-824a-1f6790a1377f] to /home/benson/.julia/compiled/v1.6/RNNTools/jl_Har6Xi.
WARNING: Method definition _concrete_solve_adjoint(Union{SciMLBase.NonlinearProblem{uType, isinplace, P, F, K, PT} where PT where K where F where P where isinplace where uType, SciMLBase.SteadyStateProblem{uType, isinplace, P, F, K} where K where F where P where isinplace where uType}, Any, Nothing, Any, Any, SciMLBase.ADOriginator, Any...) in module SciMLSensitivity at /home/benson/.julia/packages/SciMLSensitivity/M3EmS/src/concrete_solve.jl:215 overwritten in module DiffEqSensitivity at /home/benson/.julia/packages/DiffEqSensitivity/Pn9H4/src/concrete_solve.jl:105.
** incremental compilation may be fatally broken for this module **

WARNING: Method definition _concrete_solve_adjoint##kw(Any, typeof(DiffEqBase._concrete_solve_adjoint), Union{SciMLBase.NonlinearProblem{uType, isinplace, P, F, K, PT} where PT where K where F where P where isinplace where uType, SciMLBase.SteadyStateProblem{uType, isinplace, P, F, K} where K where F where P where isinplace where uType}, Any, Nothing, Any, Any, SciMLBase.ADOriginator, Any...) in module SciMLSensitivity at /home/benson/.julia/packages/SciMLSensitivity/M3EmS/src/concrete_solve.jl:215 overwritten in module DiffEqSensitivity at /home/benson/.julia/packages/DiffEqSensitivity/Pn9H4/src/concrete_solve.jl:105.
** incremental compilation may be fatally broken for this module **

ERROR: LoadError: LoadError: UndefVarError: initial_params not defined
Stacktrace:
[1] getproperty(x::Module, f::Symbol)
@ Base ./Base.jl:26
[2] top-level scope
@ ~/.julia/packages/RNNTools/aJ9M6/src/utils.jl:382
[3] include(mod::Module, _path::String)
@ Base ./Base.jl:384
[4] include(x::String)
@ RNNTools ~/.julia/packages/RNNTools/aJ9M6/src/RNNTools.jl:8
[5] top-level scope
@ ~/.julia/packages/RNNTools/aJ9M6/src/RNNTools.jl:86
[6] include
@ ./Base.jl:384 [inlined]
[7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
@ Base ./loading.jl:1235
[8] top-level scope
@ none:1
[9] eval
@ ./boot.jl:360 [inlined]
[10] eval(x::Expr)
@ Base.MainInclude ./client.jl:446
[11] top-level scope
@ none:1
in expression starting at /home/benson/.julia/packages/RNNTools/aJ9M6/src/utils.jl:382
in expression starting at /home/benson/.julia/packages/RNNTools/aJ9M6/src/RNNTools.jl:8
Stacktrace:
[1] pkgerror(msg::String)
@ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:55
[2] precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, warn_loaded::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1265
[3] precompile
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:921 [inlined]
[4] #precompile#196
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:919 [inlined]
[5] precompile()
@ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:919
[6] top-level scope
@ REPL[5]:1
I check your code and found that the funtion initial_params in src/utils.jl line 386, the function return initial_params(nns[i]), but there is no definition of this funtion.