This project contains minimal implementations of RNN architectures trained with Backpropagation through time (BPTT) and Reservoir Computing (RC) for high-dimensional time-series forecasting. The following models are implemented:

• Long short-term memory (LSTM) trained with BPTT
• Unitary RNNs trained with BPTT
• Reservoir computers (RC) or Echo-state-networks (ESN)
• Deep reservoir computers (Deep-RC) or (Deep-ESN) (private - ask for permission from [email protected])
• Multilayered perceptron (feedforward MLP) based on a windowing approach.

Moreover, spatial parallelization of the aforementioned models are implemented according to [1]. For implementation of the method to compute the Lyapunov spectrum, please refer to the repository RNN-Lyapunov-Spectrum.

The code requires python 3.7.3, tensorflow 1.11.0 Other required packages are: matplotlib, sklearn, psutil.

• python 3.7.3
• tensorflow 1.11.0
• matplotlib, sklearn, psutil
• mpi4py (parallel implementations)

The packages can be installed as follows: you can create a virtual environment in Python3 with:

python3 -m venv venv-RNN-RC-Chaos

Then activate the virtual environment:

source venv-RNN-RC-Chaos/bin/activate

Install a version of tensorflow (paper was compiled with version 1.11, which may no longer be available), here we also tested a more recent verion 1.14, (apart from warnings the code should run fine):

pip install tensorflow==1.14.0

Install the rest of the required packages with:

pip3 install matplotlib sklearn psutil mpi4py

The code is ready to run, you can test the following demo.

The data to run a small demo are provided in the local ./Data folder

In order to run the demo in a local cluster, you can navigate to the Experiments folder, and select your desired application, e.g. Lorenz3D. There are scripts for each model. For example, you can ran a Reservoir Computer (also called Echo state network) with the following commands:

cd ./Experiments/Lorenz3D/Local
bash 01_ESN_auto.sh.sh

A statefull GRU or a parallel ESN can be run with:

bash 04_RNNStatefull_GRU.sh
bash 05_ESN_Parallel.sh

After running the command, you will see at the terminal output the training/testing progress. You can then navigate to the folder ./Results/Lorenz3D and check the different outputs of each model.

This is only a minimal version of the code under development in the CSE-lab. Please contact [email protected] if you want to get informed, take a look at the latest version, with more features, models and capabilities.

This is joint work with:

• Jaideep Pathak (website, scholar)
• Brian R. Hunt (website, scholar)
• Themis Sapsis (website, scholar)
• Michelle Girvan (website, scholar)
• Edward Ott (website, scholar)
• Petros Koumoutsakos (website, scholar)

[1] P.R. Vlachas, J. Pathak, B.R. Hunt et al., Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics. Neural Networks, 2020 (doi: https://doi.org/10.1016/j.neunet.2020.02.016.)

[2] J. Pathak, B.R. Hunt, M. Girvan, Z. Lu, and E. Ott, Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach. Physical Review Letters 120 (2), 024102, 2018

[3] P.R. Vlachas, W. Byeon, Z.Y. Wan, T.P. Sapsis, and P. Koumoutsakos Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 474 (2213), 2018