facebookarchive / scrnns Goto Github PK

This is a self contained software accompanying the paper titled: Learning Longer Memory in Recurrent Neural Networks: http://arxiv.org/abs/1412.7753.

License: Other

Shell 1.39% Lua 98.61%

scrnns's Introduction

Structurally Constrained Recurrent Neural Network

This is a self contained software accompanying the paper titled: Learning Longer Memory in Recurrent Neural Networks: http://arxiv.org/abs/1412.7753. The code allows you to reproduce our results on two language modeling datasets:

PenntreeBank
Text8

The code implements three recurrent models:

Standard Recurrent Neural Networks
Long Short Term Memory Recurrent Neural Networks
Structurally Constrained Recurrent Neural Networks

It also allows you to play around with various hyper-parameters.

Examples

Here are some of the examples of how to use the code.

To run a standard RNN model on PenntreeBank with following hyper-parameters:
- hidden units: 100
- minibatch size: 32
- learning rate: 0.05

you type

th -i main.lua -dset ptb -name srnn_sm -nhid 100 -batchsz 32 -eta 0.05

To run a LSTM RNN model on Text8 with following hyper-parameters:
- hidden units: 100
- minibatch size: 32
- learning rate: 0.05
- unfolding depth: 20
- backprop frequency: 5

you type

th -i main.lua -dset text8 -name lstm_sm -nhid 100 -batchsz 32 -eta 0.05 -blen 20 -bfreq 5

To run a Structurally Constrained RNN model on PenntreeBank with following hyper-parameters:
- hidden units: 100
- number of constrained units: 20
- minibatch size: 32
- learning rate: 0.05
- unfolding depth: 30
- backprop frequency: 5

you type

th -i main.lua -dset text8 -name scrnn_sm -nhid 100 -nslow 20 -batchsz 32 -eta 0.05 -blen 30 -bfreq 5

To list all the options available, you need to type

th main.lua --help

Requirements

The software requires you to have the following two packages already installed on your systems:

Torch 7
fbcunn
Installation instructions for both on Ubuntu 14.04 are here: https://github.com/facebook/fbcunn/blob/master/INSTALL.md

Installing

Download the files in an appropriate directory and run the code from there. See below.

How Structurally Constrained Recurrent Neural Network Software works

The top level file is called main.lua. In order to run the code you need to run the file using torch. For example:

th -i main.lua -<option1_name> option1_val -<option2_name> option2_val ...

In order to check what all options are available, type

th -i main.lua --help

License

Structurally Constrained Recurrent Neural Network is BSD-licensed. We also provide an additional patent grant.

Other Details

See the CONTRIBUTING file for how to help out.

scrnns's People

Contributors

Stargazers

Watchers

scrnns's Issues

Questions

Hi, I have a question.

Can anybody tell me whole option values of the paper's experiment setting?

Paper represents that options used in experiment like below.

alpha: 0.95
BPTT steps: 50(SCRNN) / 10(SRNN)
BPTT freq: 5
batch size: 32
learning rate: 0.05
learning rate shrink: 1.5

Thus, I set the options with same values as above and I set rest options with default values.
But I got different results.
That's why I'm curious how to set whole option values in the paper.

Or, can what I did make differences?
When I installed this toolkit, I compiled torch GPU libraries with option "sm_30" instead of default value(sm_20) because of my GPU card's compute capability(3.0). And, before I did the experiment, I replaced nn.LookupTableGPU( ) with nn.LookupTable( ) in "mfactory.lua" file because of same reason.

Thanks.

Questions

Hello, there.

I have a question about initialization.
You know, "main.lua" has a line "torch.manualSeed(1)" at part of the initialization.
Does it work?
I thought results will not be changed because of the line.
But, I've got different results whenever I repeat it again.
Even though, most cases, the differences are not huge, sometimes they are noticeable.
Plz, give me an answer.
Thanks.

No LuaRocks module found for fb.torch.async_rng

Hi, I got the following error while running the example command:

th -i main.lua -dset text8 -name scrnn_sm -nhid 100 -nslow 20 -batchsz 32 -eta 0.05 -
blen 30 -bfreq 5
/home/nzhiltsov/torch/install/bin/luajit: /home/nzhiltsov/torch/install/share/lua/5.1/trepl/init.lua:389: /home/nzhiltsov/torch/i
nstall/share/lua/5.1/trepl/init.lua:389: /home/nzhiltsov/torch/install/share/lua/5.1/trepl/init.lua:389: module 'fb.torch.async_r
ng' not found:No LuaRocks module found for fb.torch.async_rng
no field package.preload['fb.torch.async_rng']
no file '/home/nzhiltsov/.luarocks/share/lua/5.1/fb/torch/async_rng.lua'
no file '/home/nzhiltsov/.luarocks/share/lua/5.1/fb/torch/async_rng/init.lua'
no file '/home/nzhiltsov/torch/install/share/lua/5.1/fb/torch/async_rng.lua'
no file '/home/nzhiltsov/torch/install/share/lua/5.1/fb/torch/async_rng/init.lua'
no file '/home/nzhiltsov/.luarocks/lib/lua/5.1/fb/torch/async_rng.so'
no file '/home/nzhiltsov/.luarocks/lib/lua/5.1/fb/torch/async_rng.so'
no file '/home/nzhiltsov/torch/install/lib/lua/5.1/fb/torch/async_rng.so'
no file '/home/nzhiltsov/torch/install/lib/fb/torch/async_rng.so'
no file '/home/nzhiltsov/.luarocks/lib/lua/5.1/fb.so'
no file '/home/nzhiltsov/torch/install/lib/lua/5.1/fb.so'
no file '/home/nzhiltsov/torch/install/lib/fb.so'
stack traceback:
[C]: in function 'error'
/home/nzhiltsov/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require'
main.lua:18: in main chunk
[C]: in function 'dofile'
...tsov/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x004065d0

luarocks list

Installed

rocks:

cutorch
scm-1 (installed) - /usr/local/lib/luarocks/rocks

cwrap
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

fbcomplex
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbcunn
scm-1 (installed) - /usr/local/lib/luarocks/rocks

fbdebugger
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbeditline
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbffivector
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbluaunit
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbmattorch
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbnn
scm-1 (installed) - /usr/local/lib/luarocks/rocks

fbthrift
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbtorch
scm-1 (installed) - /usr/local/lib/luarocks/rocks

fbtrepl
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

fbutil
0.1-1 (installed) - /usr/local/lib/luarocks/rocks

graph
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

lua-cjson
2.1.0-1 (installed) - /usr/local/lib/luarocks/rocks

luaffi
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

luafilesystem
1.6.3-2 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
1.6.3-2 (installed) - /usr/local/lib/luarocks/rocks

lzlib
0.4.1.53-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks

multikey
0.3-2 (installed) - /usr/local/lib/luarocks/rocks

nn
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

nngraph
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

paths
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

penlight
1.4.1-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
1.4.1-1 (installed) - /usr/local/lib/luarocks/rocks

sundown
scm-1 (installed) - /usr/local/lib/luarocks/rocks

sys
1.1-0 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
1.1-0 (installed) - /usr/local/lib/luarocks/rocks

tds
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks

torch
scm-1 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
scm-1 (installed) - /usr/local/lib/luarocks/rocks

totem
0-0 (installed) - /home/nzhiltsov/.luarocks/lib/luarocks/rocks
0-0 (installed) - /usr/local/lib/luarocks/rocks

invalid data for nn scm-1

when i run the sample in readme. th -i main.lua -dset text8 -name lstm_sm -nhid 100 -batchsz 32 -eta 0.05 -blen 20 -bfreq 5. I got this error.
torch/install/share/lua/5.1/luarocks/loader.lua:154: Invalid data in manifest file for module nn.THNN (invalid data for nn scm-1)
I have install nn. Repacing for_each with apply in /usr/local/share/lua/5.1/fbnn/Optim.lua didn't help either.
thanks.

rnn, lstm and scrnn are sharing a lot of duplicated code

wc -l rnn.lua => 365
wc -l lstm.lua => 294
wc -l scrnn.lua => 361
fgrep -xf rnn.lua lstm.lua | wc -l => 191
fgrep -xf rnn.lua scrnn.lua | wc -l => 228
fgrep -xf lstm.lua scrnn.lua | wc -l => 269

RNN not found?

When I run

   th -i main.lua -dset ptb -name srnn_sm -nhid 100 -batchsz 32 -eta 0.05

It reports the following error:

/root/torch/install/bin/luajit: main.lua:71: attempt to call global 'RNN' (a nil value)
stack traceback:
        main.lua:71: in main chunk
        [C]: in function 'dofile'
        /root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406670

Do you know how to fix it in an easy way?

facebookarchive / scrnns Goto Github PK

scrnns's Introduction

Structurally Constrained Recurrent Neural Network

Examples

Requirements

Installing

How Structurally Constrained Recurrent Neural Network Software works

License

Other Details

scrnns's People

Contributors

Stargazers

Watchers

Forkers

scrnns's Issues

rocks:

Recommend Projects

Recommend Topics

Recommend Org