carlthome / tensorflow-convlstm-cell Goto Github PK

A ConvLSTM cell with layer normalization and peepholes for TensorFlow's RNN API.

License: MIT License

Python 100.00%

tensorflow-convlstm-cell's Introduction

TensorFlow ConvLSTM Cell

A ConvLSTM cell for TensorFlow's RNN API.

import tensorflow.compat.v1 as tf

tf.disable_v2_behavior()

batch_size = 32
timesteps = 100
shape = [640, 480]
kernel = [3, 3]
channels = 3
filters = 12

# Create a placeholder for videos.
inputs = tf.placeholder(tf.float32, [batch_size, timesteps] + shape + [channels])

# Add the ConvLSTM step.
from cell import ConvLSTMCell
cell = ConvLSTMCell(shape, filters, kernel)
outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=inputs.dtype)

# There's also a ConvGRUCell that is more memory efficient.
from cell import ConvGRUCell
cell = ConvGRUCell(shape, filters, kernel)
outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=inputs.dtype)

# It's also possible to enter 2D input or 4D input instead of 3D.
shape = [100]
kernel = [3]
inputs = tf.placeholder(tf.float32, [batch_size, timesteps] + shape + [channels])
cell = ConvLSTMCell(shape, filters, kernel)
outputs, state = tf.nn.bidirectional_dynamic_rnn(cell, cell, inputs, dtype=inputs.dtype)

shape = [50, 50, 50]
kernel = [1, 3, 5]
inputs = tf.placeholder(tf.float32, [batch_size, timesteps] + shape + [channels])
cell = ConvGRUCell(shape, filters, kernel)
outputs, state= tf.nn.bidirectional_dynamic_rnn(cell, cell, inputs, dtype=inputs.dtype)

tensorflow-convlstm-cell's People

Contributors

Stargazers

Watchers

Forkers

ml-lab peratham wen036 vyraun richardkelley 4skynet wanjinchang misc-git-forks benjamesbabala iamaaditya vseledkin bityangke zengxiaoqing marcociccone dukyongyoon naivescript ryadzenine ocean2045 maozhiqiang zumbalamambo peterouzh stanfordvisionsystems frank2017 fabienbaradel sanjeeku alexyoung757 jsleep zjpd shareeff sunalbert xjwxjw merlin2013 gazzola ericchg marccoru weimianli willdamon pouyaab chengmuni66 wpfhtl sanmucode stfeiseu swrd06bp astroman5516 yaoyichen amzhanghan helloworld-9527 ngchc hedandan liangxi627 remega giserh sophie7 huangpu1 tianconghua misslikewind xibinyue congweilin huazhz tungk gjetfqu muhammadshahidwandar mattangus millerwu2014 xy0806 fendaq ava4wonder magsun hitalex rabitdash afcarl donrv mysee1989 zoonono xychenunc icewwn hsuse savour-yu cjr0106 cortez-zhang geoslegend mauriceweber matinhosseiny kaylike dragoon1989 slidersun t5eng dingmc0512 gabbyhe pharrellwang angelkuooo ryany1994 preetkhaturia mfzhang sagacious-cao wuxunjin zkwalt 148492 hkxiron jiaky777

tensorflow-convlstm-cell's Issues

About video prediction

Hi. I want to use convlstm to do video prediction(N to 1). How can I stack a few convlstm layers and pass the outputs to a conv2d layer? In keras, I can stack a few convlstm2d and use return_sequences=False in the last one to make sure the dimension is right, but I don't know how to use this convlstmcell, should I use tf.contrib.rnn.MultiRNNCell?

Add GRU option to keep memory requirements under control for deeper networks

Evaluate peephole importance

The basic intuition for peepholes is that they benefit problems that require precise timing but I wonder how they work for the convolutional RNNs. The simpler version without peepholes could be calculated with fewer operations so what are the exact trade-offs?

Error when used for decoding

TypeError: 'Tensor' object is not iterable.

I use tensorflow v1.2
It seems that state cannot be a tuple. But I don't know how to modify? Can you give some suggestions?

a question

Hi,if the input is [batch_size,height,width,channel],how should I use convLSTM?

how to obtain intermediate state in ConvLSTMCell?

How to change this class to Keras Layer?

I want to make this deep learning network with Keras. This network is proposed for compressing video

One layer of this model is ConvLSTM. ConvLSTM is good for compressing sequences of images.
I know Keras has the ConvLSTM2D layer but I want to use the ConvLSTM cell.
Now I don't know how to change this class to a custom Keras Layer. Can anyone help me?
@xdurch0

get the prediction result?

If the input is with size of [batches,time_steps,shape,channels], after running the following codes, the shape of the outputs is [batches,time_step,X_size,Y_size,filters], how should I get the prediction of last time steps?
" cell = ConvLSTMCell(shape, filters, kernel)
outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=inputs.dtype)"
I found in the RNN case, they would add two varivales Weight and bias, and using the following codes to get the final prediction, but I have problems defining the Weight and bias here.
" outputs = tf.unstack(tf.transpose(outputs, [1,0,2]))
results = tf.matmul(outputs[-1], weights['out']) + biases['out'] # shape = (128, 10)

TypeError: 'Tensor' object is not iterable.

Hi,
While trying to execute:
outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=inputs.dtype)
I am getting exception:
TypeError: 'Tensor' object is not iterable.
This error occurs when the following code is executed:
c, h = state

Don't flatten and expand on each cell.call

http://stackoverflow.com/questions/41549784/can-tensorflow-rnncell-have-more-dimensions-than-3d

About padding before each time step

Thanks for this implementation, it helps save my strength,
yet there's something confuses me about ConvLSTMCell: to ensure h and x have the same row and column, don't you think we should add padding op before each time step instead of using static_rnn directly?

ConvLSTMCell

Hi carlthome,
I got the following error when using convLSTM. I don't know how to solve the error.
super(ConvLSTMCell, self).init(_reuse=reuse)
TypeError: object.init() takes no parameters

Evaluate layernorm importance

It feels to me like layernorm is extremely important when you stack several ConvLSTM layers but I'm not sure. It would be interesting to compare versions with regular bias vs. layernorm. I also wonder if it differs when you have skip-connections between layers (e.g. tf.nn.rnn_cell.ResidualWrapper) or simply stack them.

In terms of computational time layernorm is quite a bit slower (like 30%) but that should be properly benchmarked too.

Incompatible with tensorflow 1.1

Hi, I'm using tensorflow 1.1 and when I run your code it reports module 'tensorflow.python.ops.nn' has no attribute 'rnn_cell'. Then I found 'tf.nn.rnn_cell' has been replaced by 'tf.contrib.rnn' so I changed it in 'cell.py'. But when I run it again, it further reports TypeError: object.init() takes no parameters in _init super(ConvGRUCell, self).init(reuse=reuse). Do you have any ideas to fix that? Thank you.

BatchNorm parameters should be shared, only the running averages should be separate

Compare ConvGRU and ConvLSTM

How much better is the convolutional variant of a LSTM compared to just a GRU? Is it worth the extra parameters? What about computational time?

Add zero padding around concatenated weight matrices

Add batch normalization

Incorrect recurrent initialization leads to exploding gradients

Keras's implementation of Convolutional LSTM (https://www.tensorflow.org/api_docs/python/tf/keras/layers/ConvLSTM2D) uses recurrent_intializer = orthogonal to avoid exploding gradients.

This implementation uses the default initializer for the recurrent weight matrix, which is Glorot uniform. Long time series will have very bad exploding gradients when compared to the Keras layer.

This can be solved by W = tf.get_variable('kernel', self._kernel + [n, m], initializer =tensorflow.initializers.orthogonal)

Don't assume input dimensionality == num_filters

Working with varying cell-state sizes

When working with variable shaped inputs (defined by [batch_size, None, None, channels]), I get the following error in dynamic_rnn (at line 115 of rnn_cell_impl.py in TF1.2.1) during the graph construction phase:
Provided a prefix or suffix of None: Tensor("rnn_7/strided_slice:0", shape=(), dtype=int32) and (?, ?, 1024)

I get the same error when I work with your example by changing the 'shape' to [None, None] instead of [640, 480]. So, is there a way to work with inputs of varying dimensions? (Observe that for a given unrolled RNN, thiese dimensions would be fixed)

I guess this might be a related commit: https://github.com/tensorflow/tensorflow/commit/54efd636b504aad368eea254eca2970a16d457f6

how can i train it?

I would like to use convlstm in speech, but the training effect is very poor (I use adam), and the recognition rate in the test set fixed at 0.5844444 remain unchanged, I made a lot of ways or no way to change, can you send a Damo train this network, such as handwritten numbers identify.
Thank u very much!

super() problem

Hi carlthome,

I got the following problem when I tried to use the convLSTM cell.
File "demo.py", line 17, in
cell = ConvLSTMCell(shape, filters, kernel)
File "/media/MMVCNYLOCAL/MMVC_NY/Guoxian_Dai/sourcecode/convLSTM/cell.py", line 11, in init
super(ConvLSTMCell, self).init(_reuse=reuse)
TypeError: object.init() takes no parameters

ConvLSTM in tensorflow 2.8

Excuse me @carlthome !
Could you give me a hint on how to implement ConvLSTM in tensorflow 2.8?

Thanks in advance!

RNNCell not found on contrib rnn package.

First of all thank you very much for contributing this code. I was looking for testing the ConvLSTM structure with some data and it looks that you've implemented it. I'm quite new to TensorFlow and I'm having some trouble with testing your code.

On line 3 of the cell.py file there is:

class ConvLSTMCell(tf.contrib.rnn.RNNCell):

Which throughs an error complaining of to.contrib.rnn not containing RNNCell. I've gone through the different versions of TensorFlow and I cannot find a reference to RNNCell even on the latest release. What version are you using? Am I doing something wrong?

Thank you very much for your help,
Pablo

Faulty implementation of LSTM

Your implementation is incorrect. I suggest going over the paper again. You're confusing the hidden and output states.

Usage of ConvGRU

Hi, can you provide the completed example of using ConvGRU? When I invoke the cell, the Type error of super__init__ comes out. It seem super class should not take params according to the error...

Issue while implementing sample code

Hey

I happen to run the sample code mentioned in the README.

import tensorflow as tf

batch_size = 32
timesteps = 100
shape = [640, 480]
kernel = [3, 3]
channels = 3
filters = 12

inputs = tf.placeholder(tf.float32, [batch_size, timesteps] + shape + [channels])

from cell import ConvLSTMCell
cell = ConvLSTMCell(shape, filters, kernel)
outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype=inputs.dtype)

But I am getting the following error

ValueError: Dimension must be 5 but is 3 for 'transpose_1' (op: 'Transpose') with input shapes: [32,100,640,480,3], [3].

I had done the following changes in cell.py before running the above code:

Replaced tf.nn.rnn_cell with tf.contrib.rnn. I found that my version of tensorflow didn't have tf.nn.rnn_cell. So did this change as it was suggested here: tensorflow/tensorflow#8049
I was getting the following error message

    super(ConvLSTMCell, self).__init__(_reuse=reuse)

TypeError: object.__init__() takes no parameters

So I just commented this line out.

Could you please look into these issues?