mansimov / unsupervised-videos Goto Github PK
View Code? Open in Web Editor NEWUnsupervised Learning of Video Representations using LSTMs
Home Page: http://arxiv.org/abs/1502.04681
License: MIT License
Unsupervised Learning of Video Representations using LSTMs
Home Page: http://arxiv.org/abs/1502.04681
License: MIT License
envy@ub1404:/os_pri/github/unsupervised-videos$ python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1/os_pri/github/unsupervised-videos$
invalid device ordinal
Traceback (most recent call last):
File "lstm_combo.py", line 402, in
board = LockGPU(board=board_id)
File "/home/envy/os_pri/github/unsupervised-videos/util.py", line 144, in LockGPU
cm.cuda_set_device(board)
File "/home/envy/os_pri/github/unsupervised-videos/cudamat/cudamat.py", line 2273, in cuda_set_device
raise generate_exception(err_code)
cudamat.cudamat.CUDAMatException: CUDA error: no error
envy@ub1404:
Does anyone know how data is stored in both Moving MNIST and UFC101 datasets? I want to try on my own data.
Thanks,
I got the following while running;
python display_results.py models/lstm_combo_1layer_mnist_pretrained.pbtxt datasets/bouncing_mnist_valid.pbtxt 0
Error message:
Using board 0 2048 Traceback (most recent call last): File "display_results.py", line 22, in <module> main() File "display_results.py", line 9, in main lstm_autoencoder = LSTMCombo(model) File "/home/arghya/unsupervised-videos/lstm_combo.py", line 18, in __init__ self.lstm_stack_enc_.Add(lstm.LSTM(l)) File "/home/arghya/unsupervised-videos/lstm.py", line 21, in __init__ self.w_dense_ = Param((4 * num_lstms, num_lstms), lstm_config.w_dense) File "/home/arghya/unsupervised-videos/util.py", line 50, in __init__ mat = f[config.dataset_name].value File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-build-4NvvlI/h5py/h5py/_objects.c:2682) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-build-4NvvlI/h5py/h5py/_objects.c:2640) File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/group.py", line 166, in __getitem__ oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-build-4NvvlI/h5py/h5py/_objects.c:2682) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-build-4NvvlI/h5py/h5py/_objects.c:2640) File "h5py/h5o.pyx", line 190, in h5py.h5o.open (/tmp/pip-build-4NvvlI/h5py/h5py/h5o.c:3568) KeyError: "Unable to open object (Object 'lstm_1_enc:w_dense' doesn't exist)"
envy@ub1404:/os_pri/github/unsupervised-videos$ python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1/os_pri/github/unsupervised-videos$
Traceback (most recent call last):
File "lstm_combo.py", line 1, in
from data_handler import *
File "/home/envy/os_pri/github/unsupervised-videos/data_handler.py", line 3, in
from util import *
File "/home/envy/os_pri/github/unsupervised-videos/util.py", line 16, in
import config_pb2
ImportError: No module named config_pb2
envy@ub1404:
Instead of relying on math libraries, it looks like you implement your own kernel for forward propagation. I heard that a small number adding with a large number can cause numerical problems. In your program, how do you handle this problem?
Hi there,
I'm trying to run the the first example using the MNIST dataset but it keeps crashing.
Using board 0
2048
Traceback (most recent call last):
File "lstm_combo.py", line 398, in
main()
File "lstm_combo.py", line 385, in main
lstm_autoencoder = LSTMCombo(model)
File "lstm_combo.py", line 18, in init
self.lstm_stack_enc_.Add(lstm.LSTM(l))
File "/Users/shirazis/Documents/Codes/unsupervised-videos/lstm.py", line 33, in init
self.w_input_ = Param((4 * num_lstms, self.input_dims_), lstm_config.w_input)
File "/Users/shirazis/Documents/Codes/unsupervised-videos/util.py", line 29, in init
self.dw_ = cm.empty_like(self.w_)
File "/Users/shirazis/Documents/Codes/unsupervised-videos/cudamat/cudamat.py", line 1883, in empty_like
cmat = empty(m.shape)
File "/Users/shirazis/Documents/Codes/unsupervised-videos/cudamat/cudamat.py", line 1872, in empty
raise generate_exception(err_code)
Can you please help me to figure out what's happening?
Cheers,
Sareh
In cudamat/cudamat_kernels.cu line 1934, the code is "if (!init) d_c_in[p] = grad_c;".
But in the first state of decoder and future LSTM, both of their init_state are the last state of encoder. So I guess that the derivation of encoder's last cell state should be propagated from both decoder and future LSTM. Should the code be "if (!init) d_c_in[p] += grad_c;" ?
Hi,
Thank you very much for providing the code for the paper. I have a small doubt regarding using your code to predict video frames on some other video dataset. Could you kindly share some insight on how to do that?
Also, could you kindly tell me if the .npy file contains videos (in frames) of the entire videos in dataset like in a 5-D matrix (video_number,frame_no,channels,height,width) format?
Thank you very much.
The authors mentioned about "We initialize an LSTM classifier with the weights learned by the encoder LSTM from this model. "in their paper, but I am a beginner and I don't understand how to initialize the lstm classifier?
The Figure 6 shows that the lstm classifier is two layers ,could your tell how to implement it ?
Is there anyone can help me with this? Thanks a lot.
@emansim
I am using an Amazon Machine Image (AMI) to launch a GPU instance on Amazon EC2 with CUDA 7.5 and the required libraries installed. When I type make in your cudamat directory, everything appears to work. I am able to compile the .proto file and download the datasets. However, when I attempt to train a model, I get this error:
ubuntu@ip-172-31-9-162:~/unsupervised-videos-master$ python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1
invalid device ordinal
Traceback (most recent call last):
File "lstm_combo.py", line 402, in
board = LockGPU(board=board_id)
File "/home/ubuntu/unsupervised-videos-master/util.py", line 144, in LockGPU
cm.cuda_set_device(board)
File "/home/ubuntu/unsupervised-videos-master/cudamat/cudamat.py", line 2273, in cuda_set_device
raise generate_exception(err_code)
cudamat.cudamat.CUDAMatException: CUDA error: no error
Is your cudamat setup compatible with CUDA 7.5? Is it necessary to use your custom directory?
Can you help me find the source of this error? I'd appreciate it very much.
Best,
Tina
Hi Emansim,
I try to train a model on MNIST dataset, but I cannot find libcudamat_conv_gemm.so.
Can you please help me to figure out what's happening?
Cheers,
python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 0
Traceback (most recent call last):
File "lstm_combo.py", line 1, in
from data_handler import *
File "/home/smile/project/unsupervised-videos-master/data_handler.py", line 3, in
from util import *
File "/home/smile/project/unsupervised-videos-master/util.py", line 4, in
from cudamat import cudamat_conv_gemm as cc
File "/home/smile/project/unsupervised-videos-master/cudamat/cudamat_conv_gemm.py", line 4, in
_ConvNet = ct.cdll.LoadLibrary('libcudamat_conv_gemm.so')
File "/home/smile/anaconda2/lib/python2.7/ctypes/init.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/home/smile/anaconda2/lib/python2.7/ctypes/init.py", line 365, in init
self._handle = _dlopen(self._name, mode)
OSError: libcudamat_conv_gemm.so: cannot open shared object file: No such file or directory
Dear developers,
I am a newbie in LSTM so I read your code carefully to gain a deep understanding.
Basically, the code for Fprop runs like:
for each batch: for each LSTM: LSTM.Fprop()
where batch is the "input_frame" with shape (batch_size, input_dims) and LSTM units are stored in list "models_".
Let's take video patch as an example, i.e., "python lstm_combo.py models/lstm_combo_1layer_ucf101_patches.pbtxt datasets/ucf101_patches.pbtxt datasets/ucf101_patches_valid.pbtxt 1".
If you add print statements to show their shape or length, it outputs:
len(models_) = 1
input_frame= (100, 3072)
This means there is only one LSTM object in the "models_" list. Well, if you look at Fprop of LSTM class, it allocates memory for whole network, for example, "w_input_" variable.
My questions are:
for t in xrange(self.enc_seq_length_): self.lstm_stack_enc_.Fprop(input_frame=self.v_.col_slice(t * self.num_dims_, (t+1) * self.num_dims_))
Any help will be appreciated. Thanks
Instead of giving videos, how do we test with normal matrices? for examples if i am giving 5x5 matrix then is there a way to get (let's say) a 8x8 matrix? (trained with other 5x5 matrix)
Hello,
When I try to implement
python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1
I get following error:
OSError: /nas/ei/home/ga85pav/unsupervised-videos/cudamat/libcudamat.so: cannot open shared object file: No such file or directory
I don't see the libcudamat.so in cudamat folder but it is mentioned in Makefile about it. Am I missing something?
Could someone help me on it.
Thanks!!
Hi,
I tried to run lstm_combo.py, but I got the error: OSError: dlopen(/Users/yantian/Google Drive/Yantian/DL/unsupervised-videos-master/cudamat/libcudamat.so, 6): image not found
Where can I get libcudamat.so? Thank you in advance!
Hi
thank for your code
i want only predicted output which is predicted based on previous input sequence
not to compare actual and predicted output
to predict future situation just using present situation
how can i do this??
Thx.
Hi, would you mind to share the code for generating the moving mnist video dataset?
Thanks.
I wonder if you can release an example file to demonstrate how to set up a LSTM model pre-trained by the enc-dec LSTM autoencoder network? The example you provided in the models file seems concluding only one LSTM with no timestamp.
BTW, could you explain the meaning of the xxx_num_frames.txt file?
Hi,
I have a problem with running the code. when I tried :
python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 0
I got this error:
Using board 0
2048
2048
Traceback (most recent call last):
File "lstm_combo.py", line 407, in
main()
File "lstm_combo.py", line 394, in main
lstm_autoencoder = LSTMCombo(model)
File "lstm_combo.py", line 21, in init
self.lstm_stack_dec_.Add(lstm.LSTM(l))
File "/home/fahimeh/ProgramFiles/unsupervised-videos/lstm.py", line 21, in init
self.w_dense_ = Param((4 * num_lstms, num_lstms), lstm_config.w_dense)
File "/home/fahimeh/ProgramFiles/unsupervised-videos/util.py", line 29, in init
self.dw_ = cm.empty_like(self.w_)
File "/home/fahimeh/ProgramFiles/unsupervised-videos/cudamat/cudamat.py", line 1881, in empty_like
cmat = empty(m.shape)
File "/home/fahimeh/ProgramFiles/unsupervised-videos/cudamat/cudamat.py", line 1869, in empty
raise generate_exception(err_code)
cudamat.cudamat.CUDAMatException: CUBLAS error.
Could you please help me to fix it?
Thanks a lot.
when I run
python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 1
I got this errror
"no kernel image is available for execution on the device"
my environment is
ubuntu16.10/1080ti/cuda9.1/anaconda2/python2.7
Hello Emansim and other friends,
I'm almost mad about this question and I've spent almost one week on it. This is a course project for one Master course in my university, and I have to reproduce the similar results to those in the paper.But this due time is drawing near while I'm stuck at the beginning of this project. I'm new to makefile, LSTM, google protobuf, so, please give me a hand.
Firstly, let me introduce my computer hardware and software configuration:
Windows 7 Enterprise 64-bit operating system
CUDA Toolkit v6.5
NVIDIA Quadro FX 4800 (Version 340.62)
Microsoft Visual Studio Ultimate 2013
Microsoft SDK v8.1 (Framework v4.0)
Python2.7.8
mingw32-make(after install mingw-w64-installer)
Secondly, let me introduce the steps to make Makefile in the folder cudamat:
Could anyone give me a hand? Thanks in advance.
@emansim
digit moving is in fact based on digit patch(2828), not digit itself. So the bounce-off is exactly based on digit patch. In most case, digit will bounce back before digit arrives at the edge of outer box(6464) ,due to white padding in digit patch.
Does it matter for the training? What problem will it cause?
data_handler.py about line 417
""" Put b on top of a."""
def Overlap(self, a, b):
# S1: original by author, can not make sure that b is on top of a
#return np.maximum(a, b)
#S2: digit patch overlap,consistent with bounce-off process
#return b
# S3: digit overlap, but inconsistent with bounce-off process
return np.where(b==0,a,b)
As mentioned by the notes above,
Is that a problem?
Hi, I know you guys don't answer a lot but I'll give it a shot anyway:
In util.py:
def Update(self):
if self.eps_decay_after > 0:
eps = self.eps_ * np.power(self.eps_decay_factor, self.t_ / self.eps_decay_after)
else:
eps = self.eps_
The local variable eps is never used inside the Update() function. Thus there's no learning rate decay. Is there a reason for not decaying it?
Thank you.
Hi,
I'm currently trying to do frame prediction in TensorFlow. Your paper is one of the best I have found so far.
Unfortunately, after I have downloaded your code and ran the training of the combo-1layer model for about 600,000 iterations, it looks like the frame predictor does not seem to learn anything. At least there was not improvement from the 100,000th to the 600,000th iteration, where the cross-entropy loss sticks at a value of about 2,600.
Is it therefore possible that you upload the weights (h5 file) that was used in the paper as well?
Because first, the training takes very long, seconds, even after training it on a Titan X for about 3 days (600,000 iterations) I do not get such nice results.
When running the visualization code provided in README:
python display_results.py models/lstm_combo_1layer_mnist_pretrained.pbtxt datasets/bouncing_mnist_valid.pbtxt 1
The output shows a key error in the hdf5 dataset:
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip_build_root/h5py/h5py/_objects.c:2408) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip_build_root/h5py/h5py/_objects.c:2365) File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/group.py", line 166, in __getitem__ oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip_build_root/h5py/h5py/_objects.c:2408) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip_build_root/h5py/h5py/_objects.c:2365) File "h5py/h5o.pyx", line 190, in h5py.h5o.open (/tmp/pip_build_root/h5py/h5py/h5o.c:3456)
KeyError: "Unable to open object (Object 'lstm_1_enc:w_dense' doesn't exist)"
I have changed the directories to my local dir in the pbtxt files. Any clue what's happening here? Thanks!
Hi,
I have a couple of questions regarding some design decisions of this paper/model to train on MovingMNIST:
Thx!
python lstm_combo.py models/lstm_combo_1layer_mnist.pbtxt datasets/bouncing_mnist.pbtxt datasets/bouncing_mnist_valid.pbtxt 0
Using board 0
invalid device function
Traceback (most recent call last):
File "lstm_combo.py", line 405, in
cm.CUDAMatrix.init_random(42)
File "/home/weihaoxie/unsupervised-videos/cudamat/cudamat.py", line 382, in init_random
raise generate_exception(err_code)
cudamat.cudamat.CUDAMatException: CUDA error: no error
Originally posted by @DumFun1 in Azure-Samples/nodejs-docs-hello-world#92
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.