jeffreyyihuang / two-stream-action-recognition Goto Github PK
View Code? Open in Web Editor NEWUsing two stream architecture to implement a classic action recognition method on UCF101 dataset
License: MIT License
Using two stream architecture to implement a classic action recognition method on UCF101 dataset
License: MIT License
About spatial cnn,how many images do you extract from every trainning video? thank you !
TVL1 or flownet2.0?
Thanks~
首先,感谢能够有这么完整的代码和说明文档。对我帮助很大。
但是仍存在一些问题:
1.对于空间输入数据,采样率为10。对于运动输入数据,光流数据的采样率是多少?
情况1.是根据采样图像的后10帧得到的光流volume吗?
情况2.还是采样得到的帧,每10帧计算一个光流volume?
2.训练时,对运动空间,只采样3个帧;对运动空间只采样1个光流堆。那为何最初的数据处理要存储这么多的数据呢?
3.测试,对每个视频采样19帧,是为了保有足够的帧用于10个堆栈运动流中的测试。那么是否推测问题1中,光流的采样为情况2.
希望您有空能解答一下我的疑问。感谢。
Thank you for kindly sharing this code. Is this code completely corresponding to the paper ([2] Temporal Segment Networks), i.e. the whole TSN framework, except the resnet part ? or part from the TSN framework? Thank you!
hello,When I run the motion_cnn.py,I get some questions,As follows,
1、About motion_cnn.py,When I set the bath_size as more than 32 such as 64, I find the program stuck at the training stage, and some time it run fast ,but then it stuck.(My GPU is GTX1080ti,when i run the command nvidia-smi,It shows that I just use less than 4G memory)
2、Also, I find that the data loading time is similar to the batch time,It makes me confused,Why data loading needs so long ,and similiar to batch time,does the model trainning (batch time-data loading time) need little time?
hope to get your answer,thanks!
Can you please share the pretrained model and testing code? Im having hard time while trying to extracting the rgb frames and compute optical flow for activity recognition... Thank you
According to the code, there are 2 folders inside the "UCF101/tvl1_flow/" directory, "u" and "v".
When we download the zip file from this mentioned repository and extract the file, only the "u" folder is created. Can you please explain what these two folders contain and how to generate the "v" folder?
can you upload the dataset again, thanks
After using flownet2.0 method, I have generated *.flo files. But I don't know how to convert * .flo files to 2-channel optical flow image and save its x, y channel as .jpg image. Can you help me? Thanks so much.
Thank you for your kindly sharing this code. Is this code for python 2.7 or 3 ? Thank you!
I want to know how the provided flow images are normalized. Is frame-wise min-max normalization used?
Thanks for your jobs! This repo is well organized and help me a lot.
I want to know how many GPUs did you use when training a Resnet101 on UCF101 RGB frames? And how much time did you spend on training? Also, is that Bn-inception would more computational and storage efficient?
Hey @jeffreyhuang1
Thanks very much for your work! It really help me understand the papers.
I am not sure if i understand this part right. In function stackopf, should it be j instead of j-1? E.g:
flow[2*(j),:,:] = H
flow[2*(j)+1,:,:] = V
Thanks!
Hi, the spatial cnn seems normal, but I notice that the experimental result on motion cnn is a little wired.
It is much lower than that of two-stream cnn (81.0%) and that of TSN (87.2%). Also, in the two papers mentioned above, both temporal streams perform better than spatial streams.
Hi, all
I did some investigation on why the motion-cnn result is much lower than their original paper. After a simple modification, I am able to achieve 84.1% top-1 accuracy. This modification is adding transforms.FiveCrop()
to the transformation. Before this modification, the result is only 80.5%. I use pretrained model fromhttps://github.com/feichtenhofer/twostreamfusion, I think further improvement can be down with transfroms.TenCrop()
.
I think with this modification, it can bridge the gap of performance between twostream model trained on pytorch and other frameworks.
The function UCF101_splitter is not defined anywhere.
The following error is shown on running average_fusion.py
splitter = UCF101_splitter(path=ucf_list,split=ucf_split)
NameError: name 'UCF101_splitter' is not defined
I download the pre-trained model ( model_best.pth.tar), but it's damaged. So ,can I download the pre-trained model from other places? Thanks
Hi, Thanks a lot for sharing this code!
May I ask how to start to train your two-stream network and how to pre-process data?
Thanks a lot!
sorry i make a mistake,there is no problem
In spatial_cnn.py, if use print(result) directly,OSError: [Errno 4] Interrupted system call may be happen with python2.7. The code below can solve this.
try: print (result) except IOError, e: if e.errno != errno.EINTR: raise
After getting the two streams run, how can I fuse the features from each network?
Dear jeffreyhuang,
I read your readme, the input of spatial_cnn is from feichtenhofer/twostreamfusion ,
So the path in spatial_cnn and spatial_dataloader equals to feichtenhofer’s Directory.
Is it right?
When I run the spatial_cnn, I have a problem:
Traceback (most recent call last):
File "/home/tangjun/jeffreyhuang/two-stream-action-recognition/spatial_cnn.py", line 272, in
main()
File "/home/tangjun/jeffreyhuang/two-stream-action-recognition/spatial_cnn.py", line 55, in main
train_loader, test_loader, test_video = data_loader.run()
File "/home/tangjun/jeffreyhuang/two-stream-action-recognition/dataloader/spatial_dataloader.py", line 100, in run
train_loader = self.train()
File "/home/tangjun/jeffreyhuang/two-stream-action-recognition/dataloader/spatial_dataloader.py", line 133, in train
print(training_set[1][0]['img1'].size())
File "/home/tangjun/jeffreyhuang/two-stream-action-recognition/dataloader/spatial_dataloader.py", line 60, in getitem
data[key] = self.load_ucf_image(video_name, index)
File "/home/tangjun/jeffreyhuang/two-stream-action-recognition/dataloader/spatial_dataloader.py", line 30, in load_ucf_image
img = Image.open(path +str(index)+'.jpg')
File "/home/tangjun/anaconda2/lib/python2.7/site-packages/PIL/Image.py", line 2272, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: '/home/tangjun/dataset1/jpegs_256/Swing/separated_images/v_Swing_g09_c02/v_Swing_g09_c02_29.jpg'
Best regards,
Yours sincerely
Tang Jun
Hello,
thank you for your work.
l'm asking how can l use spatial and temporal model on UCF-101 for features extraction ?
Thank you
Hi:
After I run python UCF_spatial_cnn.py , then I get "can not find pickle" error ,how to get pickle files?
I am using GTX-1060 3Gb GPU. When I am trying to train, I always face the OOM problem despite using small batch size. How to avoid that problem?
The following error occurs :-
Traceback (most recent call last):
File "average_fusion.py", line 23, in
ucf_split='01')
File "D:\IP\SignLanguage\TwoStreamFusion\two-stream-action-recognition-master
dataloader\spatial_dataloader.py", line 78, in init
splitter = UCF101_splitter(path=ucf_list,split=ucf_split)
NameError: name 'UCF101_splitter' is not defined
Also, please specify the directories inside the 'UCF101' folder.
Hi,
when I download Pre-trained model, I can not unpack the files. can you upload the Pre-trained Model again.
thanks
hello , a new learner here.
I have choose the Only Testing. but the result shows that it still train data. it is so weird.
can any one give me some tips.
THX
how are u extracting video frames from video....or ur code assume that frames are already extracted?
Could you publish your pre-trained models? Thank you !
Hello,
I get some confusion about reading spatial_cnn.py.When predict the test dataset,I find that you just add all sample's predictions to be the final result ,why not use tsn network to predict it's class? for example ,every video is divied to 3 segements,and randly choose one RGB frame to predict , then average the three predictions as the final result!
for j in range(nb_data):
videoName = keys[j].split('/',1)[0]
if videoName not in self.dic_video_level_preds.keys():
self.dic_video_level_preds[videoName] = preds[j,:]
else:
self.dic_video_level_preds[videoName] += preds[j,:]
https://github.com/jeffreyhuang1/two-stream-action-recognition/blob/master/dataloader/motion_dataloader.py#L55
variable j starts from 0, so according to line 55 and 56, first frames optical flow - u and v - will be in:
flow[-2,:,:] and flow[-1,:,:]. While it makes sense for it to be flow[0,:,:] and flow[1,:,:] ?
can you help me to solve it? thanks very mach
Hi! I'm training your spatial_cnn model. I downloaded rgb images from "feichtenhofer/twostreamfusion". Then, I followed your guide (modify some functions) and used your pretrained model, but I got only 79.xx% on test set and 99.xx% on training set. I can't achieve your result, which is 82.1%.
So what's my problem? Thank you.
Unable to decompress the tar files provided. Could you verify at your end if you are able to decompress them?
"This does not look like a tar archive.
Skipping to next header.
A lone zero block at 62938.
Exiting with failure status due to previous errors"
i try to reload the spatial model but it's not useful,,,..... the report errors as follows:
torch.Size([3, 224, 224])
==> Validation data :71877frames
torch.Size([3, 224, 224])
==> Build model and setup loss and optimizer
==> loading checkpoint 'record/spatial/spatial_video_preds.pickle'
Traceback (most recent call last):
File "/home/xm/python/action/two-stream-action-recognition-master/spatial_cnn.py", line 268, in
main()
File "/home/xm/python/action/two-stream-action-recognition-master/spatial_cnn.py", line 65, in main
model.run()
File "/home/xm/python/action/two-stream-action-recognition-master/spatial_cnn.py", line 109, in run
self.resume_and_evaluate()
File "/home/xm/python/action/two-stream-action-recognition-master/spatial_cnn.py", line 93, in resume_and_evaluate
checkpoint = torch.load(self.resume)
File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 367, in load
return _load(f, map_location, pickle_module)
File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 528, in _load
magic_number = pickle_module.load(f)
Thank you for your kindly sharing this code. But I have a problem of ‘ValueError: insecure string pickle’, when run the flowing codes in spatial_dataloader and motion_dataloader,
with open('dic/frame_count.pickle', 'rb') as file:
dic_frame = pickle.load(file)
I installed the Anaconda2-4.1.1-Linux-x86_64. Is the version too low?
I tried to modify 'rb' to 'r', but still the same problem. Can you help me? Thank you!
hi
I m totally new to deep learning having no experience with torch.
Would u please mention hardware and software requirements.... GPU used, OS...?
Do pytorch support windows 7 and other libararies used in implementation..?
Thanks
Hi,
Thanks a lot for providing the open-source code!
May I ask how did you get your pre-trained motion ResNet101? It's pre-trained on ImageNet and fine-tuned on UCF101 or the just changing the channels of the model trained on ImageNet?
Highly appreciate your time and help!
How do I test it either realtime or using video as an using through VideoCaputre?
Dear Jeffrey,
Thank you for your kindly sharing the great two-stream-action-recognition code. Though I have read the readme and reference papers[1] [2] [3], I was puzzled about some details.
In training strategies, in every mini-batch, you randomly select 64 (batch size) videos from 9537 training videos and further randomly select 1 stacked optical flow in each video. I wonder which paper this method was proposed. Is this method part of TSN paper[2]?
In testing method, for every 3783 testing videos, we uniformly sample 19 frames in each video and the video level prediction is the voting result of all 19 frame level predictions. This method seems like TSN[2], but the 19 segments of 10 frames maybe overlapped.
In your validation stage, the precision stored in opf_test.csv is the test precision or validation precision? The loaded data is the testlist01?
The motion_video_preds.pickle save the video level prediction. What does it mean? And where to find the test accuracy in split 1 testlist01?
Is this code corresponding to your own paper? I want to quote your paper. Or can you give me some details about your code? Thank you very much.
I send an E-mail for your help. Thank you very much.
Best Regard,
tangjun
Tried to extract pretrained model using tar...but its giving error:
tar: This does not look like a tar archive
tar: Skipping to next header
tar: A lone zero block at 79811
tar: Exiting with failure status due to previous errors
Any idea?
on what basis you are only taking 3 random frames per video for video-level prediction?
Could you please justify this I just want to know whats the reason behind this if you could let me, that would be great
Thanks
Hareesh
a new learner here.
I have noticed the dataloader.I have a question is that the result is the first split of result or it is the average of three splits.
hi, may i know whether this program can be modified to run in real time using webcam?
real-time extracting frames and features and make prediction.
Hi
On running code i am getting "kernel died and restarting" on line self.bn1 = nn.BatchNorm3d( mid_planes).
if i comment this code line then it runs but again same error occurs wherever batch normalization is used. Any idea what could be the reason.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.