#####Info and sample codes for "NTU RGB+D Action Recognition Dataset"
NTU RGB+D is a large scale dataset for human action recognition. It has been introduced in our CVPR 2016 paper [PDF].
Please click here for more info about obtaining the dataset.
This dataset consists of 56,880 action samples containing 4 different modalities of data for each sample:
- RGB videos
- depth map sequences
- 3D skeletal data
- infrared videos
Video samples have been captured by 3 Microsoft Kinect v.2 cameras concurrently. The resolution of RGB videos are 1920ร1080, depth maps and IR videos are all in 512ร424, and 3D skeletal data contains the three dimensional locations of 25 major body joints, at each frame.
Each file/folder name in the dataset is in the format of SsssCcccPpppRrrrAaaa (e.g. S001C002P003R002A013), for which sss is the setup number, ccc is the camera ID, ppp is the performer ID, rrr is the replication number (1 or 2), and aaa is the action class label. For more details about setups, camera IDs, ... please refer to the paper.
302 of the captured samples have missing or incomplete skeleton data. If you are working on skeleton-based analysis, please ignore these files in your training and testing procedures. The list of these samples are provided here.
The title of the 60 action classes are:
- drink water
- eat meal/snack
- brushing teeth
- brushing hair
- drop
- pickup
- throw
- sitting down
- standing up (from sitting position)
- clapping
- reading
- writing
- tear up paper
- wear jacket
- take off jacket
- wear a shoe
- take off a shoe
- wear on glasses
- take off glasses
- put on a hat/cap
- take off a hat/cap
- cheer up
- hand waving
- kicking something
- put something inside pocket / take out something from pocket
- hopping (one foot jumping)
- jump up
- make a phone call/answer phone
- playing with phone/tablet
- typing on a keyboard
- pointing to something with finger
- taking a selfie
- check time (from watch)
- rub two hands together
- nod head/bow
- shake head
- wipe face
- salute
- put the palms together
- cross hands in front (say stop)
- sneeze/cough
- staggering
- falling
- touch head (headache)
- touch chest (stomachache/heart pain)
- touch back (backache)
- touch neck (neckache)
- nausea or vomiting condition
- use a fan (with hand or paper)/feeling warm
- punching/slapping other person
- kicking other person
- pushing other person
- pat on back of other person
- point finger at the other person
- hugging other person
- giving something to other person
- touch other person's pocket
- handshaking
- walking towards each other
- walking apart from each other
I have provided some MATLAB codes here to demonstrate how to read skeleton files and map them to other modalities (RGB, depth, and IR frames).
If you are interested to recieve news, updates, and future events about this dataset please subscribe in the Google group of the dataset at: https://groups.google.com/d/forum/ntu-rgbd. If you cannot access the group's page, please email me, I will add your email to the list. You can find my email in the paper, I'm the first author :)
We are going to organize an action recognition challenge based on our dataset soon. The top performing groups will be invited to ACCV 2016 Workshop "Large Scale 3D Human Activity Analysis Challenge in Depth Videos" to present their works.
The updates about this challege will be posted to the mailing list and here. Please subscribe to the mailing list if you are interested.
To cite NTU RGB+D dataset, please use this bibtex record:
@InProceedings{Shahroudy_2016_CVPR,
author = {Shahroudy, Amir and Liu, Jun and Ng, Tian-Tsong and Wang, Gang},
title = {NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2016}
}
Coming soon :)
What is the frame rate of the provided videos and depth maps in your dataset?
We have used the default 30 fps setting of Kinect V2 to capture all the modalities.
Why the individual and mutual actions are considered together? Isn't it better to separate them in our evaluations?
Having these classes of human actions together is a part of our dataset design to cope with more realistic scenarios of human action analysis. Therefore, the ideal evalution should not provide any prior info about the type of the action.
Are you going to release the Part-aware LSTM source code?
I like to do so, but it needs some cleaning up and documentation.
I have utilized Andrej Karpathy's char-rnn code to implement Part-aware LSTM.
Please take a look at it till then.
The provided MATLAB code cannot read avi files on my Linux machine.
Most probably it's a missing codec problem.
I used this solution and it worked on my own machine, hope it would help you also.