Existing skeleton-based action recognition methods input a whole segmented action sequence and adopt later fusion to integrate the multi-stream results, which causes a large amount of computation and is not suitable for online application. This paper proposed an online skeleton-based action recognition method with multi-feature early fusion. The approach integrated different types of input feature through the early embedding layer and combined the max pooling and hierarchical pooling to extract multi-semantic spatial information. Moreover, the selection strategy of skeleton sequences was carefully designed. A new 3D skeleton dataset, NTU-GAST Skeleton, with 17 joint points was made to be compatible with existing 3D human pose estimation methods for online action recognition. Experiments on two available benchmark datasets NTU60 and 120 RGB+D indicate that the proposed method achieves competitive performance comparison to state-of-the-art results with more than 50×less computational complexity.
The code is built with the following libraries:
- Python 3.6
- PyTorch 1.3
- Download NTU-GAST Skeleton dataset from Baidu drive. Extraction code:3bid. Google drive: NTU-GAST Skeleton
- Extract the dataset to ./data/NTU-GAST-Skeleton
- We use the dataset of GAST60 (NTU-GAST60 Skeleton) as an example for description.
cd ./data/gast60/
# Preprocess .h5 file of skeletons to .pkl file
python preprocess_gast60.py
# Get the training and evaluation data
python protocol_gast60.pyWe use the dataset of NTU60 RGB+D as an example for description. We need to first dowload the NTU-RGB+D dataset.
- Extract the dataset to ./data/ntu/nturgb+d_skeletons/
- Process the data
cd ./data/ntu
# Get skeleton of each performer
python get_raw_skes_data.py
# Remove the bad skeleton
python get_raw_denoised_data.py
# Transform the skeleton to the center of the first frame
python seq_transformation.py# For the CS setting
python main.py --network ESN --train 1 --case 0
# For the CV setting
python main.py --network ESN --train 1 --case 1- Test the pre-trained models (./results/NTU60/SGN/)
# For the CS setting
python main.py --network ESN --train 0 --case 0
# For the CV setting
python main.py --network ESN--train 0 --case 1
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent