[Project Page] [Technical Paper] [Dataset] [Sample Cases]

This repository provides tools, tutorials, source codes and supporting documents for the dataset H²TC. For a quick glance at the dataset, we also refer users to the project website and our technical paper. Briefly, it has introduced tools to record, process, and annotate the dataset.

All source codes are available in ./src. It also includes documents and tutotials in in ./docthat explain in detail the data processing, data hierarchy, annotation, and the content of each data file in the dataset.

Simply follow the steps below to run the provided tools from scratch:

1. Install the dependencies.
2. Fetch the raw data. You have two options to fetch the raw data:
   • Download our captured dataset from Dropbox.
   • Capture your own data with our provided recorder and suggested sensors. It helps build your data recording system and then collect more in-distribution data.
3. Process the raw data.
4. (Optional) Annotate the processed data with our provided annotator.

To run the tools, some dependencies have to be installed first.

First, the default and well-tested system environment is

• Ubuntu: 20.04
• Python: 3.8.13
• CUDA: 11.6
• Nvidia driver: 510

We have not tested our codes on other systems yet, so it is recommended to configure the same, or at least a similar environment for the best experience.

To run the data processor, install two more applications

• spd-say: text-to-voice converter.
• ffmpeg: video decoder

They can be installed, if not have, using apt:

sudo apt update
// install spd-say
sudo apt install speech-dispatcher 
// install ffmpeg
sudo apt install ffmpeg

The Python dependencies can be installed automatically via pip:

pip install -r requirements.txt

Our recording framework employs three ZED stereo cameras, one Prophesee event camera, a pair of StretchSense MoCap Pro (SMP) Gloves, Optitrack and other streaming sensors. Therefore, their SDK tools need to be installed to record and process the dataset.

You need to install ZED SDK (3.7.6) and Metavision SDK (2.3.0) following the official guidance, so as to use the ZED stereo camera and the Prophesee event camera to record and process the data respectively.

For user's convenience of installing the specific version (3.7.6) of ZED SDK, we fetch the installer from the official repository and save it in ./dev/ZED_SDK_Installer. All you need is to download the SDK installer, run it and select the modules you prefer following the official guides.

Metavision SDK is not packaged in an installer way, so you will have to follow the official guides to install it. Particularly, Metavision SDK provides multiple optional modules. Our tool uses only the functionality from the metavision-essentials, but you are free to install other modules or not.

Now you should be able to launch the event recorder with the following command if your Prophesee event camera is well connected to the computer:

metavision_viewer

You should also be able to test the connection of ZED cameras by running the official samples.

If you do not have a camera or do not intend to record your own dataset, you can simply check if the modules pyzed and metavision_core can be successfully imported by your python program. They will be used only for post processing of the raw data by our dataset.

Our recorder integrates the functionality of recording with multiple devices and organizing the recorded contents in a hierarchical manner. To collect data with our provided recorder, simply follow the steps below:

Step 1: Enable all recording devices.

• The ZED and Prophesee event cameras should be wired to a host Ubuntu (20.04) machine, where the recorder program is supposed to run.
• StretchSense MoCap Pro gloves should be connected to a separate Windows machine with its official client software Hand Engine (HE) running on the same machine.
• The OptiTrack server can be launched either on a separate host, or on any of the two aforementioned clients. You may need to configure the firewall on each machine to enable the User Datagram Protocol (UDP) communication.

Step 2: Update the configuration of OptiTrack NatNet client in ./src/natnet_client/src/example_main.cpp, and then rebuild it by following our tutorial. Briefly, you need to configure the OptiTrack server IP address (char* ip_address), the recorder IP address (servaddr.sin_addr), and the recorder port (PORT) according to your own network setting.

cd ./src/natnet_client
mkdir build
cd build
cmake ..
make

Step 3: Initialize your lists of human subjects and objects in /register/subjects.csv and /register/objects.csv respectively. Each subject and object should lie in a sperate line. Please refer to the sample lists in our repository for a detailed format.

Step 4: Launch the main recorder with the IP and Port of the local machine and of the HE application

python src/recorder.py --addr IP:PORT --he_addr IP:PORT     

There are also some other arguments to configure the recorder optionally:

And then run the NatNet client in another terminal:

./src/natnet_client/build/natnet_client                   

Now you should be able to see a prompt indicating that two machines have successfully communicated with each other, if everything goes well.

Step 5: Follow the interactive instructions prompted in the terminal by the main recorder to perform a recording. The main recorder will automatically communicate with and command Hand Engine and NatNet client to record multiple data modalities in a synchronous manner. Nevertheless, we do recommend you to regularly check Hand Engine and the NatNet client to see if they break.

The data processor synchronizes and converts the raw captured data into the processed data of commonly used formats, and organizes them in a hierarchical manner. Using the tool, users can easily

• Process the raw data into the formats as detailed in the table below, or
• Process the raw data into other preferred formats by modifying the provided tool.

We refer users to the data processing for full technical details on how we process the multi-modal and cross-device raw data, and to the data file explanation and our technical paper for a detailed introduction of the data hierarchy and the content of each involved data file.

You can access all raw data from Dropbox . The raw data of each recorded throw&catch activity is packed in a .zip file. First download the raw data to your own folder raw_data_path

raw_data_path
└──011998.zip           // 011998 is the take number

We provide a scripted extrator to unzip the packed raw data and also to organize all raw files in a suitable data hierarchy, as mentioned before. Run the following command below to extract the raw data to your target path target_path:

python src/extract.py --srcpath raw_data_path --tarpath target_path

--srcpath is where you download and save the packed raw files. --tarpath is the target path where you expect to save the extracted raw data files.

Each extracted zip file will be organized in a hierarchical structure under the folder target_path/data. For example, the raw data files of the recording "011998" will be organized as below

target_path
└──data
    └──011998
        ├──hand
        │   ├──P1L.csv / P1R.csv
        │   ├──P1LMeta.json / P1RMeta.json
        │   ├──P1L.cal/P1R.cal
        │   └──P1L.fbx/P1R.fbx
        ├──{zed-id}.svo  
        ├──{zed-id}.csv 
        ├──event_{timestamp}.raw
        ├──event.bias
        └──optitrack.csv

{zed-id} indicates and includes three involved ZED devices, including 17471, 24483054 and 28280967, which are the fixed third-person (side), the dynamic egocentric and the fixed third-person (back) respectively. {timestamp} is the initial timestamp of the event camera in the UNIX format. A detailed explanation of the data hierarchy and the content of each raw data file is provided in data file explanation.

Once the raw data is extracted and organized appropriately, run the following command

python src/process.py --datapath target_path/data

--datapath is where the extracted raw data files are saved. There are also some other arguments that are allowed to configure the processor optionally

You can change the default settings by adding more arguments into your command. For example, we do not export the depth depth.npy and the event xypt.csvfiles by default (i.e. xypt and npy are set False by default), as they are time/space-consuming. If you need them, simply attach --npy and --xypt to the command:

python src/process.py --datapath target_path/data --npy --xypt

Once the data processing is done, as shown below, the raw data files will be moved into a new directory target_path/data/raw/, and the processed data files will be stored in the directory target_path/data/processed/.

target_path
└──data
    └──011998
        ├──raw              // all raw data as above
        │ └──......
        └──processed        // all processed data
            ├──rgbd1/rgb2/rgb3
            │    ├──left_frame_id.png
            │    ├──depth_frame_id.png
            │    └──depth.npy
            ├──event
            │    └──frame_id.png
            ├──rgb1_ts/rgb2_ts/rgb3_ts.csv
            ├──event_xypt.csv
            ├──event_frame_ts.csv
            ├──left_hand_pose.csv
            ├──right_hand_pose.csv
            ├──sub1_head_motion.csv
            ├──sub1_left_hand_motion.csv
            ├──sub2_head_motion.csv
            ├──object.csv
            └──alignment.json

The data hierarchy and the content of each processed data file are explained in detail in data file explanation and our technical paper.

If you want to customize the processing process, please:

• First follow the Steps 1 and 2 in How to Process to fetch the organized raw data.
• Then in Step 3, customize your own processing process by diving into the full technical details in the data processing. This document explains how we process the multi-modal, cross-device data streams.

If you want to reprocess the processed takes, you will have to manually remove their entir folders first. If you only want to reprocess part(s) of a take, e.g. RGB and depth streams, you just need to remove their corresponding files (folders).

The current mechanism allows for maximally 10 failed attempts to decode (or grab in ZED term) a ZED RGB-D frame. Once the decoding process fails for more 10 times, it will abort and the data processing will continue to the next part, e.g. the next ZED device or the next stream modality. Those frames that have already been decoded will be stored, while the rest frames will be ignored. This issue usually happens to decode the last frame.

To fix this issue, one can simply reprocess the problematic takes via reprocess the processed take as above.

The dataset is provided with a hierarchy of segmentation and annotations, both semantic and dense. Briefly, each recorded throw&catch activity in H²TC is segmented into four phases, including pre-throwing, object flying, catching and post-catching, with three manually annotated moments including throw, catch_touch and catch_stable.

The subjects' behaviors are manually checked and annotated with symbolic labels in terms of grasp mode and hand locations (as shown schematically below). The subjects' exact initial standing locations and the average flying speed of the object are also automatically annotated as quantitative labels.

The complete annotation hierarchy is detailed below:

In case you want to annotate our dataset and your custom-captured data, we provide an annotator, with which one can easily annotate catch&throw activities via an interactive interface.

To use the annotator, please follow the steps below:

• Step 1: Process and store all raw data in the directory YOUR_PATH/data/take_id/processed. The raw data processing can be achieved as suggested previously in data processing.

• Step 2: Run the following command to launch the annotation tool.

  python src/annotate.py --datapath YOUR_PATH/data

There are some other arguments allowed to configure the annotation optionally:

Once running the above command, an interactive prompt interface will appear excluding the orange bars (they are figure annotations).

The interface displays multi-view RGB (left column), depth (middle column), egocentric event (top right sub-window), and hand motion (middle right sub-window) streams. Annotators can check synchronized streams frame by frame via the keyboard (left arrow and right arrow respectively).

The interface also provides an information panel (bottom right sub-window), which allows annotators to annotate the streams with keyboard and display the annotation result in real-time.

Each entry in the information pannel corresponds to an annotation, as described above,

Annotators can manually interact with the interface to select semantic labels with the keyboard as defined below, while the dense labels, e.g. the object speed and subject locations, will be automatically annotated by the interface. Any modification to the annotation result will be immediately saved in the corresponding annotation file YOUR_PATH/annotations/take_id.json.

We suggest that users rely mainly on the third-person (side) and egocentric views to annotate.The third-person (back) view can be used as an auxiliary, when significant occlusion happens in the former two views.

The viewing angle of the egocentric camera is higher than the normal height of human eyes,resulting in a top-down viewing angle. This may lead to a biased observation of the vertical hand location. You can use the third-person (back) view to provide additional information.

It is possible that some data is missing in your labeled frame. Then the annotation can not be switched to the status of "finished" due to the missing data.

For example, when you label the throw moment and if there is no OptiTrack data, the information panel appears like below. To handle the issue, you should seek closer frames that include complete data and indicate the same event of the moment. If no frame is qualified, the entire take should be annotated as "problematic" and skip to the next take.