cmu-perceptual-computing-lab / panoptic-toolbox Goto Github PK

Hey folks, Thanks for the great work!
I am trying to align the Kinect depth and color images using the intrinsic of color and depth camera. So far, I have not been successful. It generates the misaligned color and depth images. Please see the Matlab script for the alignment and the generated images. It would be very helpful if you can help me to resolve the issue.

align_kineoptic.zip

Hi,
Thanks again for your great work.

Can you please release identification numbers of the subjects that appears on each sequence?
I'm asking this since I'm intending to create train-test dividing of your data, in which, non of the subjects that appears in the test set are also appearing in the training set.

Hello!

Thanks for your work! With the update to the COCO19 keypoints, I believe the mapping in the doc is outdated, and no longer supported. Is there an updated mapping available?

The closest I could find is the 18 Keypoints over on OpenPose (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/media/keypoints_pose_18.png), and I imagine the 19th is the mean of the two hip joints.

Before I manually figure out the indices from the joints19 array, I just wanted to ask if there is one readily available.

Thanks again!

Hi, I found the data of some sequences are not complete, which miss hdPose3d_stage1_coco19, hdFace3d, hdHand3d and hdMeshTrack_face. These sequences are ['171026_cello1', '171026_cello2', '160226_ultimatum1', '160224_ultimatum1', '160224_ultimatum2', '160422_mafia2', '160226_mafia1', '160226_mafia2', '160224_mafia1', '160224_mafia2', '170307_dance1', '170307_dance2', '170307_dance3', '170307_dance4', '170307_dance6', '160317_moonbaby1', '160317_moonbaby2', '160317_moonbaby3', '170915_toddler2', '170915_toddler3', '170915_toddler4', '160401_ian3', '160401_ian2', '160401_ian1'].

In the json files why are the annotations' confidence lower than 100%? Should not the confidence of ground truth annotation be 100%?

Hello!

Been trying to access the server (http://domedb.perception.cs.cmu.edu/) but it seems to be timing out, both in the browser as well as while using the getData script. The Note in the README says it should be up, so just wanted to confirm if it's down again on the server side?

Thanks!

Hi!
First of all, thank you so much for this amazing dataset! The amount of work you guys put in it seems crazy.
With the recent released of the depth data I could not get happier!
But it seems that there is some issue.
The script to get the depths works properly, however, retrieving the joints information for those scenes does not work for every scene.
The scene with no joints information at all are:

• 160226_ultimatum1
• 160317_moonbaby1
• 160317_moonbaby2
• 160317_moonbaby3
• 170307_dance2
• 170307_dance3
• 170307_dance4
• 170307_dance6
• 170915_toddler2
• 170915_toddler3
• 170915_toddler4
• 171026_cello1
• 171026_cello2
• 171026_pose2
• 171204_pose1
• 171204_pose2
• 171204_pose3
• 171204_pose4
• 171204_pose5
• 171204_pose6

Moreover, it seems that some of the available scenes only have either the coco19 joints or the "joints15".
I can provide the list if needed to.
Another question regarding those joints, what is the difference between the coco19 joints and the joints_15 except for the number of joints?

Thanks for your help!

Hi!
I hope you are doing great and once again thanks for your work!
I was wondering if you were planning to release some meshes based on your "Total Capture" paper for some of the scenes or trained Frankestein/Adam models ?

Thanks!

When attempting to use the getData.sh script to download data, a "404 Not Found" error is returned for the keypoint data files (including hdPose3d, hdHand3d, and hdFoot3d). This happens with the sampleData dataset and every other data set I tested. The videos download fine.

If I revert the getData.sh script to a version from a week ago (undoing the last change), I do get a hdPose3d_stage1.tar (which is no longer downloaded by the new script), but the hdPose3d_stage1_coco19.tar returns a 404 Not Found error.

The errors are as follows:

--2018-07-19 16:27:34-- http://domedb.perception.cs.cmu.edu/webdata/dataset/sampleData/hdPose3d_stage1_coco19.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-07-19 16:27:34 ERROR 404: Not Found.

rm: hdPose3d_stage1_coco19.tar: No such file or directory
--2018-07-19 16:27:34-- http://domedb.perception.cs.cmu.edu/webdata/dataset/sampleData/hdFace3d.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-07-19 16:27:34 ERROR 404: Not Found.

rm: hdFace3d.tar: No such file or directory
--2018-07-19 16:27:34-- http://domedb.perception.cs.cmu.edu/webdata/dataset/sampleData/hdHand3d.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-07-19 16:27:34 ERROR 404: Not Found.

rm: hdHand3d.tar: No such file or directory

When running the getData.sh script on a system with curl installed but no wget, the keypoint data (or HTML for a 404 Not Found error, if the data doesn't exist) is displayed on the console, and is not saved with the videos. It works fine if you install wget, so it seems curl isn't being invoked with the required options.

It seems to me that there is an error in README about the order of kpts annotation.
I think you used the following:
0: "Neck",
1: "Nose",
2: "MidHip",
3: "LShoulder",
4: "LElbow",
5: "LWrist",
6: "LHip",
7: "LKnee",
8: "LAnkle",
9: "RShoulder",
10: "RElbow",
11: "RWrist",
12: "RHip",
13: "RKnee",
14: "RAnkle",
15: "LEye",
16: "LEar",
17: "REye",
18: "REar",

Hi, Im trying to download the kinect depth maps.
I followed http://domedb.perception.cs.cmu.edu/171204_pose1.html using /getData.sh 171204_pose1 but the command doesn't download the depth maps.

How can I Download all data from kinect sensors.
Thanks

Hi I tried download the dataset, however, the view-03 video of sequence 160906_band3 seems broken, when trying to extract hdimages, it raises moov atom not found error

https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox

Hello,

The skeleton definitions for the left and right eyes and ears do not quite match up with what you've started in your repo. Is there another format I should be considering for hdPose3d_stage1_coco19?

From my understanding, the ordering is as follows:

'''
0: Neck
1: Nose
2: BodyCenter (center of hips)
3: lShoulder
4: lElbow
5: lWrist,
6: lHip
7: lKnee
8: lAnkle
9: rShoulder
10: rElbow
11: rWrist
12: rHip
13: rKnee
14: rAnkle
15: lEye
16: lEar
17: rEye
18: rEar
'''

Thanks for great work. However what i'm doing now need intrinsic parameters and external parameters of cameras, but I dont find them in the dataset. Could you please offer these data?

I try to download the dataset but fail to connect http://domedb.perception.cs.cmu.edu/
Is it because the website is under maintenance?

Hi,
First, Thanks a lot for your magnificent work.

It seems that the file ksynctables_161202_haggling1.json is missing while running ./scripts/getData_kinoptic.sh

--2018-12-19 16:51:47-- http://domedb.perception.cs.cmu.edu/webdata/dataset/161202_haggling1/kinect_shared_depth/ksynctables.json
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-12-19 16:51:48 ERROR 404: Not Found.

Hi, thanks for providing awesome dataset and code!

I have a question about Figure. 11 in the paper.
(https://arxiv.org/abs/1612.03153)
At row 2 of Figure 11, the merged point cloud is segmented into body parts. Is there any code to reproduce the segmentation?

Thanks for considering in advance!

Currently, downloading the sequences is still not possible. Below are the command and the prompt response

--2020-10-06 13:49:58--  http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_haggling1/videos/hd_shared_crf20/hd_00_00.mp4
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 

Hello, thank you for the nice toolbox.
I would like to have some more information about the dataset, in particular either the HD camera/sensor model(s) or the sensor size in millimeters.
Since I didn't find them in the paper, is it possible to know them? Thank you kindly

Hello,

Are there frames annotated for different characters annotation?
I am asking about the range of movement data.

I mean, when we have video{1-5}, is there some clever way to separate frames belonging to the same characters other than manually looking through all the frames?

Many sequences like '160224_ultimatum1', '160224_ultimatum2' cannot be downloaded.

I also use URLs in the web browser and still return 404 error. For example:
http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_ultimatum2/hdFace3d.tar
http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_ultimatum2/hdFace3d.tar
http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_mafia2/hdPose3d_stage1_coco19.tar

When downloading pose 1 using the command "./scripts/getData.sh 171204_pose1" i get this error output in the terminal. I have attempted this on two machines and on both of them I was able to download and extract the pose 1 sample, however, pose 1 does not work.

--2020-07-23 22:31:39-- http://domedb.perception.cs.cmu.edu/webdata/dataset/171204_pose1/hdMeshTrack_face.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-07-23 22:31:40 ERROR 404: Not Found.

Running ./scripts/getData.sh generates a lot of responses like the following:

--2017-09-13 16:47:05--  http://domedb.perception.cs.cmu.edu/webdata/dataset/141217_pose1/videos/vga_shared_crf10/vga_14_03.mp4
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-09-13 16:47:05 ERROR 404: Not Found.

At In [5] : I think that the first line "vga_skel_json_path = data_path+seq_name+'/hdPose3d_stage1/'" is typo.

The variable name "vga_skel_json_path" should be changed to "hd_skel_json_path".

Thank you for making this program.

Have a good time.

We are facing problem to download the datasets because website is not working.

Hey guys,

There are 34 videos for the CMU haggling set. However, the excel sheet found here https://docs.google.com/spreadsheets/d/1eoe74dHRtoMVVFLKCTJkAtF8zqxAnoo2Nt15CYYvHEE/edit#gid=1333444170 lists only 30. Could you update this sheet?

Hello. I am trying to download some data sets, but it seems that the server is not responding to the request.

./scripts/getData.sh 170407_haggling_a1 100 10
vga_01_01.mp4
--2021-05-01 21:28:39-- http://domedb.perception.cs.cmu.edu/webdata/dataset/170407_haggling_a1/videos/vga_shared_crf10/vga_01_01.mp4
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response...

When will be available?

The website said that "they will be available soon". Are these hidden somewhere that I didn't manage to find, or were they not yet released?

How did you guys annotated the extracted frames? I mean if there was some specific tool or a proprietary method or something else.

I didn't knew where to ask this.Sorry.

I use './scripts/getData.sh 160422_mafia2 0 31' to download the 160422_mafia2 dataset. But when I download 160422_mafia2/hdPose3d_stage1_coco19.tar, I got error bellow

--2020-08-17 16:29:11-- http://domedb.perception.cs.cmu.edu/webdata/dataset/160422_mafia2/hdPose3d_stage1_coco19.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-17 16:29:12 ERROR 404: Not Found.

removed 'hdPose3d_stage1_coco19.tar'
--2020-08-17 16:29:12-- http://domedb.perception.cs.cmu.edu/webdata/dataset/160422_mafia2/hdFace3d.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-17 16:29:12 ERROR 404: Not Found.

removed 'hdFace3d.tar'
--2020-08-17 16:29:12-- http://domedb.perception.cs.cmu.edu/webdata/dataset/160422_mafia2/hdHand3d.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-17 16:29:13 ERROR 404: Not Found.

removed 'hdHand3d.tar'
--2020-08-17 16:29:13-- http://domedb.perception.cs.cmu.edu/webdata/dataset/160422_mafia2/hdMeshTrack_face.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-17 16:29:14 ERROR 404: Not Found.

What is the meaning of Z-coordinate [x1,y1,z1,c1,...........] here? Is it depth?

I opened 171204_pose3 and tried to retrieve depth point clouds using "getDB_ptCloud_ver1.sh" . I opened .dat file using:

data = np.fromfile(fname, dtype=float)

Unfortunately, .dat files contain only zeros. Do you know why?

We want to use your multi-view images and labeled pose data to train new 2D detectors. But How can we generate projecting the skeletal reconstruction outputs on all 521 views and establish their relationship to images? Can you give us some advices?

For what I'm doing I need a huge amount of low res videos, but the 3d pose estimation is aligned with the HD videos and the VGA videos seem to be synced to a different beat. I've been having to align the time manually and this is quite labor intensive, Is there a record of the time difference between VGA and HD capture?

(I am aware that the VGA and HD videos are of 25, 29.003 fps respectively.)

Excuse me, are the R and t in the calibration file the transformation between the camera coordinate and the world coordinate? If so, how should the world coordinate system be selected?

{
"name": "00_00",
"type": "hd",
"resolution": [1920,1080],
"panel": 0,
"node": 0,
"K": [
[1633.34,0,942.256],
[0,1628.84,557.344],
[0,0,1]
],
"distCoef": [-0.220878,0.189272,7.79405e-05,0.000739643,0.0418043],
"R": [
[0.1201471256,0.03330941579,-0.9921971332],
[-0.09157840245,0.9955474156,0.02233247834],
[0.9885231734,0.08818064529,0.1226625834]
],
"t": [
[23.27031863],
[126.5360495],
[284.0106483]
]
},

When running the extractAll.sh script on macOS, I get the following error:

readlink: illegal option -- f
usage: readlink [-n] [file ...]
usage: dirname path

To resolve this, I had to install the GNU version of readlink (it's in the coreutils Homebrew package), and then change the script to reference greadlink. Perhaps this can be referenced in the readme or worked around in some way.

Appreciated with your dataset!

I want to get the pose3d information calibrated with Kinect data for training. Is it possible or not?
Not good at MATLAB script, and got puzzled with KinopticStudio Toolbox soon.
Could anyone give me some ideas to start with it?

hdPose3d_stage1_coco19.tar is empty in 60422_haggling1.

The tar file takes up 3.8MB on hard disk space, but open opening, it is empty.
I am making this an issue as it may be indicative of a larger problem.

hi, thanks for your immense contribution of this huge dataset!
I'm working on 3d pose estimation and has the following question:
is the coco19 3d body pose coordinates given in millimeters or centimeters?
thanks in advance!

I was wondering whether the speaking status annotations in the haggling scenes could be published. I'm unable to do it myself, as I can't easily match the annotations in "univTime" for the body3DScene measurements with my annotations based on the video recording time.

I run ./getData.sh 171026_pose2 and download some data:
hdVideos contains 31 files
vgaVideos contains only 1 file (vga_01_01.mp4)
calibration_171026_pose2.json
hdFace3d.tar
hdHand3d.tar
hdPose3d_stage1_coco19.tar

but next I get an error:

--2019-10-23 12:02:34--  http://domedb.perception.cs.cmu.edu/webdata/dataset/171026_pose2/hdMeshTrack_face.tar
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-23 12:02:34 ERROR 404: Not Found.

removed 'hdMeshTrack_face.tar'

It would be nice if you could update the links and the information available.

Hi,
The following file can not be found:
http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_haggling1/videos/hd_shared_crf20/hd_00_21.mp4

When I using wget with the link:
$ wget http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_haggling1/videos/hd_shared_crf20/hd_00_21.mp4

I got a response:

--2020-10-14 14:31:34--  http://domedb.perception.cs.cmu.edu/webdata/dataset/160224_haggling1/videos/hd_shared_crf20/hd_00_21.mp4
Resolving domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)... 128.2.177.92
Connecting to domedb.perception.cs.cmu.edu (domedb.perception.cs.cmu.edu)|128.2.177.92|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-10-14 14:31:55 ERROR 404: Not Found.

Hi when I used getData.sh to download the data, i got the error ERROR 500: Internal Server Error. Could you guys help fix the issues? Thanks!

Whenever I try to run ./getData.sh it gives me this error.

It wasn't doing this before.

When I try to download dance sequences, such as 160317_moonbaby and 170307_dance, I found the kinect depth exists but the hdPose3d_stage1_coco19.tar are missing?

demo_kinoptic_gen_ptcloud.m
When the above program is executed, hints appear on the screen as follows:
PLY files will be saved in: F:/panoptic-toolbox/matlab/160422_haggling1/kinoptic_ptclouds
hd_index: 102, UnivTime: 44295.918
idk: 1, -1661.1570
Skipping 1
idk: 2, -1668.9130
Skipping 2.776500e+01, depth-color diff idk: 3, -1641.4520
Skipping 3
idk: 4, -1645.5540
Skipping 4
idk: 5, -1635.2900
Skipping 5
idk: 6, -1664.9270
Skipping 2.665800e+01, depth-color diff idk: 7, -1635.9900
Skipping 7
idk: 8, -1635.0910
Skipping 8
idk: 9, -1664.5840
Skipping 2.658900e+01, depth-color diff idk: 10, -1657.6420
Skipping 10
But when I open the folder F:/panoptic-toolbox/matlab/160422_haggling1/kinoptic_ptclouds
I found that This folder is empty, and there is no PLY file.

So what's the problem? How to modify the problem? Thank you!

Hi, you provide the 3D body joint data with the coco19 format, is there however a way to also access the complete feet joint data(heel, toes)?

Thank you for the great dataset!
Best,
Manuel

It has inconsonant univ_time of hd between poses and frames in synctables_171204_pose6.json.
I observed the code in demo_kinoptic_gen_ptcloud.m, projecting hd-3d-pose to kinect color frames, and found the pose is mismatched.

(body3DScene_00000500.json-50_01/50_01_00009986.jpg)

(body3DScene_00000500.json-50_02/50_02_00010004.jpg)