Inaccuracy in Joint Annotations of Ev2Hands-R Dataset about ev2hands HOT 5 CLOSED

Hi,

Thanks for trying out the dataset. The bent fingers are an issue with the tracking, as I did not use a proper skeleton size for the hand. This led the Captury software to fit the hand-keypoints with a default bone length (longer). Hence the fingers appear bent.

To resolve this problem, we have created MANO parameters that fit very close to the GT. Below is the 3D overlay of the MANO on the Event and RGB streams.

Best,
Christen

from ev2hands.

Oh yes, I tried plotting the joints obtained from the data provided in MANO parameters; they fit much better to the input frames compared to the joints in the .pickle files in the Ev2Hands-R dataset (obtained from Captury). Thank you for your help :)

Concerning this new information, I have a couple of questions.

1. How did you estimate these MANO parameters? Could you share the corresponding code?

2. According to the code, the joints used as targets in the code for finetuning and evaluation correspond to the ones obtained by Captury and not the ones corresponding to these MANO parameters. This is based on the following observations:

   • The loss uses forward_non_mano_data (losses.py) as mano_gt is set to 0 (ev2hands_r.py)
   • The dataset uses the pickle file provided in the Ev2Hands-R dataset (evaluation_stream.py), which appears to contain the joints obtained from Captury.

Please correct me if I am missing something.

--
Pratik

from ev2hands.

Hi,

Answering to your question,

1. The following procedure is used for estimating the MANO parameters,
   1. Computing 2D keypoints from the multi-view camera images.
   2. Triangluate the 2D keypoints and obtain the 3D keypoints.
   3. Use the 3D keypoints to perform IK and obtain the MANO parameters.

I am not planning to release the code for this procedure but it is very similar to EasyMocap

2. Unfortunately, I found this issue after doing the evaluations. So, the metrics reported in the paper are evaluated with the Captury joints.

We also tentatively plan to release more data with more participants (with better joint fitting) and the camera extrinsics for the event + RGB camera.

Best,
Christen

from ev2hands.

Your answers help clear up my doubts about the dataset. Thank you!

The camera extrinsic between the Event and RGB cameras, along with the RGB camera intrinsic, would be very useful. I look forward to this release.

from ev2hands.

Hi Pratik,

The camera parameters are added along with the MANO parameters.

Best,
Christen

from ev2hands.