Handling missing values in 2D due to occlusions and other factors about semgcn HOT 5 OPEN

Hi @saichanda ,

The current version of SemGCN cannot handle occlusions.

One potential solution might use some masks to impose 2D occlusions during network training.

Best,
Long

from semgcn.

@garyzhao ,
Thank you for the response. Thanks for the solution.
But I'm curious to know, from the paper, it is mentioned that the occlusions are handled.

we improve previous methods by a large margin for the action of directions, taking photo, posing, sitting down, walking dog and walking together. We hypothesize that this is due to the severe self-occlusions in these actions, while they can be effectively encoded by our SemGCN using relations within graphs.

Can you elaborate on what severe occlusions SemGCN is effectively encoding, if you say that the current version of SemGCN cannot handle occlusions.
Thank you.

from semgcn.

Sorry, Closed the issue by mistake.

from semgcn.

Hi @saichanda ,

Never mind.

That's a good question.

The "occlusions" you mentioned here ('-1' or '0' in the 2D keypoints) are extreme cases that one or more 2D joints are totally "vanished" in the 2D output, which cannot be handled by us.

In our paper, we expected that the 2D detector can still make some reasonable guesses when there are occlusions, which means the 2D output might not be accurate but close to the ground truth (reasonable). In this case, our method could refine the 3D prediction.

Therefore, to handle your occlusions, I suggest that you can add some masks (which randomly drop some 2D outputs just like your case) during training, which might improve the performance.

Best,
Long

from semgcn.

Sure @garyzhao ,
Thank you for the time and support.

best regards,
Sai

from semgcn.