Unable to Reproduce 12 Epoch Results in the Paper about openpsg HOT 5 OPEN

Thank you for your interest in our work. Maybe check out here #33 for log reference. Some potential issues could be different configs (num_gpu and lr), and the loading of pretrained models.

from openpsg.

Thanks for your reply. I've changed the base lr from 1e-3 to 1e-4 (the paper said PSGTR uses 1e-4, but the config of PSGFormer uses 1e-3, not sure whether it is correct), and the results became R@20: 0.1550; R@50: 0.1829; R@100: 0.1845, which are still worse than the reported ones.

from openpsg.

Comparing my training log and your uploaded ones, I notice that several losses and the HTriMatcher are modified/removed in your released config. Do these modifications affect the performances?

Left: The released config & Right: The uploaded training log

from openpsg.

@Jingkang50 The 60 epochs schedule can not reproduce the results as well. Could you please share the codebase used for obtaining the checkpoints? It seems that the code used for obtaining checkpoints and the current release one are different. Thanks!

from openpsg.

The released codebase is just the one that we get the final score (as a few participants from PSG challenges was able to get the corresponding results).
Are you using the config here? It seems the loss weight aligned with the Left: The released config. For the difference in the config, we delete/change the lines either because some losses are unused or they don't matter much.

from openpsg.