ask about invert_canonicalization about equiadapt HOT 2 CLOSED

Thank you for your prompt reply, which I believe has answered my question thoroughly. I will now close this issue and once again, thank you for your time and effort!

from equiadapt.

Hi @the-full, thank you for your kind and encouraging words!

Ideally, yes, you should use the invert_canonicalization function. However, we do not rely on it during the training for two reasons:

1. Our training/experiments for instance segmentation task involves training the canonicalization function only (no fine-tuning of segmentation models) with the prior regularization. Once we train a canonicalization function to learn the mapping to the prior distribution, we can use it with any architecture (in our case, both MaskRCNN and Segment-Anything Model)! Therefore, we do not use the invert_canonicalization and set task_weight as 0 in our experiments.

2. Additionally, some specific design decisions were taken to avoid using invert_canonicalization in the case of (frozen) large pretrained models (here). Effectively, if the canonicalizer is trained well, we can expect the output of the prediction network to be identical to the output for the canonical inputs (which is the ground truth because the predominant orientation in the datasets is the canonical one). However, feel free to use the function before computing the loss (and I'd recommend this if you are training from scratch and not using the prior regularization).

During inference, instead of invert_canonicalization function, we use custom invert functions for masks and bboxes.

If your issues are addressed, please go ahead and close this issue. Thank you!

from equiadapt.