engine210 / mm-final-cosmos Goto Github PK

Take the mean score of unmatch caption instead of max score.
Testing whether it's able to minimize all the bbox score within the unmatch caption.

Originally, the paper further the unmatch caption with the bounding box which has maximum score related to itself.
Now we try the bounding box which has maximum score with match caption.

In the training process, the model will learn to match the caption and object with the highest score, but the model won’t have any information in the beginning. So we try to train with the unmatch caption only for about 3 epochs. Then add in the match caption afterward. Trying to teach the model what kind of image-text pairs should be misaligned first.