Fine-tuning vs Training from Scratch about finetune-detr HOT 6 CLOSED

Sorry for my delayed response, for some reason I do not receive GH notifications anymore.

Many thanks for the helpful hints. These ideas sound promising. I came to realize that this is far from a trivial problem though, even if conceptually what I want to achieve is pretty much straightforward / common sense.

I will do some more research, run some experiments, and keep you posted.

Again, thank you very much for the help and the quick responses.

from finetune-detr.

And more importantly, let's say the custom dataset has annotations for object categories A and B while COCO has annotations for object categories C and D. By fine-tuning on COCO then on the custom dataset (or the opposite), it would be possible to have a final model that detects all 4 categories, right? Which obviously would not be possible if fine-tuning only on custom or COCO.

from finetune-detr.

I agree with you: in most use cases, it is better to fine-tune one of the pre-trained models rather than to train a model from scratch.

However, in the case where your dataset is both i) big enough, and ii) much different from the dataset used for the pre-trained models, then it could make sense to train from scratch, because i) you would have enough data to train a model from scratch, and ii) the pre-trained model might not help, or worse may lead to sub-optimal results compared to training from scratch.

That being said, fine-tuning could help you get good results much faster, so it is debatable whether training from scratch should be preferred over fine-tuning, even in this rare case scenario.

I believe the idea conveyed by the DETR team was that you should not expect much if you train from scratch on a small dataset, which was the case of many people eager to test the code of the official repository back then. A clarified statement could have been:

• if your dataset has less than 10k images, then fine-tuning is the only recommended option.
• otherwise, you can choose between fine-tuning and training from scratch, but fine-tuning is still the option to consider first. :)

from finetune-detr.

Many thanks for the quick response and the helpful clarification.

Also, since I posted my questions above, I did some research, and apparently, fine-tuning a model on different datasets sequentially is far from trivial (if possible at all).

I wanted to fine-tune a model to detect object categories A and B on dataset 1 and then fine-tune it on dataset 2 to detect object categories C, D, and E, hoping to have a final model that would detect A, B, C, D and E.
But I realized you need a different classification head for each task (basically you cannot have the exact same architecture across tasks, only the same backbone, right?), and even if the architecture was exactly the same, the model would forget about classes A and B after being fine-tuned on dataset 2.

Merging datasets 1 and 2 beforehand is not an option, since dataset 1 has annotations only for A and B while dataset 2 has annotations only for C, D, and E... The set of annotations for a given dataset need to cover the same categories. Merging the two datasets if this is not the case would cause a "forgetting" phenomenon, as the model would basically be taught to "unlearn" about A and B during fine-tuning on dataset 2.

This is a pity because there are similarities between all these objects, so in terms of learning, the model would benefit from getting exposed to both datasets. Having to finetune separate models is not very elegant and not optimal...

Anyhow, this is where I am at the moment... Am I missing something, do you have any thoughts on the above?

Thx

from finetune-detr.

Wait, maybe fine-tuning sequentially on datasets 1 and 2 could still be beneficial even if the upper layers are changed, because the weights of the backbone are still updated during fine-tuning, and these guys do transfer across tasks... 🤔

from finetune-detr.

As you mentioned, there is a risk that the model forgets about one dataset after being fine-tuned on the other dataset.

I won't be able to help much about fine-tuning sequentially on different datasets.

That being said, I would consider one of these options:

• merging the two datasets so that fine-tuning can be done with one model in one go,
• independently fine-tuning two models (one per dataset), which would differ by the classification head, the rest being frozen,
• sequentially fine-tuning the classification head with all classes of interest on both datasets (*), hoping there is no forgetting,
• freezing the backbone right away, then fine-tuning as in (*),
• fine-tuning on the biggest or the most diverse of the two datasets, then freezing the backbone, finally fine-tuning as in (*).

Actually, I imagine that if you freeze everything but the classification head at the opportune time (right away or after fine-tuning on one dataset), then fine-tuning the classification heads on both datasets independently and merging the results in a unified classification head should be doable. This way, you would have a single model at the end. Maybe there is some hidden difficulty.

Anyway, the best solution must be using a bit of:

• freezing the right weights at the right time,
• sequentially fine-tuning the classification head on both datasets,
• (optionally) merging datasets or models.

There may be other ways to tackle this issue, but I don't know enough about this subject to be able to provide better advice.

from finetune-detr.