Advantage of weights-free operations(skip_connection) about pc-darts HOT 3 CLOSED

hi. @xingyueye, this phenomenon has been founded and explained by many papers. To my way of thinking , there are two aspects. First, in the early stage of the training, weights are not well learned and may produce worse features (not consistent) and weights-free operations are easier to bypass gradients. That's why we use a few epochs to warm up the super-net. Second, the if we enlarge the epochs, e.g.(100 or 200), darts may suffer from overfitting problem. Some works try to solve this problem by regularizing or early stopping [1,2,3].
[1] DARTS+: Improved Differentiable Architecture Search with Early Stopping
[2] Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
[3] Understanding and Robustifying Differentiable Architecture Search

from pc-darts.

Thanks a lot, your respond is helpful, And I would study the papers you recommended asap.
My guess is similar to your first opion, So I add warm_up to Darts sa well, but results are terrible. I set up three sets of experiments with warm_up epoch of 20, 30, and 40, but the search results show that skip_connect would appear rapidly and exceed 2 when the warmup period ends and the architecture search begins. I saw that your warm_up epoch nums is 15. So I guess that the warm_up epoch in my exp were setted unsuitable or that it differ between Darts and PC-Darts？

from pc-darts.

@xingyueye , sorry for the late reply as CVPR deadline is approaching. Original DARTS suffer lots of problems besides the mentioned. If warm up epochs are too large, DASRTS still prefer SKIP.

from pc-darts.