MIG-DARTS: towards effective differentiable architecture search by gradually mitigating the initial-channel gap between search and evaluation

Neural architecture search (NAS) based on differentiable methods has made significant progress in both search cost (GPU-days and GPU memory consumption) and network performance. However, there still exists a large gap between the search and evaluation due to the unaffordable search cost, which will cause the searched architecture to be suboptimal in the evaluation. Based on the observation of the large initial-channel gap between search and evaluation, this paper is the first to propose to gradually mitigate the initial-channel gap as the search stage proceeds to elevate the performance of evaluation architecture; meanwhile, we remove poorly performing candidate operations after each search stage to keep an acceptable search cost. To further alleviate the excessive growth of search cost brought by the progressive increase of initial-channels, this paper proposes to separate the search space, by which an individual search space with reduced candidate operations is built for normal cell and reduction cell, respectively. Moreover, this paper proposes a stability-aware stopping strategy to alleviate the problem of invalid search to reduce the search cost in GPU-days. By conducting experiments on CIFAR10 and CIFAR100 datasets, the results show that the proposed method can achieve state-of-the-art performance with a small search cost.