1. 超网分割
Close: Curriculum learning on the sharing extent towards better one-shot nas
K-shot nas: Learnable weight-sharing for nas with k-shot supernets
Few-shot neural architecture search
2. 梯度匹配
NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension
Generalizing few-shot nas with gradient matching
Overcoming multi-model forgetting in one-shot nas with diversity maximization
3. 手工设计
Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie-Yan Liu, and Jian Li. Analyzing and mitigating interference in neural architecture search. In ICML, 2022. 1, 2, 6, 7
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: eliminating unfair advantages in differentiable architecture search. In ECCV, 2020. 2
4. Sampling technology
4.1 PA&DA: Jointly Sampling PAth and DAta for Consistent NAS
Aim: | 提升超网权重共享时,内部子网排序的一致性(正确性)。 |
Abstract: | 权重共享机制: One-shot NAS方法训练一个超级网络,然后继承预先训练的权重来评估子模型,从而大大降低了搜索成本。然而,一些研究指出共享权重在训练过程中存在梯度下降方向不同的问题。在超网络训练过程中会出现较大的梯度方差,从而降低了超网络排序的一致性。 |
Methods: | 为了缓解这一问题,作者提出通过联合优化PAth和DAta(PA&DA)的采样分布来显式地最小化超网训练的梯度方差。作者从理论上推导出梯度方差与采样分布之间的关系,并发现最优采样概率与路径和训练数据的归一化梯度范数成正比。 因此,作者将归一化梯度法作为路径和训练数据的重要性指标,并采用重要性采样策略进行超网训练。该方法只需要微小的计算成本来优化路径和数据的采样分布,但在超网训练过程中获得了较低的梯度方差和较好的泛化效果。 |
Conclusion: | 通过重要路径采样,和重要数据采样,来优化Ktau大小,提升超网内部子模型的一致性。 |
Keyresults: | CVPR 、Sampling、SuperNet Training 、Evolutionary Search、采样专题、中科院计算所 |
Code: | https://github.com/ShunLu91/PA-DA |