《Teacher Guided Architecture Search》论文阅读笔记

Motivation: "if we could evaluate the quality of a network’s internal features(KeyWord1) even in a very premature state, we might be able to more quickly determine if a given architecture is likely to obtain high levels of mature performance. "

Q1-(KeyWord1): How to determine the quality of a network’s internal features？

解决思路： use features of a high performing “teacher” network as a reference to identify promising sample architectures at a much earlier premature state.

具体做法：

‘Teacher Guided Search for Architectures by Generation and Evaluation’-

1、确定$n_a$输入集。

2、计算teacher和search结构中每一层activations的RDM矩阵。

3、计算search结构与teacher结构RDMs的相似程度Score，计算公式为——

4、取所有search得到结构的RDMs-Score的平均值为TG，最终的模型性能评估公式为（P为传统的性能评估，如交叉熵等，$\alpha$是超参数）：

算法亮点：

1、虽然每一回training的计算量增大了，但整体需要的epoch次数变少了，因此最终需要的计算量也变少了。【Section4.1】

2、如果某模型无法用计算机实现（如大脑的视觉皮层的感知），也可以用该模型作teacher。【Section4.4】

3、RDM的使用。