International Journal of Computer Vision 2021
Jianping Gou1 · Baosheng Yu 1 · Stephen J. Maybank 2 · Dacheng T ao1
1 UBTECH Sydney AI Centre, School of Computer Science,
Faculty of Engineering, The University of Sydney, Darlington,
NSW 2008, Australia.
2 Department of Computer Science and Information Systems,
Birkbeck College, University of London, UK.
1.知识
在知识蒸馏中,最重要的三个部分是:
知识类型(knowledge type)、蒸馏方法(distillation strategies)、师生结构(teacher-student architecture)
本文着重研究的是知识类型:基于响应的知识 response-based knowledge、基于特征的知识feature-based knowledge和基于关系的知识 relation-based knowledge.
1.1基于响应的知识 response-based knowledge
response-based knowledge 通常是指教师模型中最后一个输出层的神经响应。其主要思想是直接模仿教师模型的最终预测结果,hintons设计的蒸馏算法如下:
1.2基于特征的知识feature-based knowledge
利用多级特征,这种蒸馏方式适用于更狭窄但是层数更深的网络的训练。其主要思想是直接匹配教师和学生网络的中间层的激活特征