论文链接:https://arxiv.org/pdf/1707.01219.pdf
模型加速的三种方式:
* Network pruning 剪枝
Network pruning iteratively prunes the neurons or weights of low importance based on certain criteria,
* Network quantization 量化
Network quantization tries to reduce the precision of the weights or features.
* knowledge transfer 知识迁移/蒸馏
KT based methods directly train a smaller student network, which accelerates the original networks in terms of wall time without bells and whistles. The basic idea of KD is to distill knowledge from a large teacher model into a small one by learning the class distributions provided by the teacher via softened softmax.
主要概念
Maximum Mean Discrepancy
作者引入了MMD的概念,主要是用于表达两个分布之间的距离。
a distance metric for probability distributions based on the data samples sampled from them
Neuron Selectivity Transfer
NST,是作者在这篇论文里主要介绍的方法
动机
我们要模仿teacher model特征层的输出。
为什么不直接对比teacher model 和 student model的输出呢?作者给出的解释如下:
As for distribution matching, it is not a good choice to directly match the samples from it, since it ignores the sample density in the space.
所以,我们要使用 advanced distribution alignment method 匹配。
Loss定义
作者定义的MMD loss 如下:
而其中,kernel函数的选择有如下方式
实验
目标分类
从在ImageNet数据的实验中可以看到,单独使用一个学习方式时,KD得到了最好的方式;
而KD+NST在两种方法结合的选项中,取得了最好的效果。
目标检测
而在目标检测任务中,KD+NST方法在PASCAL VOC 2007中取得了更好的效果。
参考链接
如何评价图森科技连发的三篇关于深度模型压缩的文章? - 知乎
Youtube上找到了MIT的一个tinyml的课程,很有意思
https://www.youtube.com/playlist?list=PL80kAHvQbh-ocildRaxjjBy6MR1ZsNCU7