Filter Grafting for Deep Neural Networks

Filter Grafting for Deep Neural Networks

文章地址:https://arxiv.org/pdf/2001.05868.pdf
github地址https://github.com/fxmeng/filter-grafting
Abstract

文章提出While filter pruning removes these invalid filters for efficiency consideration, filter grafting re-activates them from an accuracy boosting perspective.表示通过将有用的filters替换无用的filters,则可以提升准确率。
To better perform the grafting process,we develop an entropy-based criterion to measure the information of filters and an adaptive weighting strategy for balancing the grafted information among networks。
除此之外,文章还提出了 entropy-based criterion 的filters评估方式和adaptive weighting strategy 来平衡多个network之间filters替换。

Introduction
Removing certain filters could accelerate the inference of DNNs without hurting much performance. This discovery inspires many works studying how to decide which filters are unimportant [13] and how to effectively remove the filters with tolerable performance drop。
提出了两个疑问:

  1. it is unclear that whether directly abandoning such filters and components is the best choice
  2. Besides, given multiple networks, it is unclear whether one network can learn from the others.
    在这里插入图片描述

Related Work

  • Filter Pruning.
    剪枝的作用:Filter pruning aims to remove the invalid filters to accelerate the inference of the network;剪枝采用的评估方式 L1 norm criterion、Principal Component Analysis (PCA) 、s subspace clustering to feature maps
  • Distillation and Mutual Learning
  • RePr
    在这里插入图片描述
    其中 Distillation and Mutual Learning 和 RePr 和文章提出的思路差多,区别见上表。

Filter Grafting

  • 1 . Information Source for Grafting
    文章认为the invalid filters have smaller l1 norm and have little effects for the output,但是实验证明:But after grafting, the invalid filters have larger l1 norm and begin to make more effects to DNNs
  • 2 Internal Filters 间的替换
    -在这里插入图片描述
  • 3.External Filters as Scions
    文章提出了In response to the shortcomings of adding random noise and weights inside a single network, we select external filters from other networks as scions.
    在这里插入图片描述在这里插入图片描述

Criterions for Calculating Information of Filters and Layers

  • L1 norm
    在这里插入图片描述

  • Entropy
    While l1 norm criterion only concentrates on the absolute value of filter’s weight, we pay more attention to the variation of the weight。
    在这里插入图片描述
    在这里插入图片描述
    在这里插入图片描述

  • . Adaptive Weighting in Grafting
    在这里插入图片描述
    在这里插入图片描述

Experiment

  • Selecting Useful Information Source
    在这里插入图片描述
    • Comparison of L1 norm & Entropy Criterions
      在这里插入图片描述
    • . Evaluation of Training Diversity in Grafting
      在这里插入图片描述
    • Comparing Grafting with Other Methods
      在这里插入图片描述
  • Grafting with Multiple Networks
    在这里插入图片描述

Conclusion and Discussi
作者认为 grafting algorithm有以下两个优点:

  • How to choose proper criterion to calculate the inherent information of filters in DNNs.
  • How to balance the coefficients of information among networ
  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值