[读论文] Dynamic Neural Networks: A Survey

1 Introduction

Advantages
1) Efficiency
2) Representaion Power
3) Adaptiveness.
4) Compatibility.
5) Generality.
6) Interpretability.

2 SAMPLE-WISE DYNAMIC NETWORKS

2.1 Dynamic Architectures

2.1.1 Dynamic Depth
1) Early Exiting.
a) Cascading of DNNs.
b) Intermediate Classifiers.
c) Multi-Scale Architecture With Early Exits.

MSDNet 1) a multiscale architecture; 2) dense connections
RANet

2) Layer Skipping.
a) Halting score
b) Gating function
c) Policy network

2.1.2 Dynamic Width
1) Skipping Neurons in Fully-Connected (FC) Layers.
2) Skipping Branches in Mixture-of-Experts (MoE).
3) Skipping Channels in CNNs.
a) Multi-Stage Architectures Along the Channel Dimension.
b) Dynamic Pruning With Gating Functions.
c) Dynamic pruning based on feature activations directly

2.1.3 Dynamic Routing
1) Path Selection in Multi-Branch Structures.
2) Neural Trees and Tree-Structured Networks.
a) Soft decision tree (SDT)
b) Neural trees with deterministic routing policies
c) Tree-structured DNNs.
3) Others

2.2 Dynamic Parameters

2.2.1 Parameter Adjustment
1) Attention on Weights.
2) Kernel Shape Adaptation.

2.2.2 Weight Prediction
1) General Architectures.
2) Task-specific information

2.2.3 Dynamic Features
1) Channel-wise attention
2) Spatial-Wise Attention.
3) Dynamic Activation Functions.

3 SPATIAL-WISE DYNAMIC NETWORKS
3.1 Pixel-Level Dynamic Networks
3.1.1 Pixel-Wise Dynamic Architectures
    1) dynamic sparse convolution
    2) additional refinement,

1) Dynamic Sparse Convolution.
2) Dynamic Additional Refinement.

3.1.2 Pixel-Wise Dynamic Parameters
1) Dynamic Weights.
2) Dynamic Reception Fields.

3.2 Region-Level Dynamic Networks

3.2.1 Dynamic Transformations

3.2.2 Hard Attention on Selected Patches
1) Hard Attention With RNNs.
2) Hard Attention With Other Implementations.

3.3 Resolution-Level Dynamic Networks

3.3.1 Adaptive Scaling Ratios
3.3.2 Dynamic Resolution in Multi-Scale Architectures

4 TEMPORAL-WISE DYNAMIC NETWORKS

4.1 RNN-Based Dynamic Text Processing

4.1.1 Dynamic Update of Hidden States
1) Skipping the Update
2) Coarse Update.
3) Selective Updates in Hierarchical RNNs.

4.1.2 Temporally Early Exiting in RNNs

4.1.3 Jumping in Texts

4.2 Temporal-Wise Dynamic Video Recognition
4.2.1 Video Recognition With Dynamic RNNs
1) Dynamic Update of Hidden States.
2) Temporally Early Exiting
3) Jumping in Videos.

4.2.2 Dynamic Key Frame Sampling
1) Temporal attention
2) Sampling module

5 INFERENCE AND TRAINING