Dropout network, DropConnect network

Notations

• input $v$
• output $r$
• weight parameter $W\in {\mathbb{R}}^{d\times m}$
• activation function $a$
• mask $m$ for vector and $M$ for matrix

Dropout

• Randomly set activations of each layer to zero with probability $1-p$.
  $$r=m\circ a(Wv),$$
  $m_j\sim \text{Bernoulli}(p)$.
• As many activation functions have the property that $a(0)=0)$, we have
  $$r=a(m\circ Wv).$$

DropConnect

• Randomly set the weight of each layer to zero with probability $1-p$.
  $$r=a(M\circ Wv),$$
  $M_{ij}\sim \text{Bernoulli}(p)$.
• Each $M_{ij}$ is drawn independently for each example during training.
  The memory requirement for $M$'s grows with the size of each mini-batch, and therefore, the implementation needs to be carefully designed.
• overall model $f(x;\theta ,M)$, where θ = { W g , W , W s } \theta = \{W_g,W,W_s\} θ={Wg​,W,Ws​}
  $$\begin{array}{rl}o={\mathbb{E}}_M[f(x;\theta ,M)]& =\sum _{M}p(M)f(x;\theta ,M)\\ & =\frac{1}{|M|}\sum _{M}s(a(M\circ W)v);W_s)\text{if }p=0.5\end{array}$$

• inference (test stage)
  $$\begin{array}{rl}r& =\frac{1}{|M|}\sum _{M}a((M\circ W)v))\\ r& \approx \frac{1}{Z}\sum _{z=1}^Z{r}_z\\ & \approx \frac{1}{Z}\sum _{z=1}^{Z}a(u_z),\end{array}$$
  where $u_z\sim \mathcal{N}(pWv,p(1-p)(W\circ W)(v\circ v)$; $Z$ denotes the number of randoml samples drawn from the Gaussian distribution.
  Idea: approximate a sum of weighted Bernoulli random variables by a Gaussian random variable. Partially supported by the central limit theorem.

$\text{局限性}$:
Both techniques are suitable for fully connected layers only.