Dropout network, DropConnect network

Notations

• input $v$
• output $r$
• weight parameter $W\in {\mathbb{R}}^{d\times m}$
• activation function $a$
• mask $m$ for vector and $M$ for matrix

Dropout

• Randomly set activations of each layer to zero with probability $1-p$.
  $$r=m\circ a(Wv),$$
  $m_j\sim \text{Bernoulli}(p)$.
• As many activation functions have the property that $a(0)=0)$, we have
  r = a ( m ∘ W v ) . r = a(m \circ Wv). r=a(m∘<