6 训练神经网络(上)——激活函数、数据预处理

训练神经网络(上)——激活函数、数据 预处理

Activateion

Sigmoid

$$\sigma (x)=\frac{1}{1+e^{-x}}$$

• Squashes numbers to range [0,1]
• Historically popular

3 problems:

1. Saturated neurons kill the gradient
2. Sigmoid outputs are not zero-centered
3. exp() is a bit compute expensive

tanh(x)

• Squanshes numbers to range [-1, 1]
• zero centered 😃
• still kill gradients when saturated 😦

ReLU

$$f(x)=max(0,x)$$

• Does not saturate 😃
• very computationally efficient 😃
• Converges much faster than sigmoid/tanh in practice 😃
• Actually more biologically plausible than sigmoid 😃

problems:

• Not zero-centered output

Leaky ReLU

$$f(x)=max(0.01x,x)$$

Exponential Linear Units(ELU)

$$f(x)=\{\begin{array}{l}xifx>0\\ \alpha (exp(x)-1)ifx\le 0\end{array}$$

Maxout Neuron

$$max(w_1^{T}x+b_1,w_2^{T}x+b_2)$$

• double parameters 😦

Data Preprocessing

Preprocess the data

• zero-centered data
• normalized data
• PCA
• Whitening

Weight Normalization

• First idea: Small random numbers
  (gaussian with zero mean and 1e-2 standard deviation)
  $$W=0.01\ast np.random.randn(D,H)$$
  Work Okey for small networks, but problems with deeper networks
• Xavier initialization
  $$W=np.random.randn(fan\_in,fan\_out)/np.sqrt(fan\_in)$$