chapter 6 deep feed forward networks

chapter 6 deep feed forward networks

chapter 6.1 example learning xor

$J(\Theta) = \frac{1}{4}\sum_{x\in \Bbb{x}}(f^*(x)-f(x;\Theta))^2$

now we must chose our model $f(x;\Theta)$,
linear model
$f(x;w, b)=x^{\mathsf{T}}w+b$
it can not describe the xor logic

add a hidden linear layer
$h=f^1(x;W, c)$, $y=f^2(h; w, b)$
$f(x; W, c, w, b)=f^2(f^1(x))$

$f^1(x)=W^\mathsf{T}x$ and $f^2(h)=h^\mathsf{T}w$
we get $f(x)=w^\mathsf{T}W^\mathsf{T}x$.

clearly, we must use a nonlinear layer to represent the features.

ReLU
rectified linear unit
$f(x; W, c, w, b)=w^\mathsf{T}max\{0, W^\mathsf{T}x+c\}+b$

$W=\begin{bmatrix} 1 & 1 \\1 & 1\end{bmatrix}$
$c=\begin{bmatrix} 0 \\ -1 \end{bmatrix}$
$w=\begin{bmatrix} 1 \\ -2 \end{bmatrix}$

calculate
$x=\begin{bmatrix} 0 & 0\\ 0 & 1 \\ 1 & 0 \\ 1 & 1 \end{bmatrix}$
$xW=\begin{bmatrix}0 & 0 \\ 1 & 1 \\ 1 & 1 \\ 2 & 2 \end{bmatrix}$
$xW+c = \begin{bmatrix} 0&-1 \\ 1 & 0 \\ 1 & 0 \\ 2 & 1 \end{bmatrix}$
ReLU $\begin{bmatrix} 0 & 0 \\ 1 & 0 \\ 1 & 0 \\ 2 & 1 \end{bmatrix}$
$w^\mathsf{T}h=\begin{bmatrix} 0 \\ 1 \\ 1 \\ 0 \end{bmatrix}$

get it

chapter 6.2 gradient based learning

chapter 6.3 hidden units

chapter 6.4 architecture design