Neural Network
1. Why Need NN?
-Mimic Brain
之前的人工经验( 符号主义)已经不太行了所以采用仿生主义(连接主义)的方法
-Non Linear
Not possible by just combining features
2. Logistic Unit
Activation Function (非线性)
定义:
x
0
=
1
,
θ
0
=
1
x_0=1, \theta_0=1
x0=1,θ0=1,作为偏置,
g
(
⋅
)
g(\cdot)
g(⋅)为非线性激活函数(必须是非线性的,否则一顿加乘之后还是线性回归的结果,就白搞了):
h
θ
(
x
)
=
g
(
θ
1
x
1
+
θ
2
x
2
+
θ
3
x
3
+
θ
0
x
0
)
=
θ
T
x
h_\theta(x)=g(\theta_1x_1+\theta_2x_2+\theta_3x_3+\theta_0x_0)=\boldsymbol\theta^T\boldsymbol{x}
hθ(x)=g(θ1x1+θ2x2+θ3x3+θ0x0)=θTx
3. Neural Network
Input Layer -> Hidden Layer -> Output Layer
(图中偏置都没有画出)上标表示神经元所在层数。每一个神经元的激活函数都可以选择不一样的。(但是只是可以,实际上还是用的一样的,没什么必要)
数学表达:
规定:
则有:
权重
W
\boldsymbol{W}
W是一个4行3列的矩阵,4是input的dim,3是神经元的dim。
example: AND OR NOT ;
XNOR异同,相同是1不同是0,单层感知机没办法实现。多层可以,因为
A
⨀
B
=
A
B
+
A
′
B
′
A\bigodot B =AB+A'B'
A⨀B=AB+A′B′ 就可以用3个逻辑单元去做。
用于Classification:
最后加一层softmax归一化概率。
4. Back Propagation
用来解决NN中的梯度下降问题。
多元函数的链式法则(来源于百度):
4.1 Softmax相关
nn.Softmax(dim: Union[int, NoneType] = None) -> None
Docstring:
Applies the Softmax function to an n-dimensional input Tensor
rescaling them so that the elements of the n-dimensional output Tensor
lie in the range [0,1] and sum to 1.
Softmax is defined as:
Softmax ( x i ) = e x i ∑ j e x j \text{Softmax}(x_{i}) = \frac{e^{x_i}}{\sum_j e^{x_j}} Softmax(xi)=∑jexjexi
When the input Tensor is a sparse tensor then the unspecifed
values are treated as -inf
.
Shape:
- Input: (*)
where *
means, any number of additional
dimensions
- Output: (*)
, same shape as the input
Returns:
a Tensor of the same dimension and shape as the input with
values in the range [0, 1]
Arguments:
dim (int): A dimension along which Softmax will be computed (so every slice
along dim will sum to 1).
… note::
This module doesn’t work directly with NLLLoss,
which expects the Log to be computed between the Softmax and itself.
Use LogSoftmax
instead (it’s faster and has better numerical properties).
Softmax+CrossEntropy求导:
只写个结论吧,假设某一层输出
z
1
,
z
2
,
z
3
z_1,z_2,z_3
z1,z2,z3经过softmax得到概率
P
1
,
P
2
,
P
3
P_1,P_2,P_3
P1,P2,P3, 损失函数使用CrossEntropy,
L
o
s
s
=
−
∑
i
=
1
m
y
i
l
o
g
(
P
i
)
Loss = - \sum_{i=1}^m{y_ilog(P_i)}
Loss=−i=1∑myilog(Pi)
记为Loss,softmax结果中是第k项为1别的都是0,
即
P
k
=
1
,
P
i
=
0
(
i
≠
k
)
P_k=1,P_i=0(i\neq k)
Pk=1,Pi=0(i=k)那么结论如下:
当
i
=
k
i=k
i=k的时候,
∂
L
o
s
s
∂
z
i
=
P
i
−
1
\frac{\partial{Loss}}{\partial{z_i}}=P_i-1
∂zi∂Loss=Pi−1,
当
i
≠
k
i\neq k
i=k的时候,
∂
L
o
s
s
∂
z
i
=
P
i
\frac{\partial{Loss}}{\partial{z_i}}=P_i
∂zi∂Loss=Pi
推导过程:先对CE求导,再对softmax分类讨论求导后裂项表示,再根据chain rule相乘得到结果。