逻辑回归 (logistic regression)
1. 逻辑分布(logistic distribution)
定义 设
X
X
X是连续随机变量,
X
X
X服从logistic分布是指,
X
X
X具有下列分布函数和密度函数:
F
(
x
)
=
P
(
X
⩽
x
)
=
1
1
+
e
−
(
x
−
μ
)
/
γ
F(x)=P(X \leqslant x)=\frac{1}{1+\mathrm{e}^{-(x-\mu) / \gamma}}
F(x)=P(X⩽x)=1+e−(x−μ)/γ1
f
(
x
)
=
F
′
(
x
)
=
e
−
(
x
−
μ
)
/
γ
γ
(
1
+
e
−
(
x
−
μ
)
/
γ
)
2
f(x)=F^{\prime}(x)=\frac{\mathrm{e}^{-(x-\mu) / \gamma}}{\gamma\left(1+\mathrm{e}^{-(x-\mu) / \gamma}\right)^{2}}
f(x)=F′(x)=γ(1+e−(x−μ)/γ)2e−(x−μ)/γ
![图片名称](https://i-blog.csdnimg.cn/blog_migrate/798ec1e56dcb610ac70b1a7e0bb6440e.png)
式中, μ \mu μ为位置参数, γ > 0 \gamma>0 γ>0为形状参数。分布函数 F ( x ) F(x) F(x)其图形是一条 S S S形曲线,以点 ( μ , 1 2 ) \left(\mu, \frac{1}{2}\right) (μ,21)为中心对称,即满足 F ( − x + μ ) − 1 2 = − F ( x + μ ) + 1 2 F(-x+\mu)-\frac{1}{2}=-F(x+\mu)+\frac{1}{2} F(−x+μ)−21=−F(x+μ)+21
2. 二项logistic回归模型
定义: 二项logistic回归模型是如下条件概率分布:
P
(
Y
=
1
∣
x
)
=
exp
(
w
⋅
x
+
b
)
1
+
exp
(
w
⋅
x
+
b
)
P
(
Y
=
0
∣
x
)
=
1
1
+
exp
(
w
⋅
x
+
b
)
\begin{array}{l}{P(Y=1 | x)=\frac{\exp (w \cdot x+b)}{1+\exp (w \cdot x+b)}} \\ {P(Y=0 | x)=\frac{1}{1+\exp (w \cdot x+b)}}\end{array}
P(Y=1∣x)=1+exp(w⋅x+b)exp(w⋅x+b)P(Y=0∣x)=1+exp(w⋅x+b)1
x
∈
R
n
x \in \mathbf{R}^{n}
x∈Rn是输入,
Y
∈
{
0
,
1
}
Y \in\{0,1\}
Y∈{0,1}是输出,
w
∈
R
n
w \in \mathbf{R}^{n}
w∈Rn和
b
∈
R
b \in \mathbf{R}
b∈R是参数,
w
w
w称为权值向量,
b
b
b称为偏置,
w
⋅
x
w \cdot x
w⋅x为
w
w
w和
x
x
x的内积。
记
w
=
(
w
(
1
)
w
(
2
)
,
⋯
,
w
(
n
)
,
b
)
T
,
x
=
(
x
(
1
)
,
x
(
2
)
,
⋯
,
x
(
n
)
,
1
)
T
w=\left(w^{(1)} w^{(2)}, \cdots, w^{(n)}, b\right)^{\mathrm{T}}, x=\left(x^{(1)}, x^{(2)}, \cdots, x^{(n)}, 1\right)^{\mathrm{T}}
w=(w(1)w(2),⋯,w(n),b)T,x=(x(1),x(2),⋯,x(n),1)T,此时
P
(
Y
=
1
∣
x
)
=
exp
(
w
⋅
x
)
1
+
exp
(
w
⋅
x
)
P
(
Y
=
0
∣
x
)
=
1
1
+
exp
(
w
⋅
x
)
\begin{aligned} P(Y=1 | x) &=\frac{\exp (w \cdot x)}{1+\exp (w \cdot x)} \\ P(Y=0 | x) &=\frac{1}{1+\exp (w \cdot x)} \end{aligned}
P(Y=1∣x)P(Y=0∣x)=1+exp(w⋅x)exp(w⋅x)=1+exp(w⋅x)1如果事件发生的概率是
p
p
p,那么该事件的几率是
p
1
−
p
\frac{p}{1-p}
1−pp,对数几率是
logit
(
p
)
=
log
p
1
−
p
\operatorname{logit}(p)=\log \frac{p}{1-p}
logit(p)=log1−pp
于是有上式logistic回归
log
P
(
Y
=
1
∣
x
)
1
−
P
(
Y
=
1
∣
x
)
=
w
⋅
x
\log \frac{P(Y=1 | x)}{1-P(Y=1 | x)}=w \cdot x
log1−P(Y=1∣x)P(Y=1∣x)=w⋅x这表明,在logistic回归模型中,输出
Y
=
1
Y=1
Y=1的对数几率是输入
x
x
x的线性函数,即输出
Y
=
1
Y=1
Y=1的对数几率是由输入
x
x
x的线性函数表示的模型,即logistic回归模型。通过逻辑斯谛回归模型的定义可将线性函数
w
⋅
x
w \cdot x
w⋅x转换为概率:
P
(
Y
=
1
∣
x
)
=
exp
(
w
⋅
x
)
1
+
exp
(
w
⋅
x
)
P(Y=1 | x)=\frac{\exp (w \cdot x)}{1+\exp (w \cdot x)}
P(Y=1∣x)=1+exp(w⋅x)exp(w⋅x)
3. 模型参数估计
给定训练数据集
T
=
{
(
x
1
,
y
1
)
,
(
x
2
,
y
2
)
,
⋯
(
x
N
,
y
N
)
}
,
x
i
∈
R
n
,
y
i
∈
{
0
,
1
}
T=\left\{\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right), \cdots\left(x_{N}, y_{N}\right)\right\}, x_{i} \in \mathbf{R}^{n}, y_{i} \in\{0,1\}
T={(x1,y1),(x2,y2),⋯(xN,yN)},xi∈Rn,yi∈{0,1},可以应用极大似然估计法估计模型参数,从而得到logistic回归模型
设
P
(
Y
=
1
∣
x
)
=
π
(
x
)
,
P
(
Y
=
0
∣
x
)
=
1
−
π
(
x
)
P(Y=1 | x)=\pi(x), \quad P(Y=0 | x)=1-\pi(x)
P(Y=1∣x)=π(x),P(Y=0∣x)=1−π(x)
似然函数为
∏
i
=
1
N
[
π
(
x
i
)
]
y
i
[
1
−
π
(
x
i
)
]
1
−
y
i
\prod_{i=1}^{N}\left[\pi\left(x_{i}\right)\right]^{y_{i}}\left[1-\pi\left(x_{i}\right)\right]^{1-y_{i}}
∏i=1N[π(xi)]yi[1−π(xi)]1−yi
对数似然函数为
L
(
w
)
=
∑
i
=
1
N
[
y
i
log
π
(
x
i
)
+
(
1
−
y
i
)
log
(
1
−
π
(
x
i
)
)
]
=
∑
i
=
1
N
[
y
i
log
π
(
x
i
)
1
−
π
(
x
i
)
+
log
(
1
−
π
(
x
i
)
)
]
=
∑
i
=
1
N
[
y
i
(
w
⋅
x
i
)
−
log
(
1
+
exp
(
w
⋅
x
i
)
]
\begin{aligned} L(w) &=\sum_{i=1}^{N}\left[y_{i} \log \pi\left(x_{i}\right)+\left(1-y_{i}\right) \log \left(1-\pi\left(x_{i}\right)\right)\right] \\ &=\sum_{i=1}^{N}\left[y_{i} \log \frac{\pi\left(x_{i}\right)}{1-\pi\left(x_{i}\right)}+\log \left(1-\pi\left(x_{i}\right)\right)\right] \\ &=\sum_{i=1}^{N}\left[y_{i}\left(w \cdot x_{i}\right)-\log \left(1+\exp \left(w \cdot x_{i}\right)\right]\right.\end{aligned}
L(w)=i=1∑N[yilogπ(xi)+(1−yi)log(1−π(xi))]=i=1∑N[yilog1−π(xi)π(xi)+log(1−π(xi))]=i=1∑N[yi(w⋅xi)−log(1+exp(w⋅xi)] 对
L
(
w
)
L(w)
L(w)求极大值,得到
w
w
w的估计值。问题就变成了以对数似然函数为目标的最优化问题,这里通常采用的方法是梯度下降法及拟牛顿法。
假设
w
w
w的极大似然估计值是
w
^
\hat{\boldsymbol{w}}
w^,那么学习得到的logistic回归模型为
p
(
Y
=
1
∣
x
)
=
exp
(
w
^
⋅
x
)
1
+
exp
(
w
^
⋅
x
)
P
(
Y
=
0
∣
x
)
=
1
1
+
exp
(
w
^
⋅
x
)
\begin{aligned} p(Y=1 | x) &=\frac{\exp (\hat{w} \cdot x)}{1+\exp (\hat{w} \cdot x)} \\ P(Y=0 | x) &=\frac{1}{1+\exp (\hat{w} \cdot x)} \end{aligned}
p(Y=1∣x)P(Y=0∣x)=1+exp(w^⋅x)exp(w^⋅x)=1+exp(w^⋅x)14. 多项逻辑斯谛回归(multi-nominal logistic regression model)
假设随机变量
Y
Y
Y的取值集合是
{
1
,
2
,
⋯
,
K
}
\{1,2, \cdots, K\}
{1,2,⋯,K},那么多项逻辑斯谛回归模型是
P
(
Y
=
k
∣
x
)
=
exp
(
w
k
⋅
x
)
1
+
∑
k
=
1
K
−
1
exp
(
w
k
⋅
x
)
,
k
=
1
,
2
,
⋯
,
K
−
1
P(Y=k | x)=\frac{\exp \left(w_{k} \cdot x\right)}{1+\sum_{k=1}^{K-1} \exp \left(w_{k} \cdot x\right)}, \quad k=1,2, \cdots, K-1
P(Y=k∣x)=1+∑k=1K−1exp(wk⋅x)exp(wk⋅x),k=1,2,⋯,K−1
P
(
Y
=
K
∣
x
)
=
1
1
+
∑
k
=
1
K
−
1
exp
(
w
k
⋅
x
)
P(Y=K | x)=\frac{1}{1+\sum_{k=1}^{K-1} \exp \left(w_{k} \cdot x\right)}
P(Y=K∣x)=1+∑k=1K−1exp(wk⋅x)1这里,
x
∈
R
n
+
1
,
w
k
∈
R
n
+
1
x \in \mathbf{R}^{n+1}, w_{k} \in \mathbf{R}^{n+1}
x∈Rn+1,wk∈Rn+1