logistic回归公式推导

假设函数

$h(\theta)={1\over{1+e^{-\theta^TX}}}$

为什么使用sigmod

这个网上有很多文章，但是还是不太看懂。大概就是0-1之间增函数，还有是指数分布簇。

代价函数

$J(\theta)=-{1\over m}\sum\limits_{i=1}^m[y^i\log(h_{\theta}(x^i))+(1-y^i)\log(1-h_{\theta}(x^i))]$

代价函数推导

伯努利分布

$$P(X=n)=\left\{ \begin{aligned} &1-p,n=0 \\ &p,n=1 \end{aligned} \right.$$

求p的最大似然估计量

P{X=x}=$p^x(1-p)^{1-p} =0,1$

设$x_1,x_2,…,x_n$是给定的样本值

对应的似然函数

$L(p)=\prod\limits_{i=1}^n p^{x_i}(1-p)^{1-x_i} (0<p<1)$ 求L(p)的最大值点

取对数

$\ln{L(p)}=\sum\limits_{i=1}^n \ln{[p^{x_i}(1-p)^{1-x_i}]}$

​ $=\sum\limits_{i=1}^n [x_i\ln{p} + (1-x_i)\ln{(1-p)}]$

替换成logistic回归

$$Cost(h(\theta),y)=\left\{ \begin{aligned} &-log(h_{\theta}(x)),y=1 \\ &-log(1-h_{\theta}(x)),y=0 \end{aligned} \right.$$ (价函数
$J(\theta)=-{1\over m}[\sum\limits_{I=1}^my^ilogh_{\theta}(x^i)+(1-y^i)log(1-h_{\theta}(x^i))]$

代价函数求导

$J(\theta)=-{1\over m}[\sum\limits_{I=1}^my^ilogh_{\theta}(x^i)+(1-y^i)log(1-h_{\theta}(x^i))]$
$\frac{\partial }{\partial \theta_j}=-{1\over m}\sum\limits_{I=1}^m\frac{\partial }{\partial \theta_j}[y^ilogh_{\theta}(x^i)+(1-y^i)log(1-h_{\theta}(x^i))]$
$=-{1\over m}\sum\limits_{I=1}^m [y^ilogh_{\theta}(x^i)]'+[(1-y^i)log(1-h_{\theta}(x^i))]'$ ……….($(u+v)'=u'+v'$)
$=-{1\over m}\sum\limits_{I=1}^m [y^ilogh_{\theta}(x^i)]'+[(1-y^i)log(1-h_{\theta}(x^i))]'$ ……….($(uv)'=u'v-uv'$)
$=-{1\over m}\sum\limits_{I=1}^m [(y^i)'logh_{\theta}(x^i)+y^i(logh_{\theta}(x^i))']+[(1-y^i)'log(1-h_{\theta}(x^i))+(1-y^i)log(1-h_{\theta}(x^i))']$
$=-{1\over m}\sum\limits_{I=1}^m [(y^i)'logh_{\theta}(x^i)+y^i(logh_{\theta}(x^i))']+[(1-y^i)'log(1-h_{\theta}(x^i))+(1-y^i)log(1-h_{\theta}(x^i))']$……….($h(\theta)={1\over{1+e^{-\theta^TX}}}$带入)
$=-{1\over m}\sum\limits_{I=1}^m [y^i(log({1\over{1+e^{-\theta^Tx^{i}}}})']+[(1-y^i)log({{1+e^{-\theta^Tx^{i}}-1}\over{1+e^{-\theta^Tx^{i}}}})']$ ……….($(Cu)'=Cu',(log(u))'={1\over u}u'$)
$=-{1\over m}\sum\limits_{I=1}^m [y^i({1+e^{-\theta^Tx^{i}}})({1\over{1+e^{-\theta^Tx^{i}}}})']+[(1-y^i)({{1+e^{-\theta^Tx^{i}}}\over{e^{-\theta^Tx^{i}}}})({{e^{-\theta^Tx^{i}}}\over{1+e^{-\theta^Tx^{i}}}})']$ ……….($({u \over v})'={{u'v-uv'}\over{v^2}},(e^{-Cx})'=-Ce^{-Cx}$)
$=-{1\over m}\sum\limits_{I=1}^m [y^i({1+e^{-\theta^Tx^{i}}}) ({{0-(1+e^{-\theta^Tx^{i}})'}\over{(1+e^{-\theta^Tx^{i}})^2}}) ]-[(1-y^i)({{1+e^{-\theta^Tx^{i}}}\over{e^{-\theta^Tx^{i}}}})({{(e^{-\theta^Tx^{i}})'}\over{(1+e^{-\theta^Tx^{i}})^2}})]$
$=-{1\over m}\sum\limits_{I=1}^m [y^i({{-(1+e^{-\theta^Tx^{i}})'}\over{(1+e^{-\theta^Tx^{i}})}}) ]-[(1-y^i)({{x^i}\over{(1+e^{-\theta^Tx^{i}})}})]$……….($(e^{-Cx})'=-Ce^{-Cx}$)
$=-{1\over m}\sum\limits_{I=1}^m {{y^ixe^{-\theta^Tx^{i}}-x+xy^i}\over{1+e^{-\theta^Tx^{i}}}}$
$=-{1\over m}\sum\limits_{I=1}^m {{y^i(1+e^{-\theta^Tx^{i}}-1)}\over{1+e^{-\theta^Tx^{i}}}}x_j$
$=-{1\over m}\sum\limits_{I=1}^m {{y^i(1+e^{-\theta^Tx^{i}}-1)}\over{1+e^{-\theta^Tx^{i}}}}x_j$
$=-{1\over m}\sum\limits_{I=1}^m y^i-{{1}\over{1+e^{-\theta^Tx^{i}}}}x^i$
$=-{1\over m}\sum\limits_{I=1}^m[y^i-h_{\theta}(x^i)]x_j$