机器学习---基础算法二--回归实践

机器学习—基础算法二

https://blog.csdn.net/fan2312/article/details/100854485

回归

• 均方误差：$MSE=\frac{1}{m}{\sum }_{i=1}^m(y_i-\widehat{y_i}{)}^2$

• 均方根误差: 标准误差：$RMSE=\sqrt{MSE}$

• 总平方和：$TSS={\sum }_{i=1}^m(y_i-\overline{y}{)}^2$

• 伪方差：$Var(Y)=TSS/m$

• 残差平方和：$RSS={\sum }_{i=1}^m(\widehat{y_i}-y_i{)}^2$

  • 即误差平方和 SSE

• $R^2=\frac{TSS-RSS}{TSS}=1-\frac{RSS}{TSS}$

  • $R^2$越大，拟合效果越好
  • $R^2$的最优值为1；若模型预测为随机值，$R^2$有可能为负
  • 若预测值恒为样本期望， $R^2$为0

• 回归平方和：$ESS={\sum }_{i=1}^m(\widehat{y_i}-\overline{y}{)}^2$

  • $TSS\ge ESS+RSS$
  • 只有无偏估计时，$TSS=ESS+RSS$

• 局部加权线性回归（LWR）

  • 目标函数
  • 权值的设置：
    • 高斯核函数
      • $w^{(i)}=\exp (-\frac{(x^{(i)}-x{)}^2}{2{\tau }^2})$
      • $\tau$称为带宽，控制训练样本随着与$x^{(i)}$距离的衰减速率
    • 多项式核函数：
      • $\kappa (x_1,x_2)=(<x_1,x_2>+R{)}^d$

• Logistic回归（sigmoid）

  • $h_{\theta }(x)=g({\theta }^{T}x)=\frac{1}{1+e^{-{\theta }^{T}x}}$
  • $g^{\prime }(x)=(\frac{1}{1+e^{-x}}{)}^{\prime }=g(x)\cdot (1-g(x))$
  • 参数估计
    • 假定
      • $P(y=1|x;\theta )=h_{\theta }(x)$
      • $P(y=0|x;\theta )=1-h_{\theta }(x)$
    • $p(y|x;\theta )=(h_{\theta }(x){)}^y(1-h_{\theta }(x){)}^{1-y}$
    • 似然函数：$L(\theta )=p(\overrightarrow{y}|X;\theta )$
                  $={\prod }_{i=1}^{m}p(y^{(i)}|x^{(i)};\theta )$
                  $={\prod }_{i=1}^m(h_{\theta }(x^{(i)}){)}^{y^{(i)}}(1-h_{\theta }(x^{(i)}){)}^{1-y^{(i)}}$
    • $l(\theta )=logL(\theta )={\sum }_{i=1}^m{y}^{(i)}logh(x^{(i)})+(1-y^{(i)})log(1-h(x{}^{(i)}))$
    • $\frac{\partial l(\theta )}{\partial {\theta }_j}={\sum }_{i=1}^m(y^{(i)}-g({\theta }^T{x}^{(i)}))\cdot x_j^{(i)}$
    • 参数更新
      • ${\theta }_j:={\theta }_j+\alpha (y^{(i)}-h_{\theta }(x^{(i)}))x_j^{(i)}$
  • 用于分类
  • 损失
    • $y_i=\left\{\begin{array}{c}-1,1\end{array}\right\}$
    • ${\hat{y}}_i=\{\begin{array}{ll}{p}_i& y_i=1\\ 1-p_i& y_i=-1\end{array}$
    • 似然函数 $L(\theta )={\prod }_{i=1}^m{p}_i^{(y_i+1)/2}(1-p_i{)}^{-(y_i-1)/2}$
    • $lnL(\theta )\Rightarrow l(\theta )={\sum }_{(i=1)}^{m}ln[p_i^{(y_i+1)/2}(1-p_i{)}^{-(y_i-1)/2}]$
      $\underrightarrow{p_i=\frac{1}{1+e^{-f_i}}}l(\theta )={\sum }_{i=1}^{m}ln[(\frac{1}{1+e^{-f_i}}{)}^{(y_i+1)/2}(\frac{1}{1+e^{f_i}}{)}^{(y_i-1)/2})]$
    • $loss(y_i,{\hat{y}}_i)=-l(\theta )$
      = ${\sum }_{i=1}^m[\frac{1}{2}(y_i+1)ln(1+e^{-f_i})-\frac{1}{2}(y_i-1)ln(1+e^{f_i})]$
      =$\{\begin{array}{ll}{\sum }_{i=1}^m[ln(1+e^{-f_i})]& y_i=1\\ {\sum }_{i=1}^m[ln(1+e^{f_i})]& y_i=-1\end{array}\Rightarrow loss(y_i,\widehat{y_i})={\sum }_{i=1}^m[ln(1+e^{-y_i\cdot f_i})]$

• 对数线性模型

  • 一个事件的机率odds，是指该事件发生的概率与该事件不发生概率的比值
  • 对数几率：logit函数
    • $P(y=1|x;\theta )=h_{\theta }(x)$
    • $P(y=0|x;\theta )=1-h_{\theta }(x)$
    • $logit(p)=log\frac{p}{1-p}=log\frac{h_{\theta }(x)}{1-h_{\theta }(x)}={\theta }^{T}x$

• Softmax回归

  • K分类
    • 第k类的参数为${\vec{\theta }}_k$，组成二维矩阵${\theta }_{k\times n}$
  • 概率
    • $p(c=k|x;\theta )=\frac{exp({\theta }_k^{T}x)}{{\sum }_{l=1}^{K}exp({\theta }_l^{T}x)},k=1,2,\dots ,K$
  • 似然函数
    • $L(\theta )={\prod }_{i=1}^m{\prod }_{k=1}^{K}p(c=k|x^{(i)};\theta {)}^{y_k^{(i)}}={\prod }_{i=1}^m{\prod }_{k=1}^K(\frac{exp({\theta }_k^{T}x)}{{\sum }_{l=1}^{K}exp({\theta }_l^{T}x)}{)}^{y_k^{(i)}}$
  • 对数似然
    • $J(\theta )={\sum }_{k=1}^K{y}_k\cdot ({\theta }_k^{T}x-ln{\sum }_{l=1}^{K}exp({\theta }_l^T{x}^{(i)}))$
  • 随机梯度
    • $\frac{\partial J(\theta )}{\partial {\theta }_k}=(y_k-p(y_k|x;\theta ))\cdot x$