机器学习--线性回归

机器学习–线性回归

如有错误或者疑问，欢迎交流，转载请注明出处

---

线性回归定义

$$h(x)=\sum_{i=1}^n \theta_i x_i = \theta^T x$$
上式中n是特征维度，目标是使损失函数 J最小，其中m是训练样本数目
$$J(\theta)=\sum_{i=1}^m \frac{1}{2m}(h(x^{(i)})-y^{(i)})^2$$

梯度下降

$$\begin{eqnarray*} \frac{\partial J(\theta)}{\partial\theta_j} &= &\frac{1}{m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})\centerdot \frac{\partial h(x^{(i)})}{\partial \theta_j}\\ &=&\frac{1}{m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})\centerdot x_j^{(i)}\end{eqnarray*}$$

• batch gradient descent
  对每个j:
  $\theta_j := \theta_j - \alpha \frac{1}{m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})\centerdot x_j^{(i)}$

def batch_gradient_descent(X, y, theta, alpha, num_iters):
    '''
    X: (m,n) ndarray, m是训练样本数目，n是特征维度
    y: (m,) ndarray
    '''
    m = y.size
    J_history = np.zeros(num_iters)
    for i in range(num_iters):
        predictions = np.dot(X,theta)
        updates = X.T.dot(predictions - y)
        theta = theta  - alpha * (1.0/m) * updates
        J_history[i] = compute_cost(X,y,theta)
    retrun theta, J_history

• stochastic gradient descent
  训练样本过多时，训练一轮的代价比较大，一个一个样本来，一批一批的来叫mini_batch~~
  for i=1:m{
    for j = 1:n{
      $\theta_j := \theta_j - \alpha (h(x^{(i)})-y^{(i)})\centerdot x_j^{(i)}$}
  }

def stochastic_gradient_descent(X, y, theta, alpha, num_iters):
    '''
    X: (m,n) ndarray, m是训练样本数目，n是特征维度
    y: (m,) ndarray
    '''
    m = y.size
    J_history = np.zeros(num_iters)

    for i in range(num_iters):
        predictions = np.dot(X,theta)
        for j in range(m):
            updates = X[j,:]*(predictions[j] - y[i])
            theta = theta - alpha*updates
        J_history[i] = compute_cost(X, y, theta)

    return theta, J_history

最小二乘法

将损失函数写成矩阵形式

$$\begin{eqnarray*} \frac{1}{2}(X\theta-y)^T(X\theta-y) &=& \frac{1}{2} \sum_{i=1}^m (h(x^{(i)})-y^{(i)})^2\\ &=& J(\theta) \end{eqnarray*}$$

对$\theta$求导得 $\nabla_\theta J(\theta)=X^TX\theta-X^Ty$，极值处导数为0得

$$\theta=(X^TX)^{-1}Xy$$
最小化 $J(\theta)$的 $\theta$由闭式解(closed form)一步求得

def normal_eqn(X,y):
    theta = np.zeros((X.shape[1], 1))
    X_temp = np.mat(X.T.dot(X))#转成mat矩阵，后面才能求逆
    X_pinv = np.array(X_temp.I)
    theta = (X_pinv.dot(X.T)).dot(y)
    return theta