Cost Function的原理及实现(Python, matlab)

成本函数(Cost Function)

$$J(\theta_0,\theta_1) = \frac{1}{2m}\sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2$$
$m$ : Number of training examples.
$x$ : input.
$y$ : output.
Parameters: $\theta_0$, $\theta_1$.

以下为MATLAB实现方式：

function J = computeCost(X, y, theta)
    %COMPUTECOST Compute cost for linear regression
    %   J = COMPUTECOST(X, y, theta) computes the cost of using theta as the parameter for linear regression to fit the data points in X and y
    m = length(y); % number of training examples
    J = 0;
    J = sum((X * theta - y) .^2) / 2 / m;
end

以下为Python实现方式：

def computeCost(X, y, theta):
    """Cost Functions

    Parameters
    ----------
    X : np.ndarray, like (49 * 2)
    y : np.ndarray, like (49 * 1)
    theta : np.ndarray, like (2 * 1)

    Returns
    -------
    J : float, cost

    """
    y = np.transpose(y)
    J = sum((np.dot(X, theta) - y.reshape(len(y),1)) **2) / 2.0 / len(y)
    # np.dot(A, B) 矩阵乘积，A * B 矩阵点乘
    # pow(A, 2) 多次乘积，**n 点次方
    return J

注意：np中点乘和矩阵乘积，点次方与多次乘积的区别，与MATLAB不同。

梯度下降法(Gradient Descent)

$$repeat\quad until\quad convergence \{ \quad \theta_j:=\theta_j-\alpha \frac{\partial J(\theta_0, \theta_1)}{\partial \theta_j} \quad (j=0,1) \quad \}$$
$\alpha$ : learning rate.
$\partial$ : 偏微分

以下为MATLAB实现：

function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)
    %GRADIENTDESCENT Performs gradient descent to learn theta
    %   theta = GRADIENTDESCENT(X, y, theta, alpha, num_iters) updates theta by 
    %   taking num_iters gradient steps with learning rate alpha

    m = length(y); % number of training examples
    J_history = zeros(num_iters, 1);

    for iter = 1:num_iters
        theta = theta -  X' * (X * theta - y) * (alpha / m);  
        J_history(iter) = computeCost(X, y, theta);
    end
end

以下为Python实现：

def gradientDescent(X, y, theta, alpha, num_iters):
    """Gradient Descent for (Multivariate) Linear Regression

    Parameters
    ----------
    X : np.ndarray, like (49 * 2)
    y : np.ndarray, like (49 * 1)
    theta : np.ndarray, like (49 * 1)
    alpha : learning rate
    num_iters : number of iter

    Returns
    -------
    tuple(J_history, theta)
    J_history : np.ndarray, like (num_iters, 1)
    theta : theta of convergence, like (2 * 1)
    """

    J_history = np.zeros((num_iters, 1))
    for n_iter in range(num_iters):
        theta = theta - np.dot(X.T, np.dot(X, theta) - y.reshape(len(y),1)) *alpha / len(y)
        J_history[n_iter, 0] = computeCost(X, y, theta)
    return J_history, theta

线性回归中的梯度下降公式

$$\theta_0:= \theta_0-\alpha\frac{1}{m} \sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})$$
$$\theta_1:= \theta_1-\alpha\frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})x_1^{(i)}$$
同理可得：
$$\theta_j:= \theta_j-\alpha\frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})x_j^{(i)}$$
在（多元）线性回归中，由于在代码中直接使用矩阵进行运算，因此代码同上。