cs231n作业，assignment1-softmax详解

打酱油QAQ

已于 2024-06-26 15:18:57 修改

阅读量1.2k

点赞数

文章标签：机器学习人工智能

于 2019-04-20 22:23:51 首次发布

本文链接：https://blog.csdn.net/qq_24906797/article/details/89421926

版权

文章目录

交叉熵 Cross Entropy

可以衡量两个分布之间的距离，记 $x_i$ 为第i个样本，p(x)预测分布，q(x)真实分布。交叉熵损失为
$H(p,q)=-\sum_{i=1}^{n}q\left(x_i\right)log\left(p\left(x_i\right)\right)$
总的来说分布相差越大，损失越大。

二分类

sigmoid+二分类交叉熵损失
记x为特征，y可以取两个值，0或1
可以使用sigmoid函数 $\sigma(x)=\frac{1}{1+e^{-x}}$ 计算概率
$p(y=1|x)=p=\frac{1}{1+e^{-x}}$
$\frac{e^{-x}}{1+e^{-x}}$
$H(p,q)=-log\frac{1}{1+e^{-x}}$

多分类

softmax+多分类交叉熵损失
softmax + cross entropy loss
softmax是将分数归一化，归一化后的数值可以视为概率

前向传播公式：

x是样本
$y_i$ 是真实标签
$j$ 是所有的类别
$s$ 是x经过w得到的分数
这里为了与代码保持一致记作 $s = X W$ ，其实通常公式是 $s=W^Tx$
总损失：
$=-\frac{1}{N} \log \sum{\left( \frac { e ^ { s_ { y _ { i } } } } { \sum _ { j } e ^ { s _ { j } } } \right)}+0.5\times\lambda\|W\| ，其中s=XW$ 总损失可以分为经验风险和正则化项两部分

经验风险
$-\frac{1}{N}\log\sum{\left( \frac{e^{s_{y_i}}}{\sum_{j}e^{s_j}} \right)}$
正则化项
$0.5\times\lambda\|W\|$ 其中0.5是为了计算梯度方便乘的

为了求梯度，公式分解，记 $L _ { i }$ ：
$\frac{ e ^ { s_{y_i} }}{\sum_j{e^{s_j}}}$ （其中i是样本数量，j是类别）， $L _ { i }$ 是第i个样本的损失

将 $L _ { i }$ 带入损失函数中，最后：
$L=\frac{1}{N}\sum{L_i}+0.5\times\lambda||W||，其中L _ { i } = - log \frac{ e ^ { s_{y_i} }}{\sum_j{e^{s_j}}} ，s=XW$

tips:
- 如果s是很小的负数， $e^s$ 会接近0，分母接近于0意味着最后的结果是未定义的。（无穷小/无穷小结果是什么来着）
- 当s是非常大的正数时， $e^s$ 上溢，再次导致整个表达式未定义。
解决办法：减去 $max_{j}f_{s_j}$ ， $e^{s_j-max_{j}f_{s_j}}$ 。
- exp的指数最大为0，这排除了上溢的可能性。
- 同样地，分母中至少有一个值为1的项，这就排除了因分母下溢而导致被零除的可能性。（摘自花书）
$\frac { e ^ { s _ { y _ { i } } } } { \sum _ { j } e ^ { s _ { j } } } = \frac { C e ^ { s _ { y _ { i } } } } { C \sum _ { j } e ^ { s _ { j } } } = \frac { e ^ { s _ { y _ { i } } + \log C } } { \sum _ { j } e ^ { fs_ { j } + \log C } }（\log C = - \max _ { j } fs_ { j }）$

反向传播，计算梯度：

$\frac{dL}{ds_j}=\frac{dL}{dL_i}\times\frac{dL_i}{ds_j}\times\frac{ds_j}{x}=?$ $\frac{dL}{dL_i}=\frac{1}{N}$ $\frac{ e ^ { s_{y_i} }}{\sum{e^{s_j}}}=-s_{y_i}+log(\sum e^{s_j})$

注意这里的 $s_{y_i}$ 是只关于正确类别的值，所以这里要分两种情况
在这里插入图片描述
(这一块写的有问题，周末要改一下)

使用循环计算梯度

#encoding:utf-8
from builtins import range
import numpy as np
from random import shuffle
from past.builtins import xrange

def softmax_loss_naive(W, X, y, reg):
    """
    Softmax loss function, naive implementation (with loops)
    Inputs:
    - W: A numpy array of shape (D, C) containing weights.
    - X: A numpy array of shape (N, D) containing a minibatch of data.
    - y: A numpy array of shape (N,) containing training labels; y[i] = c means
      that X[i] has label c, where 0 <= c < C.
    - reg: (float) regularization strength

    Returns a tuple of:
    - loss as single float
    - gradient with respect to weights W; an array of same shape as W
    """
    # Initialize the loss and gradient to zero.
    loss = 0.0
    dW = np.zeros_like(W)

    #############################################################################
    # TODO: Compute the softmax loss and its gradient using explicit loops.     #
    # Store the loss in loss and the gradient in dW. If you are not careful     #
    # here, it is easy to run into numeric instability. Don't forget the        #
    # regularization!                                                           #
    #############################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    num_classes = W.shape[1]
    num_train = X.shape[0]
    #W(D, C)  X(N, D)
    for i in xrange(num_train):
        scores = X[i].dot(W) #(1,C)
        shift_scores = scores - max(scores)
        loss +=( - shift_scores[y[i]] + np.log(sum(np.exp(shift_scores))) )
        
        for j in xrange(num_classes):
            softmax_output = np.exp(shift_scores[j])/sum(np.exp(shift_scores)) #是一个矩阵[C,1]
            if j == y[i]:
                dW[:,j] += (-1 + softmax_output) *X[i] 
            else: 
                dW[:,j] += softmax_output *X[i] 

    loss /= num_train 
    loss +=  0.5* reg * np.sum(W * W)
    dW = dW/num_train + reg* W 

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return loss, dW

向量化（不使用循环）

def softmax_loss_vectorized(W, X, y, reg):
    """
    Softmax loss function, vectorized version.

    Inputs and outputs are the same as softmax_loss_naive.
    """
    # Initialize the loss and gradient to zero.
    loss = 0.0
    dW = np.zeros_like(W)

    #############################################################################
    # TODO: Compute the softmax loss and its gradient using no explicit loops.  #
    # Store the loss in loss and the gradient in dW. If you are not careful     #
    # here, it is easy to run into numeric instability. Don't forget the        #
    # regularization!                                                           #
    #############################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    num_classes = W.shape[1]
    num_train = X.shape[0]
    
    scores = X.dot(W)
    shift_scores = scores - np.max(scores, axis = 1, keepdims = True)
    probabilities = np.exp(shift_scores)/np.sum(np.exp(shift_scores), axis = 1, keepdims = True) #[n,c]
    loss = -np.sum(np.log(probabilities[range(num_train), list(y)]))
    loss = loss / num_train + 0.5* reg * np.sum(W * W)
  
    dscores = softmax_output.copy()
    dscores[range(num_train), list(y)] += -1
    dW = (X.T).dot(dscores)
    dW = dW/num_train + reg* W 

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return loss, dW

打酱油QAQ

关注

0
点赞
踩
8

收藏

觉得还不错? 一键收藏
3
评论
cs231n作业，assignment1-softmax详解

softmax总公式L=−1Nlog⁡∑(esyi∑jesj)+0.5×λ∣∣W∣∣L =-\frac{1}{N} \log \sum{\left( \frac { e ^ { s_ { y _ { i } } } } { \sum _ { j } e ^ { s _ { j } } } \right)}+0.5\times\lambda||W||L=−N1log∑(∑jesjesyi...
复制链接

扫一扫