# cs231n - assignment1 - softmax 梯度推导

2016年07月16日 15:19:07

## Softmax exercise

Complete and hand in this completed worksheet (including its outputs and any supporting code outside of the worksheet) with your assignment submission. For more details see the assignments page on the course website.

This exercise is analogous to the SVM exercise. You will:

- implement a fully-vectorized loss function for the Softmax classifier
- implement the fully-vectorized expression for its analytic gradient
- use a validation set to tune the learning rate and regularization strength
- optimize the loss function with SGD
- visualize the final learned weights

L=1NiLi+λR(W)1

Li=logpyi=logefyijefj=fyi+logjefj2

j=yi$j ！= y_i$ 时：

LiWj=efjjefjfjWj=efjjefjXTi3

j==yi$j == y_i$ 时：
LiWj=efjjefjfjWj=efjjefjXTiXTi4

pk=efkjefj,Li=logpyi5

Lifk=pk1(yi=k)6

Lif=p[0...1...](yi1)6a

fW=hidden_layer.T(H×1)7

LiW=fWLif(8)

fW=hiddenlayerT[N×H]7m

LW=fWLf[H×C](8m)

 # compute the gradient on scores
dscores = probs
dscores[range(num_examples),y] -= 1

# softmax.py
import numpy as np
from random import shuffle

def softmax_loss_naive(W, X, y, reg):
"""
Softmax loss function, naive implementation (with loops)

Inputs have dimension D, there are C classes, and we operate on minibatches
of N examples.

Inputs:
- W: A numpy array of shape (D, C) containing weights.
- X: A numpy array of shape (N, D) containing a minibatch of data.
- y: A numpy array of shape (N,) containing training labels; y[i] = c means
that X[i] has label c, where 0 <= c < C.
- reg: (float) regularization strength

Returns a tuple of:
- loss as single float
- gradient with respect to weights W; an array of same shape as W
"""
# Initialize the loss and gradient to zero.
loss = 0.0
dW = np.zeros_like(W)

#############################################################################
# TODO: Compute the softmax loss and its gradient using explicit loops.     #
# Store the loss in loss and the gradient in dW. If you are not careful     #
# here, it is easy to run into numeric instability. Don't forget the        #
# regularization!                                                           #
#############################################################################
num_train = X.shape[0]
num_classes = W.shape[1]
for i in xrange(num_train):
scores = X[i].dot(W)
scores -= np.max(scores) #prevents numerical instability
correct_class_score = scores[y[i]]

exp_sum = np.sum(np.exp(scores))
loss += np.log(exp_sum) - correct_class_score

dW[:, y[i]] -= X[i]
for j in xrange(num_classes):
dW[:,j] += (np.exp(scores[j]) / exp_sum) * X[i]

loss /= num_train
loss += 0.5 * reg * np.sum( W*W )
dW /= num_train
dW += reg * W

#############################################################################
#                          END OF YOUR CODE                                 #
#############################################################################

return loss, dW

def softmax_loss_vectorized(W, X, y, reg):
"""
Softmax loss function, vectorized version.

Inputs and outputs are the same as softmax_loss_naive.
"""
# Initialize the loss and gradient to zero.
loss = 0.0
dW = np.zeros_like(W)

#############################################################################
# TODO: Compute the softmax loss and its gradient using no explicit loops.  #
# Store the loss in loss and the gradient in dW. If you are not careful     #
# here, it is easy to run into numeric instability. Don't forget the        #
# regularization!                                                           #
#############################################################################
num_train = X.shape[0]
num_classes = W.shape[1]

scores = X.dot(W)
scores -= np.max(scores, axis = 1)[:, np.newaxis]
exp_scores = np.exp(scores)
sum_exp_scores = np.sum(exp_scores, axis = 1)
correct_class_score = scores[range(num_train), y]

loss = np.sum(np.log(sum_exp_scores)) - np.sum(correct_class_score)

exp_scores = exp_scores / sum_exp_scores[:,np.newaxis]

# maybe here can be rewroten into matrix operations
for i in xrange(num_train):
dW += exp_scores[i] * X[i][:,np.newaxis]
dW[:, y[i]] -= X[i]

loss /= num_train
loss += 0.5 * reg * np.sum( W*W )
dW /= num_train
dW += reg * W
#############################################################################
#                          END OF YOUR CODE                                 #
#############################################################################

return loss, dW

# softmax.ipynb
# Use the validation set to tune hyperparameters (regularization strength and
# learning rate). You should experiment with different ranges for the learning
# rates and regularization strengths; if you are careful you should be able to
# get a classification accuracy of over 0.35 on the validation set.
from cs231n.classifiers import Softmax
results = {}
best_val = -1
best_softmax = None
learning_rates = [5e-6, 1e-7, 5e-7]
regularization_strengths = [1e4, 5e4, 1e5]

################################################################################
# TODO:                                                                        #
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save    #
# the best trained softmax classifer in best_softmax.                          #
################################################################################
params = [(x,y) for x in learning_rates for y in regularization_strengths ]
for lrate, regular in params:
softmax = Softmax()
loss_hist = softmax.train(X_train, y_train, learning_rate=lrate, reg=regular,
num_iters=700, verbose=True)
y_train_pred = softmax.predict(X_train)
accuracy_train = np.mean( y_train == y_train_pred)
y_val_pred = softmax.predict(X_val)
accuracy_val = np.mean(y_val == y_val_pred)
results[(lrate, regular)] = (accuracy_train, accuracy_val)
if(best_val < accuracy_val):
best_val = accuracy_val
best_softmax = softmax
################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Print out results.
for lr, reg in sorted(results):
train_accuracy, val_accuracy = results[(lr, reg)]
print 'lr %e reg %e train accuracy: %f val accuracy: %f' % (
lr, reg, train_accuracy, val_accuracy)

print 'best validation accuracy achieved during cross-validation: %f' % best_val
• 本文已收录于以下专栏：

## cs231n assignment1--Softmax

svm实现完了，这部分会相对比较轻松，大部分和svm类似。 关于梯度的推导，我主要参考这篇文章 http://www.jianshu.com/p/004c99623104multiclass 梯...
• zt_1995
• 2017年03月15日 16:00
• 958

## CS231n作业笔记1.5：Softmax的误差以及梯度计算

CS231n简介详见 CS231n课程笔记1：Introduction。 本文都是作者自己的思考，正确性未经过验证，欢迎指教。作业笔记Softmax的误差函数以及正则项请参考 CS231n课程笔记...
• silent56_th
• 2016年12月20日 17:35
• 966

## cs231n：assignment1——Q3: Implement a Softmax classifier

Jupyter notebook softmaxipynb 内容 Softmax exercise Softmax Classifier Inline Question 1 softmaxpy 内容...
• xieyi4650
• 2016年11月25日 12:14
• 1789

## cs231n 卷积神经网络与计算机视觉 2 SVM softmax

linear classification上节中简单介绍了图像分类的概念，并且学习了费时费内存但是精度不高的knn法，本节我们将会进一步学习一种更好的方法，以后的章节中会慢慢引入神经网络和convol...
• bea_tree
• 2016年05月25日 01:56
• 4525

## [CS231n@Stanford] Assignment1-Q3 (python) Softmax实现

softmax.py import numpy as np from random import shuffle def softmax_loss_naive(W, X, y, reg): "...
• zzhangjizhi
• 2016年09月07日 13:09
• 1824

## cs231n的第一次作业Softmax

Softmax分类器损失函数softmax的损失函数为 这里log的底数为e，即等价于 这里将最后得到的score归一化了。这位大神说的好 SVM只选自己喜欢的男神，Softmax把...
• icameling
• 2017年02月16日 16:13
• 412

## CS231n——Assignmen1之Softmax

Softmax 分类器的实现 1.类的实现 import numpy as np from random import shuffle def softmax_loss_naive(W, X,...
• MargretWG
• 2017年04月05日 13:57
• 353

## cs231n作业1--softmax

1、思想： 与SVM不同，SVM直接利用线性映射的结果进行计算损失值，而softmax需要对线性映射得到的值进行指数归一化,然后在进行损失值计算。在SVM损失函数中使用的是折叶函数，而在sof...
• swj110119
• 2017年04月26日 16:21
• 384

## cs231n-assignment1-SVM/Softmax/two-layer-nets梯度求解

• pjia_1008
• 2017年03月27日 15:19
• 1166

## cs231n课程作业assignment1（Softmax）

• han784851198
• 2016年11月26日 11:44
• 647

举报原因： 您举报文章：cs231n - assignment1 - softmax 梯度推导 色情 政治 抄袭 广告 招聘 骂人 其他 (最多只允许输入30个字)