cs231n - assignment1 - softmax 梯度推导

Softmax exercise

Complete and hand in this completed worksheet (including its outputs and any supporting code outside of the worksheet) with your assignment submission. For more details see the assignments page on the course website.

This exercise is analogous to the SVM exercise. You will:

- implement a fully-vectorized loss function for the Softmax classifier
- implement the fully-vectorized expression for its analytic gradient
- check your implementation with numerical gradient
- use a validation set to tune the learning rate and regularization strength
- optimize the loss function with SGD
- visualize the final learned weights

和linear_svm一样，主要难点是求导操作，不过softmax的求导更简单一些。
首先还是给出 Loss 的公式：

对于每一个样本 Xi ， 由于 softmax 的分母对所有的 fj 进行了累积求和， 所以 Li 对 W 的导数对 W的每一列都又贡献， 即 ∂LiWj 对所有的 j 都不为 0:
当 j！=yi 时：

对所有样本都求出对应的Loss, 累积求和，并加上正则项即可以得到最终要求的Loss了。

上面求导数过程是把 Loss 对于 W 的导数显示的写出来，然后直接对 W 求导数，在这个简单的例子中可以这样，但是一旦网络变得复杂了，就很难直接写出Loss 对于要求的表达式的导数了。一种比较好的方式是利用 chain rule 逐级的求导数：

现在考虑第二层 fully connected layer，也就是紧连着 softmax 的那一层全连接层，这一层的输入是隐藏层的输出 hiddenlayer[1×H] ， 所以softmax的输入 f=hidden_layer.dot(W2)+b2 ， 检查一下维度， f 为 C 维向量， W2 为 H×C 的二维矩阵， b2 为 C 维向量，没问题。 现在就可以来求 f 对 W2 的导数了：

综合以上结果就可以求得：

最后对所有 N 个样本写成矩阵形式：

其中(6m)中的 MskMat为 N <script type="math/tex" id="MathJax-Element-351">N</script> 个(6a)中向量组成，具体形式可以参见如下python代码：

 # compute the gradient on scores
dscores = probs
dscores[range(num_examples),y] -= 1

# softmax.py
import numpy as np
from random import shuffle
def softmax_loss_naive(W, X, y, reg):
"""
Softmax loss function, naive implementation (with loops)
Inputs have dimension D, there are C classes, and we operate on minibatches
of N examples.
Inputs:
- W: A numpy array of shape (D, C) containing weights.
- X: A numpy array of shape (N, D) containing a minibatch of data.
- y: A numpy array of shape (N,) containing training labels; y[i] = c means
that X[i] has label c, where 0 <= c < C.
- reg: (float) regularization strength
Returns a tuple of:
- loss as single float
- gradient with respect to weights W; an array of same shape as W
"""
# Initialize the loss and gradient to zero.
loss = 0.0
dW = np.zeros_like(W)
#############################################################################
# TODO: Compute the softmax loss and its gradient using explicit loops.
#
# Store the loss in loss and the gradient in dW. If you are not careful
#
# here, it is easy to run into numeric instability. Don't forget the
#
# regularization!
#
#############################################################################
num_train = X.shape[0]
num_classes = W.shape[1]
for i in xrange(num_train):
scores = X[i].dot(W)
scores -= np.max(scores) #prevents numerical instability
correct_class_score = scores[y[i]]
exp_sum = np.sum(np.exp(scores))
loss += np.log(exp_sum) - correct_class_score
dW[:, y[i]] -= X[i]
for j in xrange(num_classes):
dW[:,j] += (np.exp(scores[j]) / exp_sum) * X[i]
loss /= num_train
loss += 0.5 * reg * np.sum( W*W )
dW /= num_train
dW += reg * W
#############################################################################
#
END OF YOUR CODE
#
#############################################################################
return loss, dW
def softmax_loss_vectorized(W, X, y, reg):
"""
Softmax loss function, vectorized version.
Inputs and outputs are the same as softmax_loss_naive.
"""
# Initialize the loss and gradient to zero.
loss = 0.0
dW = np.zeros_like(W)
#############################################################################
# TODO: Compute the softmax loss and its gradient using no explicit loops.
#
# Store the loss in loss and the gradient in dW. If you are not careful
#
# here, it is easy to run into numeric instability. Don't forget the
#
# regularization!
#
#############################################################################
num_train = X.shape[0]
num_classes = W.shape[1]
scores = X.dot(W)
scores -= np.max(scores, axis = 1)[:, np.newaxis]
exp_scores = np.exp(scores)
sum_exp_scores = np.sum(exp_scores, axis = 1)
correct_class_score = scores[range(num_train), y]
loss = np.sum(np.log(sum_exp_scores)) - np.sum(correct_class_score)
exp_scores = exp_scores / sum_exp_scores[:,np.newaxis]
# maybe here can be rewroten into matrix operations 
for i in xrange(num_train):
dW += exp_scores[i] * X[i][:,np.newaxis]
dW[:, y[i]] -= X[i]
loss /= num_train
loss += 0.5 * reg * np.sum( W*W )
dW /= num_train
dW += reg * W
#############################################################################
#
END OF YOUR CODE
#
#############################################################################
return loss, dW

# softmax.ipynb
# Use the validation set to tune hyperparameters (regularization strength and
# learning rate). You should experiment with different ranges for the learning
# rates and regularization strengths; if you are careful you should be able to
# get a classification accuracy of over 0.35 on the validation set.
from cs231n.classifiers import Softmax
results = {}
best_val = -1
best_softmax = None
learning_rates = [5e-6, 1e-7, 5e-7]
regularization_strengths = [1e4, 5e4, 1e5]
################################################################################
# TODO:
#
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save
#
# the best trained softmax classifer in best_softmax.
#
################################################################################
params = [(x,y) for x in learning_rates for y in regularization_strengths ]
for lrate, regular in params:
softmax = Softmax()
loss_hist = softmax.train(X_train, y_train, learning_rate=lrate, reg=regular,
num_iters=700, verbose=True)
y_train_pred = softmax.predict(X_train)
accuracy_train = np.mean( y_train == y_train_pred)
y_val_pred = softmax.predict(X_val)
accuracy_val = np.mean(y_val == y_val_pred)
results[(lrate, regular)] = (accuracy_train, accuracy_val)
if(best_val < accuracy_val):
best_val = accuracy_val
best_softmax = softmax
################################################################################
#
END OF YOUR CODE
#
################################################################################
# Print out results.
for lr, reg in sorted(results):
train_accuracy, val_accuracy = results[(lr, reg)]
print 'lr %e reg %e train accuracy: %f val accuracy: %f' % (
lr, reg, train_accuracy, val_accuracy)
print 'best validation accuracy achieved during cross-validation: %f' % best_val

最后

以上就是不安帽子为你收集整理的cs231n - assignment1 - softmax 梯度推导的全部内容，希望文章能够帮你解决cs231n - assignment1 - softmax 梯度推导所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。