CS224d Assignment1 答案, Part(2/4)

Assignment1的答案一共被我分成了4部分，分别包含第1，2，3，4题。这部分包含第2题的答案。

2. Neural Network Basics (30 points)

(a). (3 points) Derive the gradients of the sigmoid function and show that it can be rewritten as a function of the function value (i.e. in some expression where only $\sigma(x)$, but not $x$, is present). Assume that the input $x$ is a scalar for this question. Recall, the sigmoid function is

$$\sigma(x)=\frac{1}{1+e^{-x}} \quad(2)$$

解：

$$\begin{align} \sigma'(x)&=-(1+e^{-x})^{-2} (-e^{-x})\\ &=\frac{e^{-x}}{(1+e^{-x})^2}\\ &=\frac{1}{1+e^{-x}}\left(1-\frac{1}{1+e^{-x}}\right)\\ &=\sigma(x)[1-\sigma(x)] \end{align}$$

---

(b). (3 points) Derive the gradient with regard to the inputs of a softmax function when cross entropy loss is used for evaluation, i.e. fi nd the gradients with respect to the softmax input vector $\theta$, when the prediction is made by $\hat{y}=softmax(\theta)$. Remember the cross entropy function is

$$\begin{equation} CE(y,\hat{y}) = -\sum_iy_i\text{log}(\hat{y}_i) \quad(3) \end{equation}$$
where $y$ is the one-hot label vector, and $\hat{y}$ is the predicted probability vector for all classes. (Hint: you might want to consider the fact many elements of $y$ are zeros, and assume that only the k-th dimension of $y$ is one.)

解：根据提示，假设$y$的第k个值为1，其余值都为0，即$y_k=1$那么有：

$$CE(y,\hat{y}) =-y_k\text{log}(\hat{y}_k) = -\text{log}(\hat{y}_k)$$
对于 $\theta$中的第 $i$个元素 θi