CNN基础(1)

classification = score function + loss function +optimization

score function

• linear
• nonlinear

---

An example of mapping an image to class scores. For the sake of visualization, we assume the image only has 4 pixels (4 monochrome pixels, we are not considering color channels in this example for brevity), and that we have 3 classes (red (cat), green (dog), blue (ship) class). (Clarification: in particular, the colors here simply indicate 3 classes and are not related to the RGB channels.) We stretch the image pixels into a column and perform matrix multiplication to get the scores for each class. Note that this particular set of weights W is not good at all: the weights assign our cat image a very low cat score. In particular, this set of weights seems convinced that it’s looking at a dog.

---

loss function (cost function, objective)

• Multicass Support Vector Machine(SVM) loss
  

  $$L_i = \sum_{j\neq y_i} \max(0, s_j - s_{y_i} + \Delta) \ \ \ \ (hinge \ loss)$$

• squared hinge loss

• cross-entropy loss
  

  $$L_i = -\log\left(\frac{e^{f_{y_i}}}{ \sum_j e^{f_j} }\right) \hspace{0.5in} \text{or equivalently} \hspace{0.5in} L_i = -f_{y_i} + \log\sum_j e^{f_j}$$
  where
  $$f_j(z) = \frac{e^{z_j}}{\sum_k e^{z_k}}$$
  is called softmax function

---

the loss function quantifies our unhappiness with predictions on the training set

The Multiclass Support Vector Machine “wants” the score of the correct class to be higher than all other scores by at least a margin of delta. If any class has a score inside the red region (or higher), then there will be accumulated loss. Otherwise the loss will be zero. Our objective will be to find the weights that will simultaneously satisfy this constraint for all examples in the training data and give a total loss that is as low as possible.

---

• Regularization
  

  1. L1 Regularization $$R(W) = \sum|W|$$
     
  2. L2 $$R(W) = \sum_k\sum_l W_{k,l}^2$$
     
  3. Drop out

the full loss becomes: $$L = \underbrace{ \frac{1}{N} \sum_i L_i }_\text{data loss} + \underbrace{ \lambda R(W) }_\text{regularization loss} \\\\$$

---

optimization

• Gradient Descent

  • Stochastic Gradient Descent (SGD or on-line gradient descent)

  • Mini-batch Gradient Descent

• backpropagation

---

An example circuit demonstrating the intuition behind the operations that backpropagation performs during the backward pass in order to compute the gradients on the inputs. Sum operation distributes gradients equally to all its inputs. Max operation routes the gradient to the higher input. Multiply gate takes the input activations, swaps them and multiplies by its gradient.