Neural Networks and Deep Learning 1

本章我做的矩阵乘法的地方与这本书后面刚好相反，我这地方答题用的矩阵乘相当于书本中交换转置相乘

目录

Ch 01 Using neural nets to recognize handwritten digits

在线书籍http://neuralnetworksanddeeplearning.com/chap1.html

Sigmoid neurons

• Sigmoid neurons simulating perceptrons, part I:
  Suppose we take all the weights and biases in a network of perceptrons, and multiply them by a positive constant, $c>0$. Show that the behaviour of the network doesn’t change.

证明: 由文章中公式(2)$$output=\{\begin{array}{rcl}0& & if{\sum }_i{w}_i{x}_i+b\le 0\\ 1& & if{\sum }_i{w}_i{x}_i+b>0\end{array}$$
因此对任意感知元，参数$w$与偏置$b$增大$c>0$倍，则每个感知元输出为$$output=\{\begin{array}{rcl}0& & if{\sum }_{i}c\cdot w_i{x}_i+c\cdot b\le 0\\ 1& & if{\sum }_{i}c\cdot w_i{x}_i+c\cdot b>0\end{array}$$
由于$c>0$，且${\sum }_{i}c\cdot w_i{x}_i+c\cdot b=c\cdot ({\sum }_i{w}_i{x}_i+b)$，若${\sum }_i{w}_i{x}_i+b>0$，则$c\cdot ({\sum }_i{w}_i{x}_i+b)>0$，若$c\cdot ({\sum }_i{w}_i{x}_i+b)\le 0$ 则$c\cdot ({\sum }_i{w}_i{x}_i+b)\le 0$，所以对于每个感知元输出相同。

• Sigmoid neurons simulating perceptrons, part II
  Suppose we have the same setup as the last problem - a network of perceptrons. Suppose also that the overall input to the network of perceptrons has been chosen. We won’t need the actual input value, we just need the input to have been fixed. Suppose the weights and biases are such that $w\cdot x+b\ne 0$ for the input x to any particular perceptron in the network. Now replace all the perceptrons in the network by sigmoid neurons, and multiply the weights and biases by a positive constant $c>0$. Show that in the limit as $c\to \infty$ the behaviour of this network of sigmoid neurons is exactly the same as the network of perceptrons. How can this fail when $w\cdot x+b=0$ for one of the perceptrons?
  **证明：**对于每个感知元，
  $$output=\{\begin{array}{rcl}0& & if{\sum }_i{w}_i{x}_i+b\le 0\\ 1& & if{\sum }_i{w}_i{x}_i+b>0\end{array}$$
  由于$c\to \infty$，当${\sum }_i{w}_i{x}_i+b<0$时，$c\cdot ({\sum }_i{w}_i{x}_i+b)\to -\infty$，$\sigma (x)=\frac{1}{1+e^{-({\sum }_i{w}_i{x}_i+b)}}\to 0$，同理，当${\sum }_i{w}_i{x}_i+b>0$时，$c\cdot ({\sum }_i{w}_i{x}_i+b)\to +\infty$，$\sigma (x)=\frac{1}{1+e^{-({\sum }_i{w}_i{x}_i+b)}}\to 1$，与感知元输出结果相同。而当${\sum }_i{w}_i{x}_i+b=0$时，$c\cdot ({\sum }_i{w}_i{x}_i+b)=0$，$\sigma (x)=\frac{1}{1+e^{-({\sum }_i{w}_i{x}_i+b)}}=\frac{1}{2}$，与感知元输出不同。

A simple network to classify handwritten digits

• There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first $3$ layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least $0.99$, and incorrect outputs have activation less than $0.01$.
  
  **证明：**最后一层为sigmoid函数输出，因此神经元输出值为$0$或$1$，我们将做后一层$4$个神经元组成的二进制向量，如下表所示。
  |数字| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
  |-| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
  |编码| 0000 | 0001 | 0010 | 0011 | 0100 | 0101 | 0110 | 0111 | 1000 | 1001 |
  原输出层期望输出如下表，
  | 数字 | 编码 |
  | :–: | :–: |
  | 0 | 1000000000 |
  | 1 | 0100000000 |
  | 2 | 0010000000 |
  | 3 | 0001000000 |
  | 4 | 0000100000 |
  | 5 | 0000010000 |
  | 6 | 0000001000 |
  | 7 | 0000000100 |
  | 8 | 0000000010 |
  | 9 | 0000000001 |

因此$$W=[$$