BP neural network note

Note系列以学习笔记为主，可能会很零散，主要跟我的需求相关。鉴于我的写作水平，不论中英文，都很差。我就怎么舒服怎么写啦。

Today, I read the paper from Rumelhart et al. and their report. Here I write down some important points and statements. Most of them are words in the paper.

1. They generated a gradient descent method for finding weights in any feedforward network with semilinear units. And they got some further observations:

    Any number of weights in the network can be fixed.

    Some units recieve two different kinds of error: that from the direct comparison with the target and that passed through the other output units whose activation it affects.

2. I think the most important part is learning procedure, which could evolve a set of weights to produce an arbitrary mapping from input to output. So the essential part is weight.

3. The gradient descent procedure is bound by the problem of local maxima or minima. It's a common problem for any hill climbing procedure.

 

        (From: https://en.wikipedia.org/wiki/Maxima_and_minima#/media/File:Extrema_example_original.svg)

4. A useful activation function. The linear threshold function is discontinuous and hence will not suffice for the generalized delta rule. Thus, we need a continuous, nonlinear activation function. Most are logistic activation function in their experiments. In a practical learning situation in which the desired outputs are binary {0,1}, the system can never actually achieve these values. Therefore, we typically use the values of 0.1 and 0.9 as the targets.

5. The learning rate. The constant of proportionality is the learning rate. The larger this constant, the larger the changes in the weights.