训练过程中的sigmoid激活值特性

• sigmoid神经网络在每层的激活值均值和标准偏差在训练过程中的演变情况

 

• 原因：

     Our proposed explanation rests onthe hypothesis that the transformation that the lower layersof the randomly initialized network computes initially isnot useful to the classification task, unlike the transforma-tion obtained from unsupervised pre-training. The logisticlayer outputsoftmax(b+W h)might initially rely more onits biasesb(which are learned very quickly) than on the tophidden activationshderived from the input image (becausehwould vary in ways that are not predictive ofy, maybecorrelated mostly with other and possibly more dominantvariations ofx). Thus the error gradient would tend topushW htowards 0, which can be achieved by pushinghtowards 0. In the case of symmetric activation functionslike the hyperbolic tangent and the softsign, sitting around0 is good because it allows gradients to flow backwards.However, pushing the sigmoid outputs to 0 would bringthem into a saturation regime which would prevent gradi-ents to flow backward and prevent the lower layers fromlearning useful features. Eventually but slowly, the lowerlayers move toward more useful features and the top hiddenlayer then moves out of the saturation regime. Note how-ever that, even after this, the network moves into a solutionthat is of poorer quality (also in terms of generalization) then those found with symmetric activation functions, ascan be seen in figure 11.