Coursera | Andrew Ng (01-week-3-3.11)—随机初始化

该系列仅在原课程基础上部分知识点添加个人学习笔记，或相关推导补充等。如有错误，还请批评指教。在学习了 Andrew Ng 课程的基础上，为了更方便的查阅复习，将其整理成文字。因本人一直在学习英语，所以该系列以英文为主，同时也建议读者以英文为主，中文辅助，以便后期进阶时，为学习相关领域的学术论文做铺垫。- ZJ

Coursera 课程 |deeplearning.ai |网易云课堂

---

转载请注明作者和出处：ZJ 微信公众号-「SelfImprovementLab」

知乎：https://zhuanlan.zhihu.com/c_147249273

CSDN：http://blog.csdn.net/junjun_zhao/article/details/79004114

---

3.11 Random Initialization (随机初始化 )

(字幕来源：网易云课堂)

When you train your neural network.It is important to initialize the weights randomly,for logistic regression it was okay to initialize the weights to zero,but for a neural network of initialize the arrays of parameters to all zero,and apply gradient descent it won’t work,let’s see why,so you have here two input features,so n^[0] is equal to two and two hidden units,so $n^{[1]}$ is equal to two,and so the matrix associated with a hidden layer or $W^{[1]}$,it is going to be 2x2,let’s say that you initialize it to all zeros 0 0 0 0, 2 by 2 matrix,and let’s say $b^{[1]}$ is also equal to 0 0,turns out initializing the bias terms b to 0 is actually okay,but the initializing W to all zeros is a problem.

当你训练神经网络时，随机初始化权重非常重要，对于 logistic 回归 可以将权重初始化为零，但如果将神经网络的各参数数组全部初始化为 0，再使用梯度下降算法 那会完全无效，我们来看看为什么，所以你有两个输入特征，所以 $n^{[0]}$等于 2 然后有两个隐藏单元，所以 $n^{[1]}=2$，所以和隐层相关的矩阵$W^{[1]}$，是 2×2 的，假设你把它全部初始化成0, 0 0 0 0, 2×2 矩阵，我们也令$b^{[1]}$ 等于0 0，将偏置项 b 初始化为 0 实际上是可行的，但把$W$初始化成全零就成问题了。

so the problem with this form of initialization is that ,for any example you give it you have that $a^{[1]}_1$ and $a^{[1]}_2$ will be equal,all right so this activation and this activation would be same,because both of these hidden units are computing exactly the same function,and then when you compute back propagation ,it turns out that $dz^{[1]}_1$ and $dz^{[1]}_2$ will also be the same kind of by symmetry right,both of these hidden units will initialize the same way,technically for what I’m saying I’m assuming that the outgoing weights are also identical,so that $W^{[2]}$ is equal to 0 0,but if you initialize the neural network this way,then this hidden unit and this hidden unit are completely identical,so they’re completely sometimes you say they’re completely symmetric,which just means that the computing exactly the same function,and by kind of a proof by induction.

这种初始化形式的问题在于，你给网络输入任何样本 你的 $a^{[1]}_1$和$a^{[1]}_2$是一样的，所以这两个激活函数完全一样，因为两个隐藏单元都在做完全一样的计算，当你计算反向传播时，事实证明 出于对称性 $dz^{[1]} _1$和 $dz^{[1]} _2$ 也是相同的，这两个隐藏单元会以同样方式初始化，技术上 我说的就是 假设输出的权重也是一样的，所以$W^{[2]}$就等于 0，但如果你以这种方式初始化神经网络，那么这个隐藏单元和这个隐藏单元就完全一样了，所以这就是所谓的完全对称，意味着节点计算完全一样的函数，我们可以通过归纳法证明。

it turns out that after every single iteration of training,your two hidden units are still computing exactly the same function,since part of the show that dW will be a matrix that looks like this,where every row takes on the same value,so we perform a weight update so you perform,when you perform a weight update $W^{[1]}$ gets updated as $W^{[1]}$ minus alpha times dW,you find that $W^{[1]}$ after every iteration will have,you know the first row equal to the second row,so it’s possible to construct a proof by induction that ,if you initialize all the ways all the values of W to 0,then because both hidden units start off computing the same function,and both hidden units have the same influence on the output unit,then after one iteration that same statement is still true,the two hidden units are still symmetric,and therefore by induction after two iterations three iterations and so on.

每次训练迭代之后，两个隐藏单元仍然在计算完全相同的函数，因为很大一部分网络记忆$dW$是这样的一个矩阵，其中每行的数值都一样，然后我们执行一次权重更新 所以你计算，所以你计算一次权重更新 $W^{[1]}$更新为$W^{[1]}-α·dW$，你会发现每次迭代后$W^{[1]}$的形式都是，第一行和第二行完全一样，所以是可以通过归纳法证明，如果你将$W$所有值初始化为 0，那么因为两个隐藏单元一开始就在做同样的计算，两个隐藏单元对输出单元的影响也一样大，那么在一次迭代之后 同样的对称性依然存在，两个隐藏单位仍然是对称的，所以通过归纳两到三次迭代就可以证明。

No matter how long you train your neural network,both hidden units are still computing exactly the same function,and so in this case there’s really no point to having more than one hidden unit,because they’re all computing the same thing,and of course for larger neural networks let’s see if you have three features,and maybe a very large number of hidden units,a similar argument works to show that what with a neural network like this,because I won’t drawn all the edges if you initialize the weight to zero,then all of your hidden units are symmetric,and no matter how long you run gradient descent,they’ll all continue to compute exactly the same function,so that’s not helpful,because you want two different hidden units to compute different functions,the solution to this is to initialize your parameters randomly.

无论你训练神经网络多长时间，两个隐藏单元仍然在计算完全一样的函数，所以在这种情况下 多个隐藏单元真的没有意义，因为它们计算的都是同样的东西，当然 对于更大的神经网络 或者输入有三个特征，或者隐藏单元的数目非常多，那么用类似的方法可以证明 这样的神经网络，如果你把所有权重都初始化成 0 我就不把所有连线画出来了，那么所有隐藏单元都是对称的，不管你跑多久梯度下降，它们都在计算完全一样的函数，所以没有一点用处，因为你需要两个不同的隐藏单元 去计算不同的函数，这个问题的解决方案是随机初始化所有参数。

so here’s what you do,you can set $W^{[1]}$ = np.random.randn,this generates on Gaussian random variable (2,2),and then usually you multiply this by a very small number such as 0.01,so you initialize it to very small random values,and then b um it turns out that b does not have this symmetry problem,what’s called a symmetry breaking problem,so is okay to initialize b to just zeros because,so long as W is initialized randomly,you start off with a different hidden units computing different things,and so you no longer have this um symmetry breaking problem,and then similarly for $W^{[2]}$,you can initialize that randomly ,and b2 you can initialize that to zero,so you might be wondering,you know where did this constant come from,and why is it 0.01 why not put the number 100 or 1000,turns out that we usually prefer to initialize the ways to,very very small random values.

你应该这么做，你可以令 $W^{[1]}$ = np.random.randn，这可以产生参数为 (2,2) 的高斯分布随机变量，然后你再乘以一个很小的数字 比如 0.01，所以你将权重初始化成很小的随机数，然后 b .. 其实 b 完全没有这个对称性问题，所谓的破坏对称性问题，把b初始化成 0 是可以的 因为，只要 W 随机初始化，你一开始还是用不同的隐藏单元计算不同的函数，这里并不需要破坏对称性，然后类似地对于 $W^{[2]}$，你也可以随机初始化，还有b2 你可以初始化成 0，你可能会奇怪，这个常数是什么地方来的，为什么是 0.01 为什么不用 100 或 1000 之类的数字，实际上 我们通常喜欢把权重矩阵，初始化成非常非常小的随机值。

because um if you’re using a say tanh or sigmoid activation function,or if you have a sigmoid even just at the output layer,if the weights are too large then when you compute the activation values,remember that $z^{[1]}$ is equal to $W^{[1]}$ x plus $b^{[1]}$,and then $a^{[1]}$ is the activation function applied to $z^{[1]}$,so if W is very big, z will be very big,or these some values of z will be either very large or very small,and so in that case you’re more likely to end up at,these flat parts of the tanh function or the sigmoid function,where the slope of the gradient is very small,meaning that gradient descent would be very slow,and so learning would be very slow.

因为 如果你用的是 tanh 或者 sigmoid 激活函数，或者你在输出层有一个 sigmoid 函数，那如果权重太大 当你计算激活函数值时，要记住 $z^{[1]}=W^{[1]} x+b^{[1]}$，然后$a^{[1]}$是应用于$z^{[1]}$ 的激活函数，所以如果 W 很大 z 就会很大，或者这些 z 值会很大或者很小，所以在这种情况下 你最后可能落在，这些 tanh 函数或者 sigmoid 函数的平缓部分，梯度的斜率非常小，意味着梯度下降法会非常慢，所以学习会很慢。

so just a recap,if W is too large you’re more likely to end up,even at the very start of training with very large values of z,which causes your tanh or a sigmoid activation function to be saturated,thus slowing down learning,if you don’t have any sigmoid or tanh activation functions throughout your neural network,this is less of an issue but if you’re doing binary classification,and your output unit is a sigmoid function,then you know you just don’t want the initial parameters to be too large,so that’s why multiplying by 0.01 would be something reasonable to try,or any other small number and same for $W^{[2]}$,this can be a np.random.randn.I guess this would be 1 by 2 in this example times 0.01, missing an s there,so finally it turns out that sometimes there can be better constants than 0.01.

所以回顾一下，如果 W 太大 那么你就很可能，在一开始训练时就落在 z 很大的区域，导致你的 tanh 或 Sigmoid 激活函数接近饱和，从而减慢学习速度，如果你的神经网络中没有任何 sigmoid 或者 tanh 激活函数，那问题可能没那么大 但如果你在做二元分类，你的输出单元是 Sigmoid 函数，那么你就不希望初始参数太大，所以用 0.01 是比较合理的，或者任意其他小数字 初始化 $W^{[2]}$ 也一样，可能用命令np.random.randn，我想这个例子中就是 1×2 这个例子里乘以 0.01 这里漏了 s，所以实际上有时有比 0.01 更好用的常数。

when you’re training a neural network with just one hidden layer,this is a relatively shallow neural network without too many hidden layers,setting to 0.01 will probably work ok,but when you’re training a very very deep neural network,then you might want to pick a different constant as 0.01,and in next week’s material will talk a little bit about,how and when you might want to choose a different constant than 0.01,but either way it will usually end up being a relatively small number,so that’s it so this week videos you now know how to set up a neural network of a hidden layer,initialize the parameters make predictions using forward prop,as well as compute derivatives and apply gradient descent using back prop,so that you should be able to do the quizzes,as well as this week’s programming exercises,best of luck with that.I hope you have fun with the exercise,and look forward to seeing you in week 4 materials.

但当你训练一个单隐层神经网络时，这是一个相对较浅的神经网络 没有太多的隐藏层，设为 0.01 应该还可以，但是当你训练一个很深的神经网络的时候，你可能要试试 0.01 以外的常数，在下周的材料中会谈一点，什么时候应该选用 0.01 以外的常数，但不管怎样 初始化参数一般都很小，所以在这周的视频里 你知道如何设立单隐层神经网络，初始化参数 并用正向传播计算预测值，还有计算导数 然后使用梯度下降 反向传播，你们应该能够完成小测，以及本周的编程练习，祝你们好运，我希望你做练习时玩得开心，并期待在第 4 周的材料中见到你。

---

重点总结：

随机初始化

如果在初始时，两个隐藏神经元的参数设置为相同的大小，那么两个隐藏神经元对输出单元的影响也是相同的，通过反向梯度下降去进行计算的时候，会得到同样的梯度大小，所以在经过多次迭代后，两个隐藏层单位仍然是对称的。无论设置多少个隐藏单元，其最终的影响都是相同的，那么多个隐藏神经元就没有了意义。

在初始化的时候，W 参数要进行随机初始化，b 则不存在对称性的问题它可以设置为 0。

以 2 个输入，2 个隐藏神经元为例：

W = np.random.rand((2,2))* 0.01
b = np.zero((2,1))

这里我们将 W 的值乘以 0.01 是为了尽可能使得权重 W 初始化为较小的值，这是因为如果使用 sigmoid 函数或者 tanh 函数作为激活函数时，W 比较小，则 $Z = WX+b$ 所得的值也比较小，处在 0 的附近，0 点区域的附近梯度较大，能够大大提高算法的更新速度。而如果 W 设置的太大的话，得到的梯度较小，训练过程因此会变得很慢。

ReLU 和 Leaky ReLU 作为激活函数时，不存在这种问题，因为在大于 0 的时候，梯度均为 1。

参考文献：

[1]. 大树先生.吴恩达Coursera深度学习课程 DeepLearning.ai 提炼笔记（1-3）– 浅层神经网络

---

PS: 欢迎扫码关注公众号：「SelfImprovementLab」！专注「深度学习」，「机器学习」，「人工智能」。以及 「早起」，「阅读」，「运动」，「英语 」「其他」不定期建群 打卡互助活动。