Coursera | Andrew Ng (01-week-2-2.11)—向量化

该系列仅在原课程基础上部分知识点添加个人学习笔记，或相关推导补充等。如有错误，还请批评指教。在学习了 Andrew Ng 课程的基础上，为了更方便的查阅复习，将其整理成文字。因本人一直在学习英语，所以该系列以英文为主，同时也建议读者以英文为主，中文辅助，以便后期进阶时，为学习相关领域的学术论文做铺垫。- ZJ

Coursera 课程 |deeplearning.ai |网易云课堂

---

转载请注明作者和出处：ZJ 微信公众号-「SelfImprovementLab」

知乎：https://zhuanlan.zhihu.com/c_147249273

CSDN：http://blog.csdn.net/JUNJUN_ZHAO/article/details/78928182

---

Vectorization 向量化

(字幕来源：网易云课堂)

Welcome back. Vectorization is basically the art of getting rid of explicit for loops in your code.In the deep learning era safety in deep learning in practice,you often find yourself training on relatively large data sets,because that’s when deep learning algorithms tend to shine.And so, it’s important that your code very quickly because otherwise,if it’s running on a big data set,your code might take a long time to run,then you just find yourself waiting a very long time to get the result.

欢迎回来，向量化通常是消除你的代码中显式 for 循环语句的艺术。在深度学习安全领域、深度学习、练习中，你经常发现在训练大数据集时，深度学习算法表现才更加优越，所以你的代码运行得非常快非常重要。否则，如果它运行在一个大的数据集上面，你的代码可能花费很长时间去运行，你会发现你将要等待非常长的时间去得到结果。

So in the deep learning era.I think the ability to perform vectorization has become a key skill.Let’s start with an example.So, what is Vectorization?In logistic regression you need to compute z equals W transpose X plus B,where W was this column vector and X is also this vector.Maybe there are very large vectors if you have a lot of features.So, w and x were both these RnX dimensional vectors.So, to compute W transpose X,if you had a non-vectorized implementation,you would do something like z equals zero.And then for I in range of n-X.So, for i equals 1, to n_x,z+=w[i]*x[i]And then maybe you do z+=b at the end.So, that’s a non-vectorized implementation.

所以在深度学习领域，我认为可以去完成向量化以及变成一个关键的技巧，让我们用一个例子开始，什么是向量化？，在 Logistic 回归中，你需要去计算 $z=w^Tx+b$，$w$ 是列向量 $x$ 也是列向量，如果你有很多的特征 它们就是非常大的向量，所以$w$和$x$都是$R$内的$nx$维向量，所以去计算$W'X$，如果你有一个非向量化的实现，你将会做一些事情 例如$z=0$，python 代码 for i in range(n_x)，python代码 for i in range(n_x)，然后z+=w[i]*x[i]，最后z+=b，所以 这是一个非向量化的实现。

Then you find that that’s going to be really slow.In contrast, a vectorized implementation would just compute W transpose X directly.In Python or a numpy,the command you use for that is z=np.dot(w,x) so this computes w transpose x.And you can also just add b to that directly.And you find that this is much faster.Let’s actually illustrate this with a little demo.So, here’s my jupyter notebook in which I’m going to write some Python code.

你会发现这是真的很慢，作为对比 一个向量化的实现，将会非常直接计算 $W^T X$，在 Python 或者 numpy ，你要用命令 z=np.dot(w,x)，这是在计算$w^Tx$，后面直接加上 $b$，你将会发现这个非常快，让我们用一个小例子说明一下，在我的 jupyter notebook，我将会写一些 Python 代码。

So, first, let me import the numpy library to import.Send np. And so, for example,I can create A as an array as follows.Let’s say print A.Now, having written this chunk of code,if I hit shift enter,then it executes the code.So, it created the array A and it prints it out.Now, let’s do the Vectorization demo.I’m going to import the time libraries, since we use that,in order to time how long different operations take.

首先 让我们导入numpy库，作为 np 例如，像下面这样我将要创建一个数组 A，让我们看下 print A，现在 写下这些代码块，如果我在键盘敲击 shift 和 enter 两个键，它将要执行这个代码，所以 它创建了数组 A 以及 print 它，现在 让我们完成向量化的例子，我将要导入time 库 因为我要使用那个，为了去计算两次不同的操作花费了多长时间。

Can they create an array A?Those random thought rand.This creates a million dimensional array with random values. b = np.random.rand.Another million dimensional array.And, now, tic=time.time, so this measure the current time,c = np.dot (a, b).toc = time.time.And this print,it is the vectorized version.

它们能创建一个数组 A 吗，通过rand函数随机得到，用随机值创建了一个百万维度的数组，b 等于 np.random.rand.，另外一个百万维度的数组，现在 tic=time.time 记录一下当前时间，c 等于 np.dot (a, b).，toc 等于 time.time. print一下，向量化的版本。

import numpy as np 

# 2.11  Vectorization| 向量化 --Andrew Ng 

a = np.array([1,2,3,4,5])

print(a)

# [1 2 3 4 5]
# [Finished in 0.3s]

import time

# 随机创建 百万维度的数组 1000000 个数据
a = np.random.rand(1000000)
b = np.random.rand(1000000)

# 记录当前时间 
tic = time.time()

# 执行计算代码 2 个数组相乘
c = np.dot(a,b)

# 再次记录时间
toc = time.time()

# str(1000*(toc-tic)) 计算运行之间 * 1000 毫秒级
print('Vectorization vresion:',str(1000*(toc-tic)),' ms')
print(c)
# Vectorization vresion: 6.009101867675781  ms
# [Finished in 1.1s]

c = 0
tic = time.time()
for i in range(1000000):
    c += a[i]*b[i]
toc = time.time()
print(c)

print('For loop :',str(1000*(toc-tic)),' ms')
# For loop : 588.9410972595215  ms
# c= 249960.353586
# NOTE: It is obvious that the for loop method  is too slow

It’s a vectorize version.And so, let’s print out.Let’s see the last time,so there’s toc - tic x 1000,so that we can express this in milliseconds.So, ms is milliseconds.I’m going to hit Shift Enter.So, that code took about three milliseconds or this time 1.5,maybe about 1.5 or 3.5 milliseconds at a time.It varies a little bit as I run it,but looks like maybe on average it’s taking like 1.5 milliseconds,maybe two milliseconds as I run this.All right.

这是一个向量化的版本，现在让我们 print 一下，让我们看一下运行时间，这是 ·toc - tic x 1000·，所以我们表达这个在毫秒级上，ms 代表毫秒，我将要同时敲击 Shift 和 Enter，所以这个代码花费 3 毫秒或者这个时间的 1.5 倍，或许大概 1.5 或者 3.5 毫秒，它有点变化当我再次运行它的时候，但是好像 平均她要花费 1.5 毫秒，或许我这次运行是 2 毫秒，是的。

Let’s keep adding to this block of code.That’s not implementing non-vectorize version.Let’s see, c = 0,tic = time.time.Now, let’s implement a for loop.For I in range of 1 million,I’ll pick out the number of zeros right.C += (a,i) x (b,i),and then toc = time.time.Finally, print more than explicit for loop.The time it takes is this 1000 x toc - tic + “ms” to know that we’re doing this in milliseconds.

让我们继续增加这个代码，这是非向量化的版本，让我们看看 c=0，then tic = time.time.，现在它实现了一个for 循环，python 代码:for i in range 1:1000000，我将要取出 0 右边的数字，而 C += (a,i) x (b,i)，以及 toc = time.time.，最后 print for loop，它花费的时间是1000*toc-tic ms，为了知道我们正在做这个在毫秒级别。

Let’s do one more thing.Let’s just print out the value of C. we compute it to make sure that it’s the same value in both cases.I’m going to hit shift enter to run this and check that out.In both cases, the vectorize version and the non-vectorize version computed the same values,as you know, 2.50 to 6.99, so on.The vectorize version took 1.5 milliseconds. The explicit for loop and non-vectorize version took about 400, almost 500 milliseconds.The non-vectorize version took something like 300 times longer than the vectorize version.

让我们再做点其他的事情，我们 print 出 C 的值。，计算一下它确认在两个案例中他们是相同的。我打算去敲击 shift 和 enter 去运行这个 检查一下结果，在两个案例中 向量化版本，和非向量化版本计算了相同的值，正如你知道的 2.50 到 6.99，向量化版本花费了 1.5 毫秒，很明确 for loop 和非向量化版本，花费了大约 400 几乎 500毫秒，非向量化版本多花费了，300 倍向量化版本的时间。

With this example you see that if only you remember to vectorize your code,your code actually runs over 300 times faster.Let’s just run it again.Just run it again.Yeah. Vectorize version 1.5 milliseconds seconds and the for loop.So 481 milliseconds, again,about 300 times slower to do the explicit for loop.If the time slows down,it’s the difference between your code taking maybe one minute to run versus taking say five hours to run.

用这个例子你将会看见，如果你仅仅记住去向量化你的代码，你的代码完全运行 300 倍快，让我们再次运行一下它，再次运行一下它，向量化版本 1.5 毫秒 循环使用了，481 毫秒，大约慢 300 倍用循环做这个，如果时间变慢，如果你的代码一分钟就出结果，那和 5 个小时出结果相差太远了。

And when you are implementing deep learning algorithms,you can really get a result back faster.It will be much faster if you vectorize your code.Some of you might have heard that a lot of scaleable deep learning implementations are done on a GPU or a graphics processing unit.But all the demos I did just now in the jupyter notebook where actually on the CPU.

当你正在实现深度学习算法，这样真的可以更快得到结果，向量化之后 运行速度会大幅上升，你可能听过很多这样的话，可扩展深度学习实现是在 GPU上 做的，GPU 也叫图像处理单元，但是我做的所有的案例都是在 jupyter notebook上面实现，这里只有 CPU。

And it turns out that both GPU and CPU have parallelization instructions.They’re sometimes called SIMD instructions.This stands for a single instruction multiple data.But what this basically means is that,if you use built-in functions such as this np.function or other functions that don’t require you explicitly implementing a for loop.,It enables Python numpy to take much better advantage of parallelism,to do your computations much faster.And this is true both computations on CPUs and computations on GPUs.

CPU 和 GPU，都有并行化的指令，有时候会叫做SIMD指令，意思是单指令流多数据流，这个词的意思是，如果你使用了这样的内置函数，np.function 或者，其他能让你去掉显式for循环的函数，这样 python 的numpy 能够充分利用并行化，去更快的计算，这点对 GPU 和 CPU上面计算都是成立的。

It’s just that GPUs are remarkably good at these SIMD calculations ,but CPU is actually also not too bad at that. ,Maybe just not as good as GPUs.,You’re seeing how vectorization can significantly speed up your code.The rule of thumb to remember is whenever possible, avoid using explicit four loops.,Let’s go onto the next video ,to see some more examples of vectorization,and also start to vectorize logistic regression.

GPU 更加擅长 SIMD 计算，但是 CPU 事实上也不是太差，可能没有 GPU 那么擅长吧，你们见到了向量化能够加速你的代码，经验法则是 只要有其他可能 就不要使用显式 for 循环，让我们进入到下一个视频，去看下更多的向量化的案例，开始向量化 Logistic 回归。

---

重点总结：

向量化（Vectorization）

在深度学习的算法中，我们通常拥有大量的数据，在程序的编写过程中，应该尽最大可能的少使用 loop 循环语句，利用 python 可以实现矩阵运算，进而来提高程序的运行速度，避免 for 循环的使用。

逻辑回归向量化

• 输入矩阵$X$：$（n_x,m）$
  • 权重矩阵$w$：$（n_x,1）$
  • 偏置$b$：为一个常数
  • 输出矩阵$Y$：$（1,m）$

  • 所有 m 个样本的线性输出 Z 可以用矩阵表示：

    $Z = w^{T}X+b$

    Z = np.dot(w.T,X) + b
    A = sigmoid(Z)

    参考文献：

    [1]. 大树先生.吴恩达Coursera深度学习课程 DeepLearning.ai 提炼笔记（1-2）– 神经网络基础

    ---

    PS: 欢迎扫码关注公众号：「SelfImprovementLab」！专注「深度学习」，「机器学习」，「人工智能」。以及 「早起」，「阅读」，「运动」，「英语 」「其他」不定期建群 打卡互助活动。