Basics of Neural Network Programming - Vectorization

The notes when study the coursera class Neural Networks & Deep Learning by Andrew Ng, section vectorization. Share it with you.

---

Vectorization is the art to get rid of the for loops in the code. The ability to perform vectorization has become a key skill.

 

Figure-1

 

Figure-1 shows two different ways to calculate of logistic regression. Using for loop will be very time consuming if we have large number of features. While vectorization will be much faster.

Let's illustrate this with a little demo.

Figure-2

For both the vectorized and non-vectorized version output the same value for the variable c. The vectorized version took 1 ms; the explicit for loop took man than 700 ms. So, if you remember to vectorize your code, your code actually runs over 700 times faster. 

A lot of scalable deep learning implementations are done on a GPU (Graphics Processing Unit), but all the demos just did in the jupyter lab were actually run on the CPU. It turns out that both GPU and CPU have parallelization instructions which sometimes called SIMD (Single Instruction Multiple Data). If you use some built in functions like np.dot here, it enables Python numpy to take much better advantage of parallelism and do your computations much faster.

So, the rule of thumb to remember is whenever possible, avoid using explicit for loops.

<end>