Vectorization of the Gradient Descent Algorithm

Vectorization of the Gradient Descent Algorithm

1. Problem Setup

   1. Given a dataset with $m$ samples $(x^{(1)};x^{(2)};...;x^{(m)})$ with each sample having $n$ features (so $x_1^{(1)}$ denotes the value of the first feature of the first sample), and the response variables $(y^{(1)};...;y^{(m)})$ with $y^{(1)}$ being the “output” for the first sample, we want to calculate parameters ${\theta }_0,...,{\theta }_n$ such that the hypothesis $h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+...+{\theta }_m{x}_m$ gives the least-square linear equation for our dataset (i.e. for each pair of $(x_j^{(i)},y^{(i)})$ for all $i$ and $j$.

   2. For convenience of calculation, let $x_0^{(i)}=1$ for all $i$ so ${\vec{x}}^{(i)}=(x_0^{(i)},x_1^{(i)},x_2^{(i)},...,x_n^{(i)})$ is of the same dimension as $\vec{\theta }=({\theta }_0,...,{\theta }_n)$, so we rewrite the hypothesis as $h_{\theta }(x)={\theta }_0{x}_0+{\theta }_1{x}_1+...+{\theta }_m{x}_m$.In gradient descent, we have a training set hypothesisthe following algorithm decreases

   3. The Gradient Descent algorithm:

      1. The cost function is $J({\theta }_0,...,{\theta }_n)=$$\frac{1}{2m}$${\sum }_{i=1}^m({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+...+{\theta }_m{x}_m^{(i)}-y^{(i)}{)}^2$, we aim to minimize this function to find the $\theta$ values.

      2. Assume we have only two features, and we have a suitable learning rate $\alpha >0$. We normally start with some initial values for the ${\theta }^{\prime }s$, as our model is linear, we will always end up at the global optimum using Gradient Descent. The algorithm updates $\theta$'s simultaneously until convergence, and upon convergence, the $\theta$ values will be our parameters:

         1. repeat until convergence
            { \{ {${\theta }_0:={\theta }_0-\alpha$$\frac{\partial J}{\partial {\theta }_0}$;
              ~     ~  ${\theta }_1:={\theta }_1-\alpha$$\frac{\partial J}{\partial {\theta }_1}$;
              ~     ~  ${\theta }_2:={\theta }_2-\alpha$$\frac{\partial J}{\partial {\theta }_2}$$\}$

         2. After taking the partials, we found that each line becomes

            ${\theta }_j:={\theta }_j-\alpha$$\frac{1}{m}$${\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})x_j^{(i)}$.

         3. As we want to update all $\theta$'s simultanerously, we may have to set several ($3$, in this case) ancillary variables to store the RHS’s of the assignment expressions, and doing the assignment together by the end of each iteration. This process can be further simplified by vectorization.

2. Vectorization:

   1. Algorithm:

      { \{ {${\theta }_0:={\theta }_0-\alpha$${\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})x_0^{(i)}$;
        ~   ${\theta }_1:={\theta }_1-\alpha$${\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})x_1^{(i)}$;
        ~   ${\theta }_2:={\theta }_2-\alpha$${\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})x_2^{(i)}$$\}$

   2. Observe that we can seal all the $\theta$'s in a vector $\vec{\theta }$, which is a straightforward $\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ {\theta }_3\end{array}\right]$

   3. We can also seal the sum parts in a vector $\vec{\delta }$:
      $\vec{\delta }=$${\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})\left[\begin{array}{c}{x}_0^{(i)}\\ x_1^{(i)}\\ x_2^{(i)}\end{array}\right]$; this expressions is a sum of $3\times 1$ vectors that evaluates to a $3\times 1$ vector whose entries each contains the change$/\alpha$ of its corresponding $\theta$ in the current iteration)

   4. Therefore, the algorithm becomes:

      $\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ {\theta }_3\end{array}\right]:=\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ {\theta }_3\end{array}\right]-\alpha$${\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})\left[\begin{array}{c}{x}_0^{(i)}\\ x_1^{(i)}\\ x_2^{(i)}\end{array}\right]$, or $\vec{\theta }:=\vec{\theta }-\alpha \vec{\delta }$, which is a faster implementation