Optimization Week 14: Stochastic gradient descent

1 Noisy Unbiased (sub) Gradient (NUS)

• $g$ is a NUS of $f$ if $E[g(x)|x]\in \partial f(x),\forall x$;
• If $\nabla f(x)$ exists, $E[g(x)|x]=\nabla f(x)$
• $f(x)=\frac{1}{n}{\sum }_i{f}_i(x)$, $g(x)=\nabla f_i(x)$, index $i$ chosen random uniform.
• Random coordinate descent: $x_{+}^i=x^i-\eta \frac{\partial }{\partial x_i}f(x)$, coordinate index $i$ chosen random uniform.

2 Stochastic gradient descent

2.1 Update rule

$$x_{k+1}=x_k-\eta g(x_k)$$

• $g(x_k)=$ gradient estimate
• Random updates at every step
• Keep track of $x_{BEST}$
  $$\underset{x}{\min }\frac{1}{n}\sum _{i=1}^n{f}_i(x)$$ Gradient descent update rule: $$x_{k+1}=x_k-{\eta }_k\frac{1}{n}\sum _i\nabla f_i(x_k)$$
  Stochastic gradient descent rule: $$x_{k+1}=x_k-{\eta }_k\nabla f_{i_k}(x_k)$$ $i_k$ random index

2.2 Convergence rate

$$E[f(x_{BEST}^T)]-f^{\ast }\le \frac{R^2+G^2{\sum }_0^T{\eta }_k^2}{2{\sum }_0^T{\eta }_k}$$

• $||x_0-x^{\ast }|{|}^2\le R^2$

• Variance $G^2\ge E[||g(x)|{|}^2|x]$

• Fixed step size not good, will jump away from $x^{\ast }$

• Error after $T$ steps:
  $$\begin{array}{ccc}& GD& SGD\\ \text{convex }f& O(1/\sqrt{T})& O(1/\sqrt{T})\\ \text{Lipschitz }\nabla f& O(1/T)& O(1/\sqrt{T})\\ \text{Strongly convex}& O(C^T)& O(1/T)\\ & O(nd)& O(d)\end{array}$$

• Stochastic gradient descent, because it is forced to use a step size of close to zero, cannot adapt to the function. If the function becomes better, gradient descent can become better, but stochastic gradient descent cannot really become better.

• Faster iteration, but slower convergence & inability to adapt to good $f$.

• Not linear convergence, since cannot self-tune caused by variance.

2.3 Step size

• ${\sum }_0^T{\eta }_k\to \infty$
• ${\eta }_k\to 0$ because var($g_k$) $\nrightarrow$ 0, but SGD will be slower than it needs to be

3 Mini-batch Stochastic Gradient Descent

3.1 Update rule

$$x_{k+1}=x_k-{\eta }_k\frac{1}{|I_k|}\sum _{i\in I_k}\nabla f_i(x_k)$$

• UPDATE$=\frac{1}{|I_k|}{\sum }_{i\in I_k}\nabla f_i(x_k)$
• $E(\text{UPDATE}|x_k)=\nabla f(x_k)$
• $Var[\text{UPDATE}|x_k]=Var[SGD|x_k]$

3.2 Convergence rate

• Error after $T$ steps:
  $$\begin{array}{cc}\text{SGD}& \text{Mini-batch SGD}\\ O(1/\sqrt{T})& O(1/\sqrt{T}+1/\sqrt{|I|T})\\ O(d)& O(|I|d)\end{array}$$
• Gain in # iterations. No gain in compute cost
• Need variance reduction
• Even for Lipschitz $\nabla f$
• We have faster convergence but it’s a for a constant batch size B
• mini-batch SGD has essentially the same trend as SGD but it has lower waves

3.3 Step size

• The problem is SGD, needs ${\eta }_k\to 0$ because $Var\nrightarrow 0$
• Remains a problem for Mini-batch SGD.

4 Variance reduction in SGD

4.1 Bias and Variance

• Bias: $E[g(x_k)|x_k]-\nabla f(x_k)$
• Variance: $Var[||g(x_k)|||x_k]$
• When $Bias=0$ and $Var\le G^2$, $O(\frac{G^2}{\sqrt{T}})$ convergence

And the same holds true if you’re doing mini-batch SGD and the constant becomes better by a factor B.

4.2 Stochastic Average Gradient

4.2.1 Update rule

Minimization problem:
$$\underset{x}{\min }\frac{1}{n}\sum _{i=1}^n{f}_i(x)$$

• Maintain $g_1^k,\dots ,g_n^k$, current estimate for $\nabla f_i^k$
• Initialize with all samples: $g_i^0=\nabla f_i^0$

Update:

• At step k, pick $i_k$ randomly
  $$g_{i_k}^k=\nabla f_{i_k}(x^{k-1})$$ $$g_j^k=g_j^{k-1},\text{ for }j\ne i_k$$

• Update $$x^k=x^{k-1}-{\eta }_k\frac{1}{n}\sum _{i=1}^n{g}_i^k$$ Use all $g_i$ including the stale ones

4.2.2 Step size

• This bias is not zero at any given $x^k$, asymptotically unbiased, (bias$\to 0$, variance$\to 0$), so that is in contrast to simple stochastic, to a simple SGD, where the bias was always zero.
• This means is that we can use a fixed step size, which was the key to having an algorithm that adapts to the niceness of F.

4.2.3 Convergence rate

• Smooth function: $O(1/T)$
• Strongly convex: $O(C^T)$, linear convergence

4.3 SAGA, variant of stochastic average gradient (note)

4.4 SVRG Stochastic variance reduction gradient descent