AdaGrad - Adaptive Subgradient Methods

AdaGrad - Adaptive Subgradient Methods

https://cs.stanford.edu/~ppasupat/a9online/1107.html

AdaGrad is an optimization method that allows different step sizes for different features. It increases the influence of rare but informative features.

Setup

Let $f:R^n\to R$ be a convex function and $X\subseteq R^n$ be a convex compact subset of $R^n$. We want to optimize$$\underset{x\in X}{min}f\left(x\right)$$

There is a tradeoff between convergence rate and computation time. Second-order methods (e.g. BFGS) have convergence rate $O\left(1/e^T\right)$ but need $O\left(n^3\right)$ computation time. We will focus on first-order methods, which need only $O\left(n\right)$ computation time. (In machine learning, speed is more important than fine-grained accuracy.)

Proximal Point Algorithm (Gradient Descent)

In proximal point algorithm, we repeatedly take small steps until we reach the optimum. If $f$ is differentiable, we estimate $f\left(y\right)$ with$$f\left(y\right)\approx f\left(x\right)+⟨\nabla f\left(x\right),y-x⟩$$

The update rule is$$\begin{array}{cc}{x}^{(k+1)}& =\underset{x\in X}{argmin}\left\{f\right(x^{\left(k\right)})+⟨\nabla f\left(x^{\left(k\right)}\right),x-x^{\left(k\right)}⟩+\frac{1}{2{\alpha }_k}{\Vert x-x^{\left(k\right)}\Vert }_2^2\}\\ & =\underset{x\in X}{argmin}\{⟨\nabla f\left(x^{\left(k\right)}\right),x⟩+\frac{1}{2{\alpha }_k}{\Vert x-x^{\left(k\right)}\Vert }_2^2\}\end{array}$$

We regularize with step size ${\alpha }_k$ (usually constant in batch descent) to ensure that $x^{(k+1)}$ is not too far from $x^{\left(k\right)}$. For $X=R^n$ (unconstrained optimization), solving for optimal $x$ gives$$x^{(k+1)}=x^{\left(k\right)}-\alpha \nabla f\left(x^{\left(k\right)}\right)$$

The convergence rate is $O\left(1/T\right)$. See the proof in the Note.

Stochastic Gradient Descent

Suppose $f$ can be decomposed into$$f\left(x\right)=\frac{1}{N}\sum _{i=1}^{N}F(x,a_i)$$where $a_i\in R^n$ are data points. If $N$ is large, calculating $\nabla f\left(x\right)$ becomes expensive. Instead, we approximate$$\nabla f\left(x^{\left(k\right)}\right)\approx g^{\left(k\right)}:=\nabla F(x^{\left(k\right)},a_i)\text{where}i\stackrel{\text{random}}{\leftarrow }\{1,\dots ,N\}$$Stochastic gradient descent uses the update rule$$x^{(k+1)}=\underset{x\in X}{argmin}\{⟨g^{\left(k\right)},x⟩+\frac{1}{2{\alpha }_k}{\Vert x-x^{\left(k\right)}\Vert }_2^2\}$$

If $i$ is uniformly random, $E^{}[g^{\left(k\right)}\mid x^{\left(k\right)}]=\nabla f\left(x^{\left(k\right)}\right)$, so the estimate is unbiased. However, we should decrease ${\alpha }_k$ over time (strengthen regularization) to prevent the noisy $g^{\left(k\right)}$ from ruining the already-good $x^{\left(k\right)}$.

The convergence rate is $O\left(1/\sqrt{T}\right)$. See the proof in the Note.

Improving the Regularizer

Consider the problem$$\underset{x\in R^2}{min}100x_1^2+x_2^2$$

Gradient descent converges slowly because the L2 norm ${\Vert x-x^{\left(k\right)}\Vert }_2^2$ in the regularizer does not match the geometry of the problem, where changing $x_1$ has much more effect than changing $x_2$.

Instead, let's use the matrix norm. For a positive definite matrix $B$, define$${\parallel x\parallel }_B^2:=x^{T}Bx\ge 0$$For simplicity, let $B$ be a diagonal matrix with positive terms. So$${\parallel x\parallel }_B^2:=\sum _j{b}_{jj}{x}_j^2\ge 0$$

The update rule becomes$$\begin{array}{cc}{x}^{(k+1)}& =\underset{x\in X}{argmin}[⟨\nabla f\left(x^{\left(k\right)}\right),x⟩+\frac{1}{2{\alpha }_k}{\Vert x-x^{\left(k\right)}\Vert }_B^2]\\ & =x^{\left(k\right)}-\alpha B^{-1}\nabla f\left(x^{\left(k\right)}\right)(\text{if}X=R^n)\end{array}$$

If $f\left(x\right)=\frac{1}{2}{x}^{T}Ax$ for some $A$, then by choosing $B=A$ or something similar to $A$, the method converges very quickly. In practice, however, picking such $B$ can be difficult or impossible (e.g. online setting), so we need to estimate $B$ as the method progresses.

Idea Use the previously computed gradients $g_k$ to estimate the geometry.$$\text{large}\frac{\partial f}{\partial x_j}⟺\text{large}{A}_{jj}⟺\text{large}{B}_{jj}$$

Another intuition is that we should care more about rare features. For rare features (small accumulative gradient), we should regularize less (use small $B_{jj}$).

AdaGrad is an extension of SGD. In iteration $k$, define$$G^{\left(k\right)}=diag{\left[\sum _{i=1}^k{g}^{\left(i\right)}\right(g^{\left(i\right)}{)}^T]}^{1/2}$$i.e., let $G^{\left(k\right)}$ be a diagonal matrix with entries$$G_{jj}^{\left(k\right)}=\sqrt{\sum _{i=1}^k(g_j^{\left(i\right)}{)}^2}$$and use the update rule$$\begin{array}{cc}{x}^{(k+1)}& =\underset{x\in X}{argmin}\{⟨\nabla f\left(x^{\left(k\right)}\right),x⟩+\frac{1}{2\alpha }{\Vert x-x^{\left(k\right)}\Vert }_{G^{\left(k\right)}}^2\}\\ & =x^{\left(k\right)}-\alpha G^{-1}\nabla f\left(x^{\left(k\right)}\right)(\text{if}X=R^n)\end{array}$$

The convergence rate is $O\left(1/\sqrt{T}\right)$ but with a better constant depending on the geometry. See the proof in the Note.

Reference

Paper http://stanford.edu/~jduchi/projects/DuchiHaSi10_colt.pdf

Slides https://web.stanford.edu/~jduchi/projects/DuchiHaSi12_ismp.pdf