Reinforcement Learning with Code 【Chapter 6. Stochastic Approximation】

Reinforcement Learning with Code

This note records how the author begin to learn RL. Both theoretical understanding and code practice are presented. Many material are referenced such as ZhaoShiyu’s Mathematical Foundation of Reinforcement Learning, .

Chapter 6. Stochastic Approximation

​ Stochastic approximation refers to a borad class of stochastic iterative algorithms solving root finding or optimization problems.

6.1 Robbins-Monro algorithm

​ Compared to many other root-finding algorithms such as gradient-based methods, stochastic approximation is powerful in the sense that it does not require knowing the expression of the objective function or its derivative. The Robbins-Monro algorithm is a pioneering work in the field of stochastic approximation.

​ Consider the function $g(w)$, that we would like to find the root of the equation $g(w)=0$, where $w\in \mathbb{R}$ is the variable to be solved and $g:\mathbb{R}\to \mathbb{R}$ is a function. Suppose the expression of the function is unkown. We only can know the input $w$ and the measured output $\tilde{g}(w,\eta )=g(w)+\eta$, where $\eta \in \mathbb{R}$ is the observation error.

​ (Robbins-Monro Theorem). In the Robbins-Monro algorithm, if

(1) $0<c_1\le {\nabla }_{w}g(w)\le c_2$ for all $w$;

(2) ${\sum }_{k=1}^{\infty }{a}_k=\infty$ and ${\sum }_{k=1}^{\infty }{a}_k^2<\infty$;

(3) $\mathbb{E}[{\eta }_k|{\mathcal{H}}_k]=0$ and $\mathbb{E}[{\eta }_k^2|{\mathcal{H}}_k]<\infty$;

where H k = { w k , w k − 1 , …   } \mathcal{H}_k = \{w_k,w_{k-1},\dots \} Hk​={wk​,wk−1​,…}. The increment or iterative fashion as follows

$$w_{k+1}=w_k-a_k\tilde{g}(w,\eta ),k=1,2,3,\cdots$$

where the $a_k$ is a positive coefficient. Then $w_k$ converges with probability $1$ (w.p.$1$) to the root $w^{\ast }$ satisfying $g(w^{\ast })=0$.

Example:

​ We use the Robbins-Monro theorem to sovle the mean eastimation problem. Consider the function
$$g(w)=w-\mathbb{E}[X]$$
When we take sample $x$ of random variable $X$, the oberservation is

$$\begin{array}{rl}\tilde{g}(w,x)& =w-x\\ \tilde{g}(w,\eta )& =w-\mathbb{E}[X]+(\mathbb{E}[X]-x)\\ \tilde{g}(w,\eta )& =\underset{g(w)}{\underbrace{w-\mathbb{E}[X]}}+\underset{\eta }{\underbrace{\mathbb{E}[X]-x}}\end{array}$$

Therefore, the observation $\tilde{g}(w,x)$ is the sum of $g(w)$ and an observation error $\eta$. Hence the Robins-Monro algorithm for solving $g(w)=0$ is

$$\begin{array}{rl}{w}_{k+1}& =w_k-a_k\tilde{g}(w,\eta )\\ w_{k+1}& =w_k-a_k\tilde{g}(w,x)\\ w_{k+1}& =w_k-a_k(w_k-x_k)\end{array}$$

when $a_k$ takes $\frac{1}{k}$, the iterative equation is $w_{k+1}=\frac{k-1}{k}{w}_k+\frac{x_k}{k}$.

6.2 Stochastic gradient descent

​ Stochastic gradient descent (SGD) is a special Robins-Monro algorithm and the mean estimation is a special SGD algorithm.

Stochastic gradient descent (SGD) is using to solve the following optimization problem:
$$\underset{w}{\min }J(w)=\mathbb{E}[f(w,X)]$$
where $w$ is the parameter to be optimized and $X$ is a random variable. And the expectation is with respect to random variable $X$. Here $w$ and $X$ can be either scalars or vectors. The function $f(\cdot )$ is a scalar.

Gradient descent (GD):

​ We can use gradient descent to solve the above optimization problem,

$$\begin{array}{rl}{w}_{k+1}& =w_k-a_k{\nabla }_{w_k}\mathbb{E}[f(w_k,X)]\\ w_{k+1}& =w_k-a_k\mathbb{E}[{\nabla }_{w_k}f(w_k,X)]\end{array}$$

The problem of the gradient descent is that the expected value on the right-hand side is difficult to calculate. One potential way to calculate the expected value is based on the probability distribution of $X$, which is unlike to know in practice.

Batch gradient descent (BGD):

​ Inspired by the Monte Carlo learning, we can collect a large number of iid samples { x i } i = 1 n \{x_i\}_{i=1}^n {xi​}i=1n​ of $X$ so that the expected value can be approximated as
$$\mathbb{E}[{\nabla }_{w_k}f(w_k)]\approx \frac{1}{n}\sum _{i=1}^n{\nabla }_{w_k}f(w_k,x_i)$$
Then the gradient descent equaition becomes
$$w_{k+1}=w_k-a_k\frac{1}{n}\sum _{i=1}^n{\nabla }_{w_k}f(w_k,x_i)$$
which is also called batch gradient descent.

Stochastic gradient descent (SGD):

​ The problem of the BGD is that it requires all the samples in each iteration. In practice, since the samples may be collected incrementally, it is favorable to optimze $w$ instantly every time a sample is collected. We use the following algorithm,
$$w_{k+1}=w_k-a_k{\nabla }_{w_k}f(w_k,x_k)$$
where $x_k$ is the sample collected as time step $k$. The reason that this algorithm is called stochastic is that it relies on stochastic samplings { x k } \{x_k \} {xk​}.

We will show that SGD is a special form of Robins-Monro algorithm. The problem to be solved by SGD is to minimize $J(w)={\mathbb{E}}_X[f(w,X)]$. This problem can be converted to root finding problem: that is to find the root of ${\nabla }_{w}J(w)={\mathbb{E}}_X[{\nabla }_{w}f(w,X)]=0$. We can get the measurement that is ${\nabla }_{w}f(w,x)$ where $x$ is a sample of random variable $X$. Hence, we have

$$\begin{array}{rl}\tilde{g}(w,\eta )& ={\nabla }_{w}f(w,x)\\ & =\underset{g(w,\eta )}{\underbrace{{\mathbb{E}}_X[{\nabla }_{w}f(w,X)]}}+\underset{\eta (w,x)}{\underbrace{{\nabla }_{w}f(w,x)-{\mathbb{E}}_X[{\nabla }_{w}f(w,X)]}}\end{array}$$

Then, the RM algorithm for sovling $g(w)=0$ is

$$w_{k+1}=w_k-a_k\tilde{g}(w_k,{\eta }_k)=w_k-a_k{\nabla }_{w}f(w_k,x_k)$$

which is exactly the SGD algorithm.

How to solve deterministic formulation?

​ Consider the optimization problem
$$\underset{w}{\min }J(w)=\frac{1}{n}\sum _{i=1}^{n}f(w,x_i)$$
The optimization problem is deterministic, which does’t have any random variable. Although no random variables are invovled in the above formulation, we can introduce a random variable manually and convert the deterministic formulation to the stochastic formulation of SGD. Introduce a random variable $X$ following distribution $p(X=x_i)=\frac{1}{n}$. Hence, we have
$$\underset{w}{\min }J(w)=\frac{1}{n}\sum _{i=1}^{n}f(w,x_i)=\mathbb{E}[f(w,X)]$$
Then, we can use the SGD to sovle the optimization problem as
$$w_{k+1}=w_k-a_k{\nabla }_{w_k}f(w_k,x_k)$$

---

Reference

赵世钰老师的课程