（3）深度强化学习基础【策略学习】

Policy-Based Reinforcement Learning

We will use a neural network to approximate the policy function. This neural network is called the policy network, which can be used to control the movement of agents.

1. Policy Function Approximation

1.1 Policy Function $\pi (a\mid s)$

• Policy function $\pi (a\mid s)$ is a probability density function (PDF).

• It takes state $s$ as input.

• It output the probabilities for all the actions, e.g.,

                                 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; $\pi (left\mid s)=0.2,$
                                 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; $\pi (right\mid s)=0.1,$
                                 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; $\pi (up\mid s)=\mathrm{0.7.}$

• The agent performs an action $a$ random drawn from the distribution.

Can we directly learn a policy function $\pi (a\mid s)?$

• If there are only a few states and actions, then yes, we can.

• Draw a table (matrix) and learn the entries.

• What if there are too many (or infinite) states or actions?

1.2 Policy Network $\pi (a\mid s;\theta )$

Policy network: Use a neural net to approximate $\pi (a\mid s)$.

• Use policy network $\pi (a\mid s;\theta )$ to approximate $\pi (a\mid s)$.

• $\theta$: trainable parameters of the neural net.

• ${\sum }_{a\in A}\pi (a\mid s;\theta )=1.$

• Here, $A$ = {“left”,“right”,“up”} is the set all actions.

• That is why we use softmax activation.

2. State-Value Function Approximation

2.1 review: Action-Value Function

2.2 State-Value Function

• $V_{\pi }(s_t)=E_A[Q_{\pi }(s_t,A)]={\sum }_a\pi (a\mid s)\cdot Q_{\pi }(s_t,a).$
               \;\;\;\;\;\;\; Integrate out action $A\backsim \pi (•\mid s_t).$

$V_{\pi }(s_t)$ can evaluate the current state. The greater the $V_{\pi }(s_t)$, the greater the chance of winning. Given the state $s$, $V_{\pi }(s_t)$ can evaluate the strategy $\pi$. If $\pi$ is good, the greater $V_{\pi }(s_t)$ is, the greater the current chance of winning.

Definition: State-value function.
$V_{\pi }(s_t)=E_A[Q_{\pi }(s_t,A)]={\sum }_a\pi (a\mid s)\cdot Q_{\pi }(s_t,a).$

Approximate state-value function.

• Approximate policy function $\pi (a\mid s_t)$ by policy network $\pi (a\mid s_t;\theta ).$

• Approximate policy function $V_{\pi }(s_t)$ by:

                                         \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; $V(s_t;\theta )={\sum }_a\pi (a\mid s;\theta )\cdot Q_{\pi }(s_t,a).$

Definition: Approximate state-value function.
• $V(s_t;\theta )={\sum }_a\pi (a\mid s;\theta )\cdot Q_{\pi }(s_t,a).$

Policy-based learning: Learn $\theta$ that maximizes $J(\theta )=E_s[V(S;\theta )].$

The better the policy network, the larger the $J(\theta )$.

How to improve $\theta$? Policy gradient ascent!

• Observe state $s$.

• Update policy by: $\theta \leftarrow \theta +\beta \cdot \frac{\partial V(s;\theta )}{\partial \theta }$

$Policygradient:\frac{\partial V(s;\theta )}{\partial \theta }$

3. Policy Gradient

Definition: Approximate state-value function.
• $V(s_t;\theta )={\sum }_a\pi (a\mid s;\theta )\cdot Q_{\pi }(s_t,a).$

Policy gradient: Derivative of $V(s;\theta )$w.r.t $\theta .$
• $\frac{\partial V(s;\theta )}{\partial \theta }=\frac{\partial {\sum }_a\pi (a\mid s;\theta )\cdot Q_{\pi }(s_t,a)}{\partial \theta }.$

Push derivative inside the summation.

$={\sum }_a\frac{\partial \pi (a\mid s;\theta )\cdot Q_{\pi }(s_t,a)}{\partial \theta }.$

Pretend $Q_{\pi }$ is independent of $\theta$.
(It may not be true.)

$={\sum }_a\frac{\partial \pi (a\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s_t,a)$ $PolicyGradient:Form1$

However, this policy gradient is not used in the actual process, but the Monte Carlo approximation of this policy gradient is actually used.

                     \;\;\;\;\;\;\;\;\;\;

Two forms of policy gradient:

• Form 1: $\frac{\partial V(s;\theta )}{\partial \theta }={\sum }_a\frac{\partial \pi (a\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s_t,a).$

• Form 2: $\frac{\partial V(s;\theta )}{\partial \theta }=E_{A\backsim \pi (•\mid s;\theta )}[\frac{\partial log\pi (A\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,A)].$

3.1 Calculate Policy Gradient for Discrete Actions

If the actions are discrete, e.g., action space $A=$ {“left”, “right”, “up”},…

Use Form1: $\frac{\partial V(s;\theta )}{\partial \theta }={\sum }_a\frac{\partial \pi (a\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s_t,a).$

1. Calculate $f(a,\theta )=\frac{\partial \pi (a\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s_t,a),$ for every action $a\in A.$
2. Policy gradient: $\frac{\partial V(s;\theta )}{\partial \theta }=f("left",\theta )+f("right",\theta )+f("up",\theta )$.

               \;\;\;\;\;\;\; If $A$ is big, this approach is costly.

3.2 Calculate Policy Gradient for Continuous Actions

If the actions are continuous, e.g., action space $A$ = [0,1],…

Use form 2: $\frac{\partial V(s;\theta )}{\partial \theta }=E_{A\backsim \pi (•\mid s;\theta )}[\frac{\partial log\pi (A\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,A)].$

1. Randomly sample an action $\hat{a}$ according to the PDF $\pi (\cdot \mid s;\theta ).$
2. Calculate $g(\hat{a},\theta )=\frac{\partial log\pi (\hat{a}\mid s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,\hat{a})$
        \;\;
3. Use $g(\hat{a},\theta )$ as an approximation to the policy gradient $\frac{\partial V(s;\theta )}{\partial \theta }$.

3.3 Update policy network using policy gradient

The Reinforce algorithm needs to play a game and observe all the rewards before updating the network.

Later…

4. Summary