CS234 RL Winter

lecture 1

---

Intro

• $r$ Reward function, can be a function of $r(s) $or $r(s,a)$
• $s$, state
• $a$, action
• $h$: history
• $\gamma$: discount factor
  • mathematically convenient. $\gamma =0$: only care about immediate reward
• ${\pi }^{\ast }$: optimal policy
• $V_k^{\pi }(s)$: state value funciton at step $k$ with policy $\pi$ with state $s$
• $Q^{\pi }(s,a)$: State-Action Value Q: Take action a, then follow the policy $\pi$
• model: how world change, given $s_t$ and $a_t$, stocastic or deterministic
• model-free: you don’t know how world change, e.x two player game
• policy: how you react given a state $s$
• value:
• expoloration try new things that might be better in the future
• Expliotation: choose actions that are expected to yield good rewared given past expereince
• Horizon: the number of actions you can take to reach termination, could be finite or infinite
• Episode: series of actions from start to end
• $G_t$ discuonted sum of rewards$G_t=r_t+\gamma r+t+1+{\gamma }^2{r}_{t+2}...$
• $V(s)$ State Value Function, expected return form starting in state s $V(s)=E[G_t|s_t=s]$
• $O$: contraction operator. $|OV-O{V}^{\prime }|\le |V-V^{\prime }|$
  lecture 2

---

Markov (Decision / reward) process

• Bandit: single state MDP: only care about current state

State $s_t$ is Markov iff
$$p(s_{t+1}|s_t,a_t)=p(s_{t+1}|h_t,a_t)$$

Markov Chain: each state has a fix distribution of the next state
Markov Reward process

• Markov chain + reward
• In a n step episode, MRP value function satisifies
  $$V(s)=\underset{\text{Immediate reward}}{\underbrace{R(S)}}+\underset{\text{Discounted sum of future rewards}}{\underbrace{\gamma \sum _{s^{\prime }\in S}p(s^{\prime }|s)V(s^{\prime })}}$$
  

Iterative algorithm for computing value of a MRP

Markov Decision Process

MRP + actions
P is transition model for each action, so taken a state and a action, $$P(s_{t+1}s=s^{\prime }|s_t=s,a_t=a)$$
The next state is usually not deterministic given a state and a action

Quiz

Suppose you hvae 7 discrete states and 2 actions, how many deterministic policies are there?

2^7

Is the optimal policy for a MDP always unique?

no

Policy Search

A: action, S: state, try all $A^S$ possibilities

Policy Iteration

• Set i = 0
• Initialize π0(s) randomly for all states s
• While i == 0 or kπi − πi−1k1 > 0 (L1-norm, measures if the policy
  changed for any state):
  $V_i^{\pi }$ ← MDP V function policy evaluation of πi
  ${\pi }_{i+1}$ ← Policy improvement
  i = i + 1

Q value

State-action value of a policy
$$Q^{\pi }(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\pi }(s^{\prime })$$

Policy Improvement

compute new poliy ${\pi }_{i+1}$, for all $s\in S$
${\pi }_{i+1}(s)=\underset{a}{\mathrm{argmax}}{Q}^{{\pi }_i}(s,a)\forall s\in S$
note: so this is update policy for all states, not just one state?

Policy Iteration quiz

Bellman backup operator

$BV(s)=\underset{a}{\max }R(s,a)+\gamma {\sum }_{s^{\prime }\in S}p(s^{\prime }|a)V(s^{\prime })$

• BV yields a value function overal all state s

Value Iteration

Bellman Backup is a contraction operator

对于$V(s),V^{\prime }(s)$, 如果两个目前都去取最优的action，a，那么自然而然两者的差距会缩小。

Policy Evaluation with dp

$V^{\pi }(s)\approx {\mathbb{E}}_{\pi }[r_t+\gamma V_{k-1}|s_t=s]$

lecture3

Monte carlo policy evaluatoin

No model

• (s,a) is not deterministic
• don’t assume Markov
• Require episode to terminate
• If trajectories are all finite, sample set of trajectories & average returns

First-visit Monte Carlo vs Every-visit Monte carlo policy evaluation

first time: only account for reward when you first reach the state in an episode
everytime: acount for all, biased estimator

This is to calculate the V of s

Temporal Difference Learning for Estimating V (TD learning)

Update immediately rather than wait until the end of episode


lecture4

Model-free control

• Model is unknown but can be sampled
• Model is known but computttionally infeasible to conpute

On/Off policy

• On policy
  • Learn from following that policy
• Off policy
  • learn form following different policy

MC for on policy Q

SARSA algorithm (TD)

1. Set initial $ϵ$-greedy policy $\pi$ randomly, t = 0, initial state $s_t=s_0$
2. Take $a_t$ ~ $\pi (s_t)$ // Sample from policy
3. Observe$(r_t,s_{t+1})$
4. loop
   5. Take action $a_{t+1}\sim \pi (S_{t+1})$
   6. Observe$(r_{t+1},s_{t+2})$
   7. Update Q given $(s_t,a_t,r_t,s_{t+1},a_{t+1})$
      8. $Q(s_t,a_t)\leftarrow Q(s_t,a_t)+(1-\alpha )(r_t+\gamma Q(s_{t+1},a_{t+1})-Q(s_t,a_t)$
   8. $\pi (s_t)=\underset{a}{\mathrm{argmax}}Q(s_t,a)$ w. prob 1 - $ϵ$, else random
   9. t = t+1

Q learning

If you want to take bad action early stages, and gain more later.

GLIE

You can’t satisfy GLIE for example the case helicopter. If you break helicopter you can’t go make and make another decision.

Q-learning

$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+(1-\alpha )(r_t+\gamma {\max }_{a^{\prime }}Q(s_{t+1},a^{\prime })-Q(s_t,a_t)$

注意这里的a是所有的action选最好的，NARSA是选目前policy的