reinforcement learning

1. Model Free

1.1 Monte Carlo

1.1.1 Value Iteration

SARSA 1. current Q -> e-greedy policy
2. sample trajectorys (s1,a1,r1,s2,a2,r2 …), first visit MC
3. update $$Q(s,a)=\frac{1}{N(s,a)}\sum _i{G}_i^t(s,a)$$
4. Inprove the policy based uppdated Q value

1.1.2 Policy interation

1.1.2 Policy Gradient

Essential formula in episodic setting
the objective is to maximize, expectation over the trajectory:
$$J=E_{\tau \sim \pi }[r(\tau )\sum _{t=0}^{t=T}\log (\pi (a_t|s_t)]$$

Essential formula in non-episodic setting
$$\begin{array}{rlr}{\nabla }_{\theta }J(\theta )& ={\nabla }_{\theta }{E}_{s\sim d^{\pi }}[V(s)]& \\ & ={\nabla }_{\theta }\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{a\in \mathcal{A}}{Q}^{\pi }(s,a){\pi }_{\theta }(a|s)& \\ & \propto \sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{a\in \mathcal{A}}{Q}^{\pi }(s,a){\nabla }_{\theta }{\pi }_{\theta }(a|s)& \\ & =\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{a\in \mathcal{A}}{\pi }_{\theta }(a|s)Q^{\pi }(s,a)\frac{{\nabla }_{\theta }{\pi }_{\theta }(a|s)}{{\pi }_{\theta }(a|s)}& \\ & ={\mathbb{E}}_{\pi }[Q^{\pi }(s,a){\nabla }_{\theta }\ln {\pi }_{\theta }(a|s)]& \text{; Because }(\ln x{)}^{\prime }=1/x\end{array}$$
Where ${\mathbb{E}}_{\pi }$ refers to ${\mathbb{E}}_{s\sim d_{\pi },a\sim {\pi }_{\theta }}$ when both state and action distributions follow the policy πθ (on policy)

REINFORCEMENT

1.2 TD

1.2.1 Value Iteration

can be done in non-episode environment)

SARSA

1. None episode setting, need tuple$(s_t,a_t,r_t,a_{t+1})$
   2. update $$Q(s_t,a_t)=Q(s,a)+\alpha (r_t+rQ(s_{t+1},a_{t+1})-Q(s_t,a_t))$$
   3. Inprove the policy based updated Q value
   

Q learning off policy learning
1. None episode setting, need tuple $(s_t,a_t,r_t)$
2. update $$Q(s_t,a_t)=Q(s,a)+\alpha (r_t+r\underset{a^{\prime }}{\max }Q(s_{t+1},a^{\prime })-Q(s_t,a_t))$$

DQN

1.2.2 Policy Gradient

link

Actor-Critic (using Q)

Advantage Actor Critic (using V)

2. Model Based

2.1 Monte Carlo Tree Search

four step: 1. selection, expansion, simulation, backpropagation
in selection, must select most promissing actions. we can use Q value to select the good try steps.
in expansion, random expand the leaf selected node into valid action.
in simulation, roll out this action quickly into terminal by using some imimated learned policy
in backprogagation, evaluate the action with the most expected rewords from all simulated path ( max-expection tree)

2.1 Forward Search Tree

at state, try different actions, and go to different states and then try the actions at that states. then using expecting-max tree

The stanford class notes:
https://github.com/Zhenye-Na/reinforcement-learning-stanford

Bellman Optimality Equation

$$V_{{\pi }^{\ast }}(s)=\max Q(s,a)$$
$${\pi }^{\ast }(s)=\mathrm{argmax}Q(s,a)$$
$$V_{\pi }(s)=\sum _{a\sim \pi (a|s)}Q(s,a)$$
$$Q_{\pi }(s,a)=r(s,a)+\sum _{s^{\prime }\sim p(s^{\prime }|a)}{V}_{\pi }(s^{\prime })=r(s,a)+\sum _{s^{\prime }\sim p(s^{\prime }|a)}\sum _{a^{\prime }\sim \pi (a^{\prime }|s^{\prime })}Q(s^{\prime },a^{\prime })$$
$$Q_{\pi \ast }(s,a)=r(s,a)+\sum _{s^{\prime }\sim p(s^{\prime }|a)}{V}_{\pi \ast }(s^{\prime })=r(s,a)+\sum _{s^{\prime }\sim p(s^{\prime }|a)}\underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })$$

Alpha go

The detail of MCTS,

1。selection phase, we search from the current node and select the action with the max( Q + CUB ),
CUB = P(s,a)/(1+N(s,a))

注意，我们只走到 边 （edge（s，a），并不是node （state），
其实第二步不是expansion， 而是simulation！！！
2. in simulation, we take two perspective, 1）fast roll out value and 2）current value function V( s) ,
3. backup
once any of them is done, back up all the edges in the current policy tree,
V_total += V_i, Z_total_sim(s,a) += Z_i, also the counting N_z(s,a), N_q(s,a).
The new Q = 0.5*V/N_v + 0.5Z/N_z
4. expansion,
if one edge is visited for more than N_ts times, this edge is expanded to be a node (initialized by N=0) and add it to the current policy tree, which means, so when in the next search phase, the action under this node will be searched by max( Q + CUB ), note, all the initial Q(s,a) and N(s,a) under new initiated nodes is zero, sp the first select under new node will be follow argmax( p(s,a) ) from policy network, because,
max( Q + CUB ) = max(CUB) = max( P(s,a) ),
CUB = P(s,a)/(1+N(s,a))