[RL] 3 Finite Markov Decision Processes (3)

Previous Blog

  在开始新的内容之前，我们先回顾一下前两篇博客[RL] 3 Finite Markov Decision Processes (1) 、[RL] 3 Finite Markov Decision Processes (2)给出的重要概念和式子：

• state：
    $S_t = s \in \mathcal S$
• action：
    $A_t = a \in \mathcal A(S_t)$
• reward：
    $R_t = r \in \mathcal R \subset \mathbb{R}$
• policy：
    $\pi_t(a \mid s) = Pr(A_t = a \mid S_t = s)$
• return：
    $G_t \doteq \sum_{k=0}^{T-t-1}\gamma^kR_{t+k+1}$
• markov property：
  $$p(s',r \mid s,a) \doteq Pr\{S_{t+1}=s',R_{t+1}=r \mid S_t=s,A_t=a\}\,\,\,\,\,\,\,\,\,\,\,(1.1)$$
• expected reward for state-action：
  $$r(s,a)\doteq \mathbb{E}[R_{t+1} \mid S_t=s,A_t=a] = \sum_{r \in \mathcal R}r\sum_{s' \in \mathcal S}p(s',r \mid s,a)\,\,\,\,\,\,\,\,\,\,\,\,(1.2)$$
• state-transition probability：
  $$p(s'\mid s,a)\doteq Pr\{S_{t+1}=s'\mid S_t=s,A_t=a\}=\sum_{r\in \mathcal R}p(s',r\mid s,a)\,\,\,\,\,\,\,\,\,\,\,\,\,(1.3)$$
• expected reward for state-action-nextState：
  
  r(s,a,s′)≐E[Rt+1∣St=s,At=a,St+1=s′]=∑r∈Rrp(s′,r∣s,a)p(s′∣s,a)(1.4)