RL:prat1:key_concepts_in_RL强化学习

本文链接：https://blog.csdn.net/weixin_39909130/article/details/86760674

强化学习概念

代理-环境交互

当前步奖励Reward: $r_t = R(s_t, a_t, s_{t+1})$
累计奖励 $R(\tau) = \sum_{t=0}^T r_t.$ $R(\tau) = \sum_{t=0}^{\infty} \gamma^t r_t.$

select a policy which maximizes expected return when the agent acts according to it.
首先计算给定策略 $\pi$ 特定 $\tau$ 发生的概率： $P(\tau | \pi)=\rho_0(s_0)\prod_{t=0}^{T-1}P(s_{t+1}| s_{t} ,a_{t}}) \pi(a_t|s_t)$
策略 $\pi$ 期望return: $J(\pi)=\int\limits_{\tau}P(\tau|\pi)R(\tau)=\mathop{E}\limits_{\tau\sim\pi}[R(\tau)]$
RL优化问题可以表示为： $\pi^* =arg \mathop{max}\limits_{\pi}J(\pi)$ ,最优策略；

value是知以指定状态，或状态-action为起始的期望回报 expected return ；
The On-Policy Value Function: $V^{\pi}(s) = \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s\right.}$
The On-Policy Action-Value Function: $Q^{\pi}(s,a) = \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s, a_0 = a\right.}$
The Optimal Value Function, $V^*(s)$ : $V^*(s) = \max_{\pi} \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s\right.}$
The Optimal Action-Value Function: $Q^*(s,a) = \max_{\pi} \underE{\tau \sim \pi}{R(\tau)\left| s_0 = s, a_0 = a\right.}$
关系： $V^{\pi}(s) = \underE{a\sim \pi}{Q^{\pi}(s,a)},$ $\mathop{E}\limits_{a\sim\pi}[Q^{\pi}(s,a)]=\int_{a}\pi(a_0=a|s_0=s)Q^{\pi}(s,a)$

$\begin{align*} V^{\pi}(s) &= \underE{a \sim \pi \\ s'\sim P}{r(s,a) + \gamma V^{\pi}(s')}, \\ Q^{\pi}(s,a) &= \underE{s'\sim P}{r(s,a) + \gamma \underE{a'\sim \pi}{Q^{\pi}(s',a')}}, \end{align*}$
$\max _{a_{0}}\left\{F(x_{0},a_{0})+\beta \left[\max _{\left\{a_{t}\right\}_{t=1}^{\infty }}\sum _{t=1}^{\infty }\beta ^{t-1}F(x_{t},a_{t}):a_{t}\in \Gamma (x_{t}),\;x_{t+1}=T(x_{t},a_{t}),\;\forall t\geq 1\right]\right\}$ $V(x_{0})\;=\;\max _{\left\{a_{t}\right\}_{t=0}^{\infty }}\sum _{t=0}^{\infty }\beta ^{t}F(x_{t},a_{t}),$
$\max _{a_{0}}\left\{F(x_{0},a_{0})+\beta \left[\max _{\left\{a_{t}\right\}_{t=1}^{\infty }}\sum _{t=1}^{\infty }\beta ^{t-1}F(x_{t},a_{t}):a_{t}\in \Gamma (x_{t}),\;x_{t+1}=T(x_{t},a_{t}),\;\forall t\geq 1\right]\right\}$