蒙特卡洛方法求解强化学习任务——基于离轨策略的蒙特卡洛控制
目录
上一节介绍了离轨策略使用重要性采样方法对价值函数的求解过程,本节将介绍使用离轨策略方法求解蒙特卡洛控制过程。
回顾:基于离轨策略的蒙特卡洛策略评估
针对离轨策略行为策略 b ( a ∣ s ) b(a \mid s) b(a∣s)与目标策略 π ( a ∣ s ) \pi(a \mid s) π(a∣s)无必然联系——使用重要性采样方法将两种策略的期望关联起来。这里以加权重要性采样 为例,近似求解状态价值函数 V π ( s ) V_\pi(s) Vπ(s)结果如下:
V π ( s ) = E π [ G t ∣ S t = s ] = E b [ ρ t : T − 1 G t ∣ S t = s ] ≈ ∑ i = 1 N ρ t : T − 1 G t ( i ) ∑ i = 1 N ρ t : T − 1 \begin{aligned} V_\pi(s) & = \mathbb E_\pi[G_t \mid S_t = s] \\ & = \mathbb E_b[\rho_{t:T-1}G_t \mid S_t = s] \\ & \approx \frac{\sum_{i=1}^N \rho_{t:T-1}G_t^{(i)}}{\sum_{i=1}^N \rho_{t:T-1}} \end{aligned} Vπ(s)=Eπ[Gt∣St=s]=Eb[ρt:T−1Gt∣St=s]≈∑i=1Nρt:T−1∑i=1Nρt:T−1Gt(i)
其中:
ρ t : T − 1 = ∏ k = t T − 1 π ( A k ∣ S k ) b ( A k ∣ S k ) \rho_{t:T-1} = \prod_{k = t}^{T-1}\frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)} ρt:T−1=k=t∏T−1b(Ak∣Sk)π(Ak∣Sk)
并且在基于离轨策略的蒙特卡洛策略评估中介绍了采样的具体流程:
- 基于给定的行为策略 b ( a ∣ s ) b(a \mid s) b(a∣s)和目标策略 π ( a ∣ s ) \pi(a \mid s) π(a∣s),(对应矩阵元素相除)求出 π ( a ∣ s ) b ( a ∣ s ) \frac{\pi(a \mid s)}{b(a \mid s)} b(a∣s)π(a∣s)(也是矩阵形式);
- 针对某一完整情节(已达到终结状态 S T S_T ST),从 t t t时刻开始,根据各时刻的状态-动作二元组在 π ( a ∣ s ) b ( a ∣ s ) \frac{\pi(a \mid s)}{b(a \mid s)} b(a∣s)π(a∣s)矩阵中进行查找——求出各时刻的 π ( A k ∣ S k ) b ( A k ∣ S k ) \frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)} b(Ak∣Sk)π(Ak∣Sk)结果;
π ( A t ∣ S t ) b ( A t ∣ S t ) , π ( A t + 1 ∣ S t + 1 ) b ( A t + 1 ∣ S t + 1 ) , . . . , π ( A T − 1 ∣ S T − 1 ) b ( A T − 1 ∣ S T − 1 ) \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)},\frac{\pi(A_{t+1} \mid S_{t+1})}{b(A_{t+1} \mid S_{t+1})},...,\frac{\pi(A_{T-1} \mid S_{T-1})}{b(A_{T-1} \mid S_{T-1})} b(At∣St)π(At∣St),b(At+1∣St+1)π(At+1∣St+1),...,b(AT−1∣ST−1)π(AT−1∣ST−1) - 根据上一步骤的结果,求出重要度系数 ρ t : T − 1 \rho_{t:T-1} ρt:T−1:
ρ t : T − 1 = ∏ k = t T − 1 π ( A k ∣ S k ) b ( A k ∣ S k ) = π ( A t ∣ S t ) b ( A t ∣ S t ) ⋅ π ( A t + 1 ∣ S t + 1 ) b ( A t + 1 ∣ S t + 1 ) ⋅ . . . ⋅ π ( A T − 1 ∣ S T − 1 ) b ( A T − 1 ∣ S T − 1 ) \begin{aligned} \rho_{t:T-1} & = \prod_{k = t}^{T-1}\frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)} \\ & = \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}\cdot \frac{\pi(A_{t+1} \mid S_{t+1})}{b(A_{t+1} \mid S_{t+1})}\cdot...\cdot \frac{\pi(A_{T-1} \mid S_{T-1})}{b(A_{T-1} \mid S_{T-1})} \end{aligned} ρt:T−1=k=t∏T−1b(Ak∣Sk)π(Ak∣Sk)=b(At∣St)π(At∣St)⋅b(At+1∣St+1)π(A