【CS229 lecture20】策略搜索

lecture20
强化学习最后一课
Agenda
-POMDPs (partially observable MDPs)
-Policy search (the main topic for today will be policy search algorithm, specificlly I’ll talk about two algorithms named Reinforced and Pegasus)
-Reinforced
-Pegasus
-Conclusion

Recap last lecture, I actually started talk about one specific example of a POMDP which was this sort of linear dynamical system(St+1=A*St+1+B*at+wt). This is sort of LQR, linear quadratic revelation problem, but I change it and said what if we only have observations yt…

POMDP的形式化定义(in general, PODMP problem is NP-hard)和policy search(I think it is the most effective classes of reinforcement learning algorithm as well both for MDPs and for POMDPs,今天先讲将policy search algorithm应用到MDP中，也就是有完全的observations中，然后再讲怎么应用到POMDP中，但是将其应用到POMDP中时，难以保证你得到的是一个全局最优policy，因为一般来讲POMDP是NP-hard的，但我认为policy search algorithm对MDP和POMDP都是最effective的)：

---

so our first policy search algorithm——Reinforced algorithm

give one specific example to present our algorithm(倒立摆)

（下图横线以上是回答同学“当有多个actions时。。。”）

具体的求解过程

证明：

value approximation approach to find the policy 以及刚才讲到的policy search algorithm 哪个更好？
本能，条件反射式的低级别决策，比如倒立摆等很有可能存在一个logistic函数从状态映射到policy 用后者
高级别决策，比如围棋，要前后考虑，使用前者。

后者还可以应用于POMDP，尽管是partially observed states, estimated states也无妨

It turns out Reinforced algorithm is effective, but it’s noisy??

---

另一个策略搜索算法： Pegasus（我们在自主直升机飞行上使用多年了）
Pegasus 是 policy evaluation of gradient and search using scenarios的缩写。

这就是Pegasus policy search algorithm. 我们用在了直升机中，而且对于大规模问题也有很好的效果。

In closing, let me just say this class has been really fun…
Thank you ！
至此，课程到此结束……