lecture18
今天來討論我最喜歡的算法之一,用以控制MDP,我覺得是一個優雅、高效、強大的算法。先討論MDP的幾個常見變種(與現有的MDP定義會有不同)。
Agenda
state-action rewards
finite horizon MDPs
linear dynamical systems 線性動力系統
-models
-linear quadratic regulation(LQR) 線性二次型調節控制
just a recap:MDP is a five tuple(下圖給出一般性的定義,然后做一些改動,生成variations)
首先在reward function上做一些改動,使之不僅僅是states的函數,而且還是action的函數。記得在上一講時,我說過對於無限連續狀態的MDP,不能直接應用value iteration,因為對於continuous MDP, we use some approximations of the optimal value function. But later, we’ll talk about a special case of MDPs, where you can actually represent the value function exactly, even if you have an infinite-state space or even if you have a continuous-state space. I’ll actually do that, talk about these special constants of infinite-state MDPs, using this new variations of the reward function and the alternative to just counting, so start to make the formulation a little easier.
first variation: state-action reward 比如機器人動起來比靜止麻煩,所以reward和action也有關系了。
second variation: finite horizon MDP
圖中說到的optimal policy may be non-stationery(非平穩的),that is to say, my optimal action to take will be different for different time steps. 然后舉了一個例子(見黑板右下的方框內)
Since we have non-stationery optimal policy ,