lqr算法 c语言,【CS229 lecture18】linear quadratic regulation(LQR) 線性二次型調節控制

lecture18

今天來討論我最喜歡的算法之一，用以控制MDP，我覺得是一個優雅、高效、強大的算法。先討論MDP的幾個常見變種(與現有的MDP定義會有不同)。

Agenda

state-action rewards

finite horizon MDPs

linear dynamical systems 線性動力系統

-models

-linear quadratic regulation(LQR) 線性二次型調節控制

just a recap：MDP is a five tuple(下圖給出一般性的定義，然后做一些改動，生成variations)

首先在reward function上做一些改動，使之不僅僅是states的函數，而且還是action的函數。記得在上一講時，我說過對於無限連續狀態的MDP，不能直接應用value iteration，因為對於continuous MDP, we use some approximations of the optimal value function. But later, we’ll talk about a special case of MDPs, where you can actually represent the value function exactly, even if you have an infinite-state space or even if you have a continuous-state space. I’ll actually do that, talk about these special constants of infinite-state MDPs, using this new variations of the reward function and the alternative to just counting, so start to make the formulation a little easier.

first variation: state-action reward 比如機器人動起來比靜止麻煩，所以reward和action也有關系了。

second variation: finite horizon MDP

圖中說到的optimal policy may be non-stationery(非平穩的)，that is to say, my optimal action to take will be different for different time steps. 然后舉了一個例子(見黑板右下的方框內)

Since we have non-stationery optimal policy ，