Markov Decision processes

Non-deterministic search

the dynamics of world adds uncertainty to the outcome
导致agent 的actions nondeterministic

Markov Decision process

a model used to solve non-deterministic search
property

1. a set of states S
2. a set of actions A
3. start state
4. possibly one or more terminal states
5. a discount factor$\gamma$
6. transition function $T(s,a,s^{\prime })$: a probablity function
7. reward function: $R(s,a,s^{\prime })$ small reward at each step, and large reward in the terminal
   states(immediate reward/long term rewards)

goal:
make a sequence of actions,最大化累计reward
$U(s^0,a^0,s^1,a^1,\cdots )=R(s^0,a^0,s^1)+R(s^1,a^1,s^2)+\cdots$

$E(r^t|s_t,a_t)$
q state:
q(s,a):actions states use problability as edge
注：q state不会花费时间

objective:
maximum the sum of rewards

3. markov process: satisfies Markov property/memoriless property
   $T(S,A,S^{\prime })=P(S^{\prime }|s,a)$
   markov reward model:$R(s^t=s)=E(r^t|s^t)$
   utility:$G_t=r_t+\gamma r_{t+1}+\cdots$
   value function: $V(s)=E(G_t|s_t=s)$
   horizon: number of steps in the traiectory
   这个模型 no actions
   

finite horizon and discounting factor

为了防止每次都采取安全的一步，无限制的获取收益
finite horizon:
nonstationay policy($\pi dependsonthetimeleft$)

addictive utility：
$U(s^0,a^0,s^1,a^1,\cdots )=R(s^0,a^0,s^1)+R(s^1,a^1,s^2)+\cdots$

discounting utility:
$U(s^0,a^0,s^1,a^1,\cdots )=R(s^0,a^0,s^1)+\gamma R(s^1,a^1,s^2)+\cdots$
收敛 $u\le \frac{R_{max}}{1-\gamma }$
small $\gamma \to$ small horizon

Markovianess

Markov property or memoryless property: past and future are conditionally independent given present
$P(s_{t+1}|s_t,a_t,s_{t-1},a_{t-1},\cdots )=P(s_{t+1}|s_t,a_t)$

solving Markov Decision Process

solution: policy ${\pi }^{\ast }=\pi (s)=a$ maximize the expected utility or total reward

the bellman equation

the optimal value of a state $s$
$V^{\ast }(s)$: expected utility starts in s and action optimally

the optimal value of a q-state:
$Q^{\ast }(s,a)$,the optimal value of an agent, starts in s, acting a and acting optimally
2. bellman equation:

a type of dynamic equation:
an equation that decomposes a problem into smaller subproblems via an inherent recursive structure
bellman equation is as a condition for optimality,如果bellman方程对于所有的$v(s)$均成立，那么这些$v(s)$就是$v^{\ast }(s)$

value iteration

time-limited values:$v_k(s)$ if games end in k time steps

value iteration is a dynamic programming algorithm

each iteration complexity:$o(s^{2}a)$
一个动作可能导致所有state

convergence:

• case1 if the tree has maximum depth M,the $V_M$ holds the actual untruncated values
• case 2
  
  

policy extraction

$\forall s\in S,{\pi }^{\ast }(s)=argma{x}_a{Q}^{\ast }(s,a)=argma{x}_a{\sum }_{s^{\prime }}T(s,a,s^{\prime })V^{\ast }(s^{\prime })$
存储q值可以减少expect的计算过程

policy iteration

• define a initial policy
• policy evaluation: solve matrix $o(n^3)$or iteration o(s^2)
• policy improvement: based on value evaluation to generate a new policy
  
  dynamic based approaches
  如果我们用iteration计算v，只迭代一次就更新policy，那么和value iteration相同

asynchronous DP

• for each selected state, apply the appropriate back up
• can significantly reduce the computation
• grantee to converge if all states continue to be selected
  3 simple ideas for 异步更新
• in-place dynamic programming
• prioritized sweeping
  use magnitude of Bellman error to guide state selection
  
  每次更新后更新bellman error
  可以用优先队列来实现
• real-time dynamic programming
  only states that are relevant to agent
  use agent’s experience to guide the selection of states
  after each time-step
  

题目

一定要注意policy evaluation在自己做题的时候是解方程，不是迭代
policy evaluation到terminal state 的value评估方式，注意一下