Exercise 4.10 What is the analog of the value iteration update (4.10) for action values,
q
k
+
1
(
s
,
a
)
q_{k+1}(s, a)
qk+1(s,a)?
Use the result of exercise 3.17:
Q
π
(
s
,
a
)
=
∑
s
′
R
s
,
s
′
a
P
s
s
′
a
+
γ
∑
s
′
[
∑
a
′
Q
π
(
s
′
,
a
′
)
π
(
s
′
,
a
′
)
]
P
s
,
s
′
a
Q_\pi(s,a) = \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} Q_\pi(s',a') \pi(s',a') \bigr] P_{s,s'}^a
Qπ(s,a)=s′∑Rs,s′aPss′a+γs′∑[a′∑Qπ(s′,a′)π(s′,a′)]Ps,s′a
easily, we have the iteration for
q
k
+
1
(
s
,
a
)
q_{k+1}(s,a)
qk+1(s,a) which is analogous to the value iteration update (4.10):
q
k
+
1
(
s
,
a
)
=
max
a
{
∑
s
′
R
s
,
s
′
a
P
s
s
′
a
+
γ
∑
s
′
[
∑
a
′
q
k
(
s
′
,
a
′
)
π
(
s
′
,
a
′
)
]
P
s
,
s
′
a
}
q_{k+1}(s, a) = \max_a \biggl \{ \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} q_k(s',a') \pi(s',a') \bigr] P_{s,s'}^a \biggr \}
qk+1(s,a)=amax{s′∑Rs,s′aPss′a+γs′∑[a′∑qk(s′,a′)π(s′,a′)]Ps,s′a}
Reinforcement Learning Exercise 4.10
最新推荐文章于 2021-06-29 17:58:34 发布
扫一扫
9653
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
