Exercise 4.3 What are the equations analogous to (4.3), (4.4), and (4.5) for the action-value function q π q_\pi qπ and its successive approximation by a sequence of functions q 0 , q 1 , q 2 , . . . q_0, q_1, q_2, . . . q0,q1,q2,...?
According to the result of exercise 3.17, we have:
Q
π
(
s
,
a
)
=
∑
s
′
R
s
,
s
′
a
P
s
s
′
a
+
γ
∑
s
′
[
∑
a
′
Q
π
(
s
′
,
a
′
)
π
(
s
′
,
a
′
)
]
P
s
,
s
′
a
Q_\pi(s,a) = \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} Q_\pi(s',a') \pi(s',a') \bigr] P_{s,s'}^a
Qπ(s,a)=s′∑Rs,s′aPss′a+γs′∑[a′∑Qπ(s′,a′)π(s′,a′)]Ps,s′a
Let
Q
k
π
Q_k^\pi
Qkπ be the previous estimated value of
Q
π
Q_\pi
Qπ and substitute it to the right side of the equation. For the next iteration,
Q
k
+
1
π
Q_{k+1}^\pi
Qk+1π can be:
Q
k
+
1
π
(
s
,
a
)
=
∑
s
′
R
s
,
s
′
a
P
s
s
′
a
+
γ
∑
s
′
[
∑
a
′
Q
k
π
(
s
′
,
a
′
)
π
(
s
′
,
a
′
)
]
P
s
,
s
′
a
Q_{k+1}^\pi(s,a) = \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} Q_k^\pi(s',a') \pi(s',a') \bigr] P_{s,s'}^a
Qk+1π(s,a)=s′∑Rs,s′aPss′a+γs′∑[a′∑Qkπ(s′,a′)π(s′,a′)]Ps,s′a
819
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
