Exercise 4.4 The policy iteration algorithm on page 80 has a subtle bug in that it may never terminate if the policy continually switches between two or more policies that are equally good. This is OK for pedagogy, but not for actual use. Modify the pseudocode so that convergence is guaranteed.
1 Initialization
V
(
s
)
∈
R
and
π
(
s
)
∈
A
(
s
)
arbitrarily
s
∈
S
For each
s
∈
S
create an empty list :
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
For each
s
∈
S
create an iterator :
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
2 Policy Evaluation
Loop:
Δ
←
0
Loop for each
s
∈
S
:
v
←
V
(
s
)
V
(
s
)
←
∑
s
′
,
r
p
(
s
′
,
r
∣
s
,
π
(
s
)
)
[
r
+
γ
V
(
s
′
)
]
Δ
←
max
(
Δ
,
∣
v
−
V
(
s
)
∣
)
until
Δ
<
θ
(a small positive number determining the accuracy of estimation)
3 Policy Improvement
p
o
l
i
c
y
-
s
t
a
b
l
e
←
t
r
u
e
For each
s
∈
S
:
V
m
a
x
(
s
)
←
max
a
∑
s
′
,
r
p
(
s
′
,
r
∣
s
,
a
)
[
r
+
γ
V
(
s
′
)
]
Create an empty list:
n
e
w
_
l
i
s
t
_
o
f
_
a
(
s
)
For each
a
∈
∣
A
(
s
)
∣
:
If
∑
s
′
,
r
p
(
s
′
,
r
∣
s
,
a
)
[
r
+
γ
V
(
s
′
)
]
is equal to
V
m
a
x
(
s
)
:
Append
a
to
n
e
w
_
l
i
s
t
_
o
f
_
a
(
s
)
If
n
e
w
_
l
i
s
t
_
o
f
_
a
(
s
)
is not equal to
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
:
p
o
l
i
c
y
-
s
t
a
b
l
e
←
f
a
l
s
e
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
←
n
e
w
_
l
i
s
t
_
o
f
_
a
(
s
)
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
←
the beginning of
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
else :
If
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
is empty :
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
←
n
e
w
_
l
i
s
t
_
o
f
_
a
(
s
)
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
←
the beginning of
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
p
o
l
i
c
y
-
s
t
a
b
l
e
←
f
a
l
s
e
else :
If
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
is not equal to the end of
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
:
Move
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
to next.
p
o
l
i
c
y
-
s
t
a
b
l
e
←
f
a
l
s
e
If
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
is not equal to the end of
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
:
π
(
s
)
←
Select
a
in
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
by
i
t
e
r
a
t
o
r
_
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
else :
π
(
s
)
←
Select
a
in
o
l
d
_
l
i
s
t
_
o
f
_
a
(
s
)
randomly
If
p
o
l
i
c
y
-
s
t
a
b
l
e
=
t
r
u
e
then stop and return
V
≈
v
∗
and
π
≈
π
∗
else go to 2.
\begin{aligned} &\text{1 Initialization} \\ &\qquad V(s) \in \mathbb R \text{ and } \pi (s) \in \mathcal A(s) \text{ arbitrarily } s \in \mathcal S \\ &\qquad \text{For each } s \in \mathcal S \text{ create an empty list : } old\_list\_of\_a(s)\\ &\qquad \text{For each } s \in \mathcal S \text{ create an iterator : } iterator\_old\_list\_of\_a(s)\\ &\text{2 Policy Evaluation} \\ &\qquad \text{Loop:} \\ &\qquad \qquad \Delta \leftarrow 0 \\ &\qquad \qquad \text{Loop for each } s \in \mathcal S: \\ &\qquad \qquad \qquad v \leftarrow V(s) \\ &\qquad \qquad \qquad V(s) \leftarrow \sum_{s',r}p(s',r \mid s,\pi(s)) \Bigl [ r + \gamma V(s')\Bigr ] \\ &\qquad \qquad \qquad \Delta \leftarrow \max (\Delta , |v-V(s)|) \\ &\qquad \text{until } \Delta \lt \theta \text{ (a small positive number determining the accuracy of estimation) } \\ &\text{3 Policy Improvement} \\ &\qquad policy\text-stable \leftarrow true \\ &\qquad \text{For each }s \in \mathcal S: \\ &\qquad \qquad V_{max}(s) \leftarrow \max_a \sum_{s',r}p(s',r \mid s,a) \Bigl [ r + \gamma V(s')\Bigr ] \\ &\qquad \qquad \text{Create an empty list: }new\_list\_of\_a(s)\\ &\qquad \qquad \text{For each }a \in |\mathcal A(s)|:\\ &\qquad \qquad \qquad \text{If }\sum_{s',r}p(s',r \mid s,a) \Bigl [ r + \gamma V(s')\Bigr ] \text{ is equal to } V_{max}(s):\\ &\qquad \qquad \qquad \qquad \text{Append }a \text{ to } new\_list\_of\_a(s)\\ &\qquad \qquad \text{If }new\_list\_of\_a(s) \text{ is not equal to } old\_list\_of\_a(s)\text{ :}\\ &\qquad \qquad \qquad policy\text{-}stable \leftarrow false \\ &\qquad \qquad \qquad old\_list\_of\_a(s) \leftarrow new\_list\_of\_a(s)\\ &\qquad \qquad \qquad iterator\_old\_list\_of\_a(s) \leftarrow \text{the beginning of }old\_list\_of\_a(s)\\ &\qquad \qquad \text{else :}\\ &\qquad \qquad \qquad \text{If }old\_list\_of\_a(s) \text{ is empty :}\\ &\qquad \qquad \qquad \qquad old\_list\_of\_a(s) \leftarrow new\_list\_of\_a(s)\\ &\qquad \qquad \qquad \qquad iterator\_old\_list\_of\_a(s) \leftarrow \text{the beginning of }old\_list\_of\_a(s)\\ &\qquad \qquad \qquad \qquad policy\text{-}stable \leftarrow false\\ &\qquad \qquad \qquad\text{else :}\\ &\qquad \qquad \qquad \qquad \text{If }iterator\_old\_list\_of\_a(s) \text{ is not equal to the end of }old\_list\_of\_a(s) \text{ :}\\ &\qquad \qquad \qquad \qquad \qquad \text{Move }iterator\_old\_list\_of\_a(s) \text{ to next.}\\ &\qquad \qquad \qquad \qquad \qquad policy\text{-}stable \leftarrow false\\ &\qquad \qquad \text{If }iterator\_old\_list\_of\_a(s) \text{ is not equal to the end of }old\_list\_of\_a(s) \text{ :}\\ &\qquad \qquad \qquad \pi(s) \leftarrow \text{Select }a \text{ in }old\_list\_of\_a(s) \text{ by } iterator\_old\_list\_of\_a(s)\\ &\qquad \qquad \text{else : }\\ &\qquad \qquad \qquad \pi(s) \leftarrow \text{Select }a \text{ in }old\_list\_of\_a(s) \text{ randomly}\\ &\qquad \text{If } policy\text-stable =true \text{ then stop and return } V \approx v_* \text{ and }\pi \approx \pi_* \text{ else go to 2.} \\ \end{aligned}
1 InitializationV(s)∈R and π(s)∈A(s) arbitrarily s∈SFor each s∈S create an empty list : old_list_of_a(s)For each s∈S create an iterator : iterator_old_list_of_a(s)2 Policy EvaluationLoop:Δ←0Loop for each s∈S:v←V(s)V(s)←s′,r∑p(s′,r∣s,π(s))[r+γV(s′)]Δ←max(Δ,∣v−V(s)∣)until Δ<θ (a small positive number determining the accuracy of estimation) 3 Policy Improvementpolicy-stable←trueFor each s∈S:Vmax(s)←amaxs′,r∑p(s′,r∣s,a)[r+γV(s′)]Create an empty list: new_list_of_a(s)For each a∈∣A(s)∣:If s′,r∑p(s′,r∣s,a)[r+γV(s′)] is equal to Vmax(s):Append a to new_list_of_a(s)If new_list_of_a(s) is not equal to old_list_of_a(s) :policy-stable←falseold_list_of_a(s)←new_list_of_a(s)iterator_old_list_of_a(s)←the beginning of old_list_of_a(s)else :If old_list_of_a(s) is empty :old_list_of_a(s)←new_list_of_a(s)iterator_old_list_of_a(s)←the beginning of old_list_of_a(s)policy-stable←falseelse :If iterator_old_list_of_a(s) is not equal to the end of old_list_of_a(s) :Move iterator_old_list_of_a(s) to next.policy-stable←falseIf iterator_old_list_of_a(s) is not equal to the end of old_list_of_a(s) :π(s)←Select a in old_list_of_a(s) by iterator_old_list_of_a(s)else : π(s)←Select a in old_list_of_a(s) randomlyIf policy-stable=true then stop and return V≈v∗ and π≈π∗ else go to 2.
Reinforcement Learning Exercise 4.4
最新推荐文章于 2020-11-04 14:44:19 发布
514
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
