Exercise 6.3 From the results shown in the left graph of the random walk example it appears that the first episode results in a change in only V ( A ) V (A) V(A). What does this tell you about what happened on the first episode? Why was only the estimate for this one state changed? By exactly how much was it changed?
As shown in the left graph of the random walk example, all the state values are initialized to 0.5, so the intermediate steps will not change the state values in the first episode. For example, from B to A, there is:
V
(
B
)
←
V
(
B
)
+
α
[
0
+
γ
V
(
A
)
−
V
(
B
)
]
∴
V
(
B
)
=
0.5
+
0.1
[
0
+
1
⋅
0.5
−
0.5
]
=
0.5
V(B) \leftarrow V(B) + \alpha[ 0 + \gamma V(A) - V(B)] \\ \therefore V(B) = 0.5 +0.1[0 +1\cdot 0.5 -0.5]=0.5
V(B)←V(B)+α[0+γV(A)−V(B)]∴V(B)=0.5+0.1[0+1⋅0.5−0.5]=0.5
Also, there is only state A changed in episode 1 shown in the left graph of the random walk example. So, the episode 1 terminate in left terminal of A, otherwise, state E should have been changed. Then, we can calculate how much the value of state A changed:
V
(
A
)
←
V
(
A
)
+
α
[
0
+
γ
V
(
t
e
r
m
i
n
a
l
)
−
V
(
A
)
]
∴
V
(
A
)
=
0.5
+
0.1
[
0
+
1
⋅
0
−
0.5
]
=
0.45
V(A) \leftarrow V(A) + \alpha[ 0 + \gamma V(terminal) - V(A)] \\ \therefore V(A) = 0.5 +0.1[0 +1\cdot 0 -0.5]=0.45
V(A)←V(A)+α[0+γV(terminal)−V(A)]∴V(A)=0.5+0.1[0+1⋅0−0.5]=0.45
The changed amount is 0.05.
558
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
