Exercise 5.13 Show the steps to derive (5.14) from (5.12).
ρ
t
:
T
−
1
R
t
+
1
=
π
(
A
t
∣
S
t
)
b
(
A
t
∣
S
t
)
π
(
A
t
+
1
∣
S
t
+
1
)
b
(
A
t
+
1
∣
S
t
+
1
)
π
(
A
t
+
2
∣
S
t
+
2
)
b
(
A
t
+
2
∣
S
t
+
2
)
⋯
π
(
A
T
−
1
∣
S
T
−
1
)
b
(
A
T
−
1
∣
S
T
−
1
)
R
t
+
1
(
5.12
)
\rho_{t:T-1}R_{t+1} = \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}\frac{\pi(A_{t+1} \mid S_{t + 1})}{b(A_{t+1} \mid S_{t + 1})}\frac{\pi(A_{t+2} \mid S_{t + 2})}{b(A_{t+2} \mid S_{t + 2})}\cdots\frac{\pi(A_{T-1} \mid S_{T - 1})}{b(A_{T-1} \mid S_{T - 1})}R_{t+1} \qquad{(5.12)}
ρt:T−1Rt+1=b(At∣St)π(At∣St)b(At+1∣St+1)π(At+1∣St+1)b(At+2∣St+2)π(At+2∣St+2)⋯b(AT−1∣ST−1)π(AT−1∣ST−1)Rt+1(5.12)
According to the property of Markov Procedure, each action step is independent, so we have
E
(
ρ
t
:
T
−
1
R
t
+
1
)
=
E
(
π
(
A
t
∣
S
t
)
b
(
A
t
∣
S
t
)
R
t
+
1
)
E
(
π
(
A
t
+
1
∣
S
t
+
1
)
b
(
A
t
+
1
∣
S
t
+
1
)
)
E
(
π
(
A
t
+
2
∣
S
t
+
2
)
b
(
A
t
+
2
∣
S
t
+
2
)
)
⋯
E
(
π
(
A
T
−
1
∣
S
T
−
1
)
b
(
A
T
−
1
∣
S
T
−
1
)
)
\mathbb E(\rho_{t:T-1}R_{t+1})=\mathbb E( \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}R_{t+1})\mathbb E(\frac{\pi(A_{t+1} \mid S_{t + 1})}{b(A_{t+1} \mid S_{t + 1})})\mathbb E(\frac{\pi(A_{t+2} \mid S_{t + 2})}{b(A_{t+2} \mid S_{t + 2})})\cdots\mathbb E(\frac{\pi(A_{T-1} \mid S_{T - 1})}{b(A_{T-1} \mid S_{T - 1})})
E(ρt:T−1Rt+1)=E(b(At∣St)π(At∣St)Rt+1)E(b(At+1∣St+1)π(At+1∣St+1))E(b(At+2∣St+2)π(At+2∣St+2))⋯E(b(AT−1∣ST−1)π(AT−1∣ST−1))
And according to equation (5.13), the expectation of each step is 1, except the first.
E
[
π
(
A
k
∣
S
k
)
b
(
A
k
∣
S
k
)
]
≐
∑
a
b
(
a
∣
S
k
)
π
(
a
∣
S
k
)
b
(
a
∣
S
k
)
=
∑
a
π
(
a
∣
S
k
)
=
1
(
5.13
)
\mathbb E\Biggl[ \frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)}\Biggr] \doteq \sum_a b(a\mid S_k) \frac{\pi(a \mid S_k)}{b(a \mid S_k)} = \sum_a \pi(a \mid S_k) = 1 \qquad{(5.13)}
E[b(Ak∣Sk)π(Ak∣Sk)]≐a∑b(a∣Sk)b(a∣Sk)π(a∣Sk)=a∑π(a∣Sk)=1(5.13)
So,
E
(
ρ
t
:
T
−
1
R
t
+
1
)
=
E
(
π
(
A
t
∣
S
t
)
b
(
A
t
∣
S
t
)
R
t
+
1
)
=
E
(
ρ
t
:
t
R
t
+
1
)
(
5.14
)
\begin{aligned} \mathbb E(\rho_{t:T-1}R_{t+1})&=\mathbb E( \frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}R_{t+1})\\ &=\mathbb E(\rho_{t:t}R_{t+1}) \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad{(5.14)} \end{aligned}
E(ρt:T−1Rt+1)=E(b(At∣St)π(At∣St)Rt+1)=E(ρt:tRt+1)(5.14)
Reinforcement Learning Exercise 5.13
最新推荐文章于 2019-10-04 14:21:04 发布
1123
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
