Exercise 3.12 Give an equation for v π v_\pi vπ in terms of q π q_\pi qπ and π \pi π.
v
π
(
s
)
=
E
π
(
G
t
∣
S
t
=
s
)
=
∑
g
t
[
g
t
⋅
p
(
g
t
∣
s
)
]
=
∑
g
t
[
g
t
⋅
p
(
g
t
,
s
)
p
(
s
)
]
=
∑
g
t
[
g
t
⋅
∑
a
∈
A
p
(
g
t
,
s
,
a
)
p
(
s
)
]
=
∑
g
t
{
g
t
⋅
∑
a
∈
A
[
p
(
g
t
∣
s
,
a
)
⋅
p
(
s
,
a
)
]
p
(
s
)
}
=
∑
g
t
{
g
t
⋅
∑
a
∈
A
[
p
(
g
t
∣
s
,
a
)
⋅
p
(
a
∣
s
)
⋅
p
(
s
)
]
p
(
s
)
]
}
=
∑
g
t
{
g
t
⋅
∑
a
∈
A
[
p
(
g
t
∣
s
,
a
)
⋅
p
(
a
∣
s
)
]
}
=
∑
a
∈
A
{
p
(
a
∣
s
)
∑
g
t
[
g
t
⋅
p
(
g
t
∣
s
,
a
)
]
}
\begin{aligned} v_\pi(s) &= \mathbb E_\pi(G_t|S_t=s) \\ &=\sum_{g_t}\bigl [ g_t \cdot p(g_t|s) \bigr ] \\ &=\sum_{g_t}\bigl [ g_t \cdot \frac {p(g_t, s)}{p(s)} \bigr ] \\ &=\sum_{g_t}\bigl [ g_t \cdot \frac{ \sum_{a \in \mathcal A} p(g_t, s, a)}{p(s)} \bigr ] \\ &=\sum_{g_t}\Bigl \{ g_t \cdot \frac{ \sum_{a \in \mathcal A} \bigl [p(g_t| s, a) \cdot p(s, a) \bigr ] }{p(s)} \Bigr \} \\ &=\sum_{g_t}\Bigl \{ g_t \cdot \frac{ \sum_{a \in \mathcal A} \bigl [p(g_t| s, a) \cdot p(a | s) \cdot p(s) \bigr ]}{p(s) \bigr ] } \Bigr \} \\ &=\sum_{g_t}\Bigl \{ g_t \cdot \sum_{a \in \mathcal A} \bigl [p(g_t| s, a) \cdot p(a | s) \bigr ] \Bigr \} \\ &=\sum_{a \in \mathcal A} \Bigl \{ p(a|s) \sum_{g_t} \bigl [ g_t \cdot p(g_t | s, a) \bigr ] \Bigr \} \end{aligned}
vπ(s)=Eπ(Gt∣St=s)=gt∑[gt⋅p(gt∣s)]=gt∑[gt⋅p(s)p(gt,s)]=gt∑[gt⋅p(s)∑a∈Ap(gt,s,a)]=gt∑{gt⋅p(s)∑a∈A[p(gt∣s,a)⋅p(s,a)]}=gt∑{gt⋅p(s)]∑a∈A[p(gt∣s,a)⋅p(a∣s)⋅p(s)]}=gt∑{gt⋅a∈A∑[p(gt∣s,a)⋅p(a∣s)]}=a∈A∑{p(a∣s)gt∑[gt⋅p(gt∣s,a)]}
According to definition,
p
(
a
∣
s
)
=
π
(
a
∣
s
)
p(a|s) = \pi(a|s)
p(a∣s)=π(a∣s),
∑
g
t
[
g
t
⋅
p
(
g
t
∣
s
,
a
)
]
=
q
π
(
s
,
a
)
\sum_{g_t} \bigl [ g_t \cdot p(g_t | s, a) \bigr ] = q_\pi(s,a)
∑gt[gt⋅p(gt∣s,a)]=qπ(s,a), so there is:
v
π
(
s
)
=
∑
a
∈
A
[
π
(
a
∣
s
)
⋅
q
π
(
s
,
a
)
]
v_\pi(s) = \sum_{a \in \mathcal A} \bigl [ \pi(a|s) \cdot q_\pi(s,a) \bigr ]
vπ(s)=a∈A∑[π(a∣s)⋅qπ(s,a)]
1118
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
