Reinforcement Learning Exercise 3.18

Exercise 3.18 The value of a state depends on the values of the actions possible in that state and on how likely each action is to be taken under the current policy. We can think of this in terms of a small backup diagram rooted at the state and considering each possible action:

Give the equation corresponding to this intuition and diagram for the value at the root node, $v_{\pi }(s)$, in terms of the value at the expected leaf node, $q_{\pi }(s,a)$, given $S_t=s$. This equation should include an expectation conditioned on following the policy, $\pi$. Then give a second equation in which the expected value is written out explicitly in terms of $\pi (a|s)$ such that no expected value notation appears in the equation.
$$\begin{gathered} \begin{array}{rl}& \\ {\upsilon }_{\pi }(s)& ={\mathbb{E}}_{\pi }(G_t|S_t=s)\\ & =\sum _{a\in \mathcal{A}}{\mathbb{E}}_{\pi }(G_t|S_t=s,A_t=a)P(A_t=a|S_t=s)\end{array} \\ \begin{array}{rl}& \because P(A_t=a|S_t=s)=\pi (a|s)\\ & \therefore {\upsilon }_{\pi }(s)=\sum _{a\in \mathcal{A}}{\mathbb{E}}_{\pi }(G_t|S_t=s,A_t=a)\pi (a|s)\end{array} \end{gathered}$$
According to definition
$${\mathbb{E}}_{\pi }(G_t|S_t=s,A_t=a)=q_{\pi }(s,a)$$
so
$${\upsilon }_{\pi }(s)=\sum _{a\in \mathcal{A}}{q}_{\pi }(s,a)\pi (a|s)$$