Reinforcement Learning Exercise 5.6

Exercise 5.6 What is the equation analogous to (5.6) for action values $Q(s,a)$ instead ofstate values $V(s)$, again given returns generated using $b$?

Given a starting state $S_t$, starting action $A_t$, the probability of the subsequent state-action trajectory, $S_{t+1},A_{t+1},\cdots ,S_T$ occurring under any policy $\pi$ is
P r ( S t + 1 , A t + 1 , ⋯   , S T − 1 , A T − 1 , S T ∣ S t , A t : T − 1 ∼ π ) = p ( S t + 1 ∣ S t , A t ) π ( A t + 1 ∣ S t + 1 ) ⋯ p ( S T − 1 ∣ S T − 2 , A T − 2 ) π ( A T − 1 ∣ S T − 1 ) p ( S T ∣ S T − 1 , A T − 1 ) = ∏ k = t T − 1 π ( A k ∣ S k ) p ( S k + 1 ∣ S k , A k ) π ( A t ∣ S t ) \begin{aligned} &Pr(S_{t+1}, A_{t+1},\cdots, S_{T-1}, A_{T-1}, S_T \mid S_t, A_{t:T-1}\sim \pi)\\ &\qquad = p(S_{t+1} \mid S_t, A_t) \pi(A_{t+1}|S_{t+1}) \cdots p(S_{T-1} \mid S_{T-2}, A_{T-2})\pi(A_{T-1} \mid S_{T-1}) p(S_T \mid S_{T-1}, A_{T-1})\\ &\qquad =\frac {\prod_{k=t}^{T - 1} \pi(A_k \mid S_k)p(S_{k+1}\mid S_k, A_k)} {\pi(A_t \mid S_t)} \end{aligned} ​Pr(St+1​,At+1​,⋯,ST−1​,AT−1​,ST​∣St​,At:T−1​∼π)=p(S