马尔科夫决策过程(MDP) : BlackJack (MC-Off Policy)

BlackJack:

    问题仍以Black Jack为例，通过off-policy的方法进行值函数的估计。

问题抽象：

• $s$ : 状态(闲);Ace,玩家(闲)的牌面，庄家的明牌牌面。
• $a$ : 动作;要牌(hit:1),停牌(stick:0)。
• $r$ : 奖励;[-1,0, 1],输，平，赢。
• $\gamma =1$

On-policy 和 Off-policy

• On-policy:只有一个策略，episode数据的生成和值函数的估计都是基于这个策略。
• Off-policy:使用两个策略，target policy(目标)和 behavior policy(行为),两个策略各有分工。
  • target policy : 学习获得一个最优策略。
  • behavior policy : 探索环境，生成行为(episode)数据。
  • target policy $\ne$ behavior policy,利用behavior policy的数据来估计target policy的值函数。
  • importance sampling : 重要性采样系数$\rho$，即:把behavior policy下的动作值期望转换为target policy的动作值期望。$V_{\pi }(s)\leftarrow \rho V_b(s)$

重要性采样系数(importance sampling ratio)

1. state-action序列的发生概率。

P r { A t , S t + 1 , A t + 1 , . . . , S T ∣ S t , A t : T − 1 ∼ π } = π ( A t ∣ S t ) p ( S t + 1 ∣ S t , A t ) π ( A t + 1 , S t + 1 ) . . . p ( S T ∣ S T − 1 , A T − 1 ) = ∏ k = t T − 1 π ( A k ∣ S k ) p ( S k + 1 ∣ S k , A k ) \begin{aligned}Pr\{A_t,S_{t+1},A_{t+1},...,S_{T}|S_t,A_{t:T-1} \sim \pi\}&=\pi(A_t|S_t)p(S_{t+1}|S_t,A_t)\pi(A_{t+1},S_{t+1})...p(S_T|S_{T-1},A_{T-1})\\ &= \prod_{k=t}^{T-1} \pi(A_k|S_k)p(S_{k+1}|S_k,A_k) \end{aligned} Pr{At​,St+1​,At+1​,...,ST​∣St​,At:T−1​∼π}​=π(At​∣St​)p(St+1​∣St​,At​)π(At+1​,St+1​)...p(ST​∣ST−1​,AT−1​)=k=t∏T−1​π(Ak​∣Sk​)p(Sk+1​∣Sk​,Ak​)​

• $S_t$ : 起始状态
• $\pi (A_t|S_t)$ : 动作概率
• $p(S_{t+1}|S_t,A_t)$ : 状态转移概率

2. 重要性采样系数:两个策略的s-a序列概率的比值。

$${\rho }_{t:T-1}=\frac{{\prod }_{k=t}^{T-1}\pi (A_k|S_k)\pi (S_{k+1}|S_k,A_k)}{{\prod }_{k=t}^{T-1}b(A_k|S_k)\pi (S_{k+1}|S_k,A_k)}=\prod _{k=t}^{T-1}\frac{\pi (A_k|S_k)}{b(A_k|S_k)}$$

• $\pi$ : 目标策略(i.e.greedy policy)
• $b$ : 行为策略(i.e.$ϵ$-greedy policy)

3. 行为值函数$\to$策略值函数

$$v_b(s)=E[G_t|S_t=s]$$
$$v_{\pi }(s)=E[{\rho }_{t:T-1}{G}_t|S_t=s]$$

模拟游戏过程

记录状态、动作、奖励。

import warnings
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from collections import namedtuple
from tqdm.notebook import tqdm

warnings.filterwarnings('ignore')

# dealer policy
def dealer_policy(cards_num):
    if cards_num < 17:
        return 1
    else:
        return 0

def play_blackjack(policy_player, policy_dealer, initial_state=None, initial_action=None):
    '''
    policy_player : state(usable_ace,player_sum,dealer_card)->action
    policy_dealer : dealer_sum
    return -> reward, trajectory
    '''
    def card_value(card):
        return 11 if card == 1 else card
    # 闲
    player_sum = 0
    # 庄
    dealer_card1 = 0
    dealer_card2 = 0
    # trajectory
    player_trajectory = []
    player_transition = namedtuple('Transition', ['state', 'action'])
    # False : Ace = 1, True Ace = 11
    usable_ace_player = False
    usable_ace_dealer = False
    if initial_state is None:
        while player_sum < 12:
            # 点数小于12，一直拿牌
            card = min(np.random.randint(1, 14), 10)
            #print(card)
            # 小于12，Ace = 11
            player_sum += card_value(card)
            # 点数超过21
            if player_sum > 21:
                # Ace = 1
                player_sum -= 10
            else:
                usable_ace_player |= (1 == card)
        # 初始化庄家牌，第一张为明牌
        dealer_card1 = min(np.random.randint(1, 14), 10)
        dealer_card2 = min(np.random.randint(1, 14), 10)
    else:
        # 指定初始状态
        usable_ace_player, player_sum, dealer_card1 = initial_state
        dealer_card2 = min(np.random.randint(1, 14), 10)
    
    dealer_sum = card_value(dealer_card1) + card_value(dealer_card2)
    usable_ace_dealer = 1 in (dealer_card1, dealer_card2)
    if dealer_sum > 21:
        # use Ace = 1
        dealer_sum -= 10
    # 闲先
    while True:
        if initial_action is not None:
            player_action = initial_action
            initial_action = None
        else:
            player_action = policy_player(usable_ace_player, player_sum, dealer_card1)
        # 状态，动作
        player_sa = player_transition((usable_ace_player, player_sum, dealer_card1), player_action)
        player_trajectory.append(player_sa)
        if player_action == 0:
            break
        # 拿牌，默认Ace = 11
        card = min(np.random.randint(1, 14), 10)
        #print(card)
        # Keep track of the ace count
        ace_count = int(usable_ace_player)
        if card == 1:
            ace_count += 1
        player_sum += card_value(card)
        # 避免bust ,Ace = 1
        while player_sum > 21 and ace_count:
            player_sum -= 10
            ace_count -= 1
        if player_sum > 21:
            return -1 , player_trajectory
        usable_ace_player = (ace_count == 1)
    # 庄
    while True:
        dealer_action = policy_dealer(dealer_sum)
        if dealer_action == 0:
            break
        # 拿牌，默认Ace = 11
        new_card = min(np.random.randint(1, 14), 10)
        #print(card)
        ace_count = int(usable_ace_dealer)
        if new_card == 1:
            ace_count += 1
        dealer_sum += card_value(new_card)
        # 避免bust,Ace = 1
        while dealer_sum > 21 and ace_count:
            dealer_sum -= 10
            ace_count -= 1
        if dealer_sum > 21:
            return 1 , player_trajectory
        usable_ace_dealer = (ace_count == 1)
    if player_sum > dealer_sum:
        return 1 , player_trajectory
    elif player_sum == dealer_sum:
        return 0 , player_trajectory
    else:
        return -1 , player_trajectory

动作值函数估计$q(s,a)$

四种不同重要性采样：

1. Ordinary importance sampling
2. Weighted importance sampling
3. Discounting-aware Importance Sampling(略)
4. Per-decision Importance Sampling(略)

1.Ordinary importance sampling(普通重要性采样)

$$Q(s,a)=\frac{{\sum }_{t\in \mathcal{J}(s,a)}{\rho }_{t:T(t)-1}{G}_t}{|\mathcal{J}(s,a)|}$$

• $\mathcal{J}(s,a)$ : 状态动作pair$(s,a)$被访问的time step集合
  • every-visit:$(s,a)$每次出现的time step
  • first-visit:$(s,a)$，第一次出现time step
• $|\mathcal{J}(s,a)|$ : 状态$s$被访问的次数
  • every-visit:$(s,a)$,出现的次数
  • first-visit:1
• $G_t$ : 状态$(s,a)$,time stpe $t$ 的$Return$

MC-off policy(Ordinary importance sampling)

• Initialize,for all $s\in \mathcal{S},a\in \mathcal{A}(s)$:
  • $Q(s,a)\in \mathbb{R}(arbitrarily)$
  • $\mathcal{C}(s,a)\leftarrow 0$
  • $\pi (s)\leftarrow \underset{a}{\mathrm{argmax}}Q(s,a)$
• Loop,for each episode:
  • $b\leftarrow$ soft policy
  • Generate an episode use $b:S_0,A_0,R_1,...,S_{T-1},A_{T-1},R_T$
  • $G\leftarrow 0$
  • $W\leftarrow 1$
  • Loop, for each step of episode,$t=T-1,T-2,...,0$:
    • $G\leftarrow \gamma G+R_{t+1}$
    • $\pi (S_t)\leftarrow \underset{a}{\mathrm{argmax}}Q(S_t,a)$
    • $\mathcal{C}(S_t,A_t)\leftarrow \mathcal{C}(S_t,A_t)+1$
    • $Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\frac{WG-Q(S_t,A_t)}{\mathcal{C}(S_t,A_t)}$
    • if $A_t\ne \pi (S_t)$, exit inner Loop (proceed to next episode)
    • $W\leftarrow W\frac{1}{b(A_t|S_t)}$

---

• π ( a ∣ s ) = 1 , { a = a r g m a x a Q ( s , a ) } \pi(a|s) = 1,\{a = \underset{a}{argmax}Q(s,a)\} π(a∣s)=1,{a=aargmax​Q(s,a)}
• $\frac{\pi (a|s)}{b(a|s)}=\frac{1}{b(a|s)}$
• $\mathcal{C}(S_t,A_t)$ : $|\mathcal{J}(S_t,A_t)|$,(s,a)累计次数和

def monte_carlo_off_policy(episodes, gamma=1.0, epsilon=0.1, threshold=0.0001):
    # nearly greedy policy ：behavior
    def soft_policy(usable_ace, player_sum, dealer_card, epsilon=epsilon):
        usable_ace = int(usable_ace)
        player_sum -= 12
        dealer_card -= 1
        values_ = state_action_values[player_sum, dealer_card, usable_ace, :]
        proba = np.random.uniform(0, 1)
        if proba <= epsilon:
            action = np.random.randint(0, 2)
        else:
            action = np.random.choice([action_ for action_, value_ in enumerate(values_) if value_ == np.max(values_)])
        return action

    # greedy policy ：target
    def greedy_policy(usable_ace, player_sum, dealer_card):
        usable_ace = int(usable_ace)
        player_sum -= 12
        dealer_card -= 1
        values_ = state_action_values[player_sum, dealer_card, usable_ace, :]
        action = np.random.choice([action_ for action_, value_ in enumerate(values_) if value_ == np.max(values_)])
        return action
    
    # random policy : behavior
    def random_policy(usable_ace, player_sum, dealer_card):
        action = np.random.randint(0, 2)
        return action
    # Initialize
    state_action_values = np.zeros((10, 10, 2, 2))
    state_action_pair_count = np.ones((10, 10, 2, 2))
    # Loop for each episode
    delta_history = []
    for episode in tqdm(range(episodes)):
        old_sa = state_action_values.copy()
        # Generate an episode
        player_reward, player_traj = play_blackjack(soft_policy, dealer_policy)
        player_states = [t.state for t in player_traj]
        player_actions = [t.action for t in player_traj]
        player_rewards = [0]*len(player_states)
        player_rewards[-1] = player_reward
        # State,Action,Return
        R = 0
        Gs = []
        for r in player_rewards[::-1]:
            R = r + gamma * R
            Gs.insert(0, R)
        # Loop ::-1
        proba_b_a = 1.0
        for player_state, action, G in zip(player_states[::-1], player_actions[::-1], Gs[::-1]):
            usable_ace_player, player_sum, dealer_card = player_state
            target_action = greedy_policy(usable_ace_player, player_sum, dealer_card)  # target policy
            usable_ace = int(usable_ace_player)
            player_sum -= 12
            dealer_card -= 1
            # Update values of state-action
            if target_action == action:
                proba_b_a *= (1-epsilon)
                
                old_val = state_action_values[player_sum, dealer_card, usable_ace, action]
                sa_count = state_action_pair_count[player_sum, dealer_card, usable_ace, action]
                new_val = old_val + (G*(1/proba_b_a) - old_val)/(sa_count + 1)
                #new_val = old_val + (1/proba_b_a)/(sa_count + 1)*(G - old_val)
                state_action_values[player_sum, dealer_card, usable_ace, action] = new_val
                state_action_pair_count[player_sum, dealer_card, usable_ace, action] += 1
            else :
                break
        delta = abs(state_action_values - old_sa).max()
        delta_history.append(delta)
    return state_action_values, delta_history

usable Ace

no usable Ace

策略可视化

weighted importance sampling(加权重要性采样)

$$Q(s,a)=\frac{{\sum }_{t\in \mathcal{J}(s,a)}{\rho }_{t:T(t)-1}{G}_t}{{\sum }_{t\in \mathcal{J}(s,a)}{\rho }_{t:T(t)-1}}$$

MC-off policy(weighted importance sampling)

• Initialize,for all $s\in \mathcal{S},a\in \mathcal{A}(s)$:
  • $Q(s,a)\in \mathbb{R}(arbitrarily)$
  • $C(s,a)\leftarrow 0$
  • $\pi (s)\leftarrow \underset{a}{\mathrm{argmax}}Q(s,a)$
• Loop,for each episode:
  • $b\leftarrow$ soft policy
  • Generate an episode use $b:S_0,A_0,R_1,...,S_{T-1},A_{T-1},R_T$
  • $G\leftarrow 0$
  • $W\leftarrow 1$
  • Loop, for each step of episode,$t=T-1,T-2,...,0$:
    • $G\leftarrow \gamma G+R_{t+1}$
    • $\pi (S_t)\leftarrow \underset{a}{\mathrm{argmax}}Q(S_t,a)$
    • $C(S_t,A_t)\leftarrow C(S_t,A_t)+W$
    • $Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\frac{WG-Q(S_t,A_t)}{C(S_t,A_t)}or(\frac{W}{C(S_t,A_t)}[G-Q(S_t,A_t)])$
    • if $A_t\ne \pi (S_t)$, exit inner Loop (proceed to next episode)
    • $W\leftarrow W\frac{1}{b(A_t|S_t)}$

---

• $C(S_t,A_t)$ : ${\sum }_{t\in \mathcal{J}(s,a)}{\rho }_{t:T(t)-1}$系数的累计和
• $W$ : 系数$\rho$

def monte_carlo_off_policy(episodes, gamma=1.0, epsilon=0.1, threshold=0.0001):
    # nearly greedy policy ：behavior
    def soft_policy(usable_ace, player_sum, dealer_card, epsilon=0.1):
        usable_ace = int(usable_ace)
        player_sum -= 12
        dealer_card -= 1
        values_ = state_action_values[player_sum, dealer_card, usable_ace, :]
        proba = np.random.uniform(0, 1)
        if proba <= epsilon:
            action = np.random.randint(0, 2)
        else:
            action = np.random.choice([action_ for action_, value_ in enumerate(values_) if value_ == np.max(values_)])
        return action

    # greedy policy ：target
    def greedy_policy(usable_ace, player_sum, dealer_card):
        usable_ace = int(usable_ace)
        player_sum -= 12
        dealer_card -= 1
        values_ = state_action_values[player_sum, dealer_card, usable_ace, :]
        action = np.random.choice([action_ for action_, value_ in enumerate(values_) if value_ == np.max(values_)])
        return action
    # Initialize
    state_action_values = np.zeros((10, 10, 2, 2))
    state_action_pair_weights = np.ones((10, 10, 2, 2))
    # Loop for each episode
    delta_history = []
    for episode in tqdm(range(episodes)):
        old_sa = state_action_values.copy()
        # Generate an episode
        player_reward, player_traj = play_blackjack(soft_policy, dealer_policy)
        player_states = [t.state for t in player_traj]
        player_actions = [t.action for t in player_traj]
        player_rewards = [0]*len(player_states)
        player_rewards[-1] = player_reward
        # State,Action,Return
        R = 0
        Gs = []
        for r in player_rewards[::-1]:
            R = r + gamma * R
            Gs.insert(0, R)
        # Loop ::-1
        proba_b_a = 1.0
        for player_state, action, G in zip(player_states[::-1], player_actions[::-1], Gs[::-1]):
            usable_ace_player, player_sum, dealer_card = player_state
            target_action = greedy_policy(usable_ace_player, player_sum, dealer_card)  # target policy
            usable_ace = int(usable_ace_player)
            player_sum -= 12
            dealer_card -= 1
            # Update values of state-action
            if target_action == action:
                proba_b_a *= 0.9
                
                old_val = state_action_values[player_sum, dealer_card, usable_ace, action]
                sa_weight = state_action_pair_weights[player_sum, dealer_card, usable_ace, action]
                new_val = old_val + (G*(1/proba_b_a) - old_val)/(sa_weight + 1/proba_b_a)
                #new_val = old_val + (1/proba_b_a)/(sa_count + 1)*(G - old_val)
                state_action_values[player_sum, dealer_card, usable_ace, action] = new_val
                state_action_pair_weights[player_sum, dealer_card, usable_ace, action] += 1/proba_b_a
            else :
                break
        delta = abs(state_action_values - old_sa).max()
        delta_history.append(delta)
    return state_action_values, delta_history

usable Ace

no usable Ace

策略可视化

3. Discounting-aware Importance Sampling(折扣重要性采样)

核心思想 flat partial returns:

• flat: 表示没有折扣。
• partial : 表示Return不是完整，是部分的。
  $${\overline{G}}_{t:h}=R_{t+1}+R_{t+2}+...+R_h,0<=t<h<=T$$
• $G_{t:h}$ : time step t到h之间的Returns
• $T$ : terminal time step
  完整的Returns可以表示为部分Returns的和：
  $$\begin{array}{rl}{G}_t& =R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+...+{\gamma }^{T-t-1}{R}_T\\ & =(1-\gamma )R_{t+1}\\ & +(1-\gamma )\gamma (R_{t+1}+R_{t+2})\\ & +(1-\gamma ){\gamma }^2(R_{t+1}+R_{t+2}+R_{t+3})\\ & +...\\ & +(1-\gamma ){\gamma }^{T-t-2}(R_{t+1}+R_{t+2}+...+R_{T-1})\\ & +{\gamma }^{T-t-1}(R_{t+1}+R_{t+2}+...+R_T)\\ & =(1-\gamma )\sum _{h=t+1}^{T-1}{\gamma }^{h-t-1}{\overline{G}}_{t:h}+{\gamma }^{T-t-1}{\overline{G}}_{t:T}\end{array}$$

Discounting-aware & ordinary importance-sampling

$$Q(s,a)=\frac{{\sum }_{t\in \mathcal{J}(s,a)}((1-\gamma ){\sum }_{h=t+1}^{T(t)-1}{\gamma }^{h-t-1}{\rho }_{t:h-1}{\overline{G}}_{t:h}+{\gamma }^{T(t)-t-1}{\rho }_{t:T(t)-1}{\overline{G}}_{t:T(t)})}{|\mathcal{J}(s,a)|}$$

Discount-aware & weight importance-sampling

$$Q(s,a)=\frac{{\sum }_{t\in \mathcal{J}(s,a)}((1-\gamma ){\sum }_{h=t+1}^{T(t)-1}{\gamma }^{h-t-1}{\rho }_{t:h-1}{\overline{G}}_{t:h}+{\gamma }^{T(t)-t-1}{\rho }_{t:T(t)-1}{\overline{G}}_{t:T(t)})}{{\sum }_{t\in \mathcal{J}(s,a)}((1-\gamma ){\sum }_{h=t+1}^{T(t)-1}{\gamma }^{h-t-1}{\rho }_{t:h-1}+{\gamma }^{T(t)-t-1}{\rho }_{t:T(t)-1})}$$

4. Per-decision Importance Sampling

关键思想：$\mathbb{E}[{\rho }_{t:T-1}{R}_{t+1}]=\mathbb{E}[{\rho }_{t:t}{R}_{t+1}]$
$$\mathbb{E}[{\rho }_{t:T-1}{R}_{t+k}]=\mathbb{E}[{\rho }_{t:t+k-1}{R}_{t+k}]$$
$$\mathbb{E}[{\rho }_{t:T-1}{G}_t]=[{\tilde{G}}_t]$$
$${\tilde{G}}_t={\rho }_{t:t}{R}_{t+1}+\gamma {\rho }_{t:t+1}{R}_{t+2}+{\gamma }^2{\rho }_{t:t+2}{R}_{t+3}+...+{\gamma }^{T-t-1}{\rho }_{t:T-1}{R}_T$$
$$Q(s,a)=\frac{{\sum }_{t\in \mathcal{J}(s,a)}{\tilde{G}}_t}{|\mathcal{J}(s,a)|}$$