强化学习实践 | Q-Learning 几个例子

参考：
例子1:
https://mofanpy.com/tutorials/machine-learning/reinforcement-learning/general-rl/
例子2:
https://mofanpy.com/tutorials/machine-learning/reinforcement-learning/tabular-q1/

Q-Learning 算法回顾

• 初始化Q表 q_table
• for episode in range(MAX_Episode):
  • S = 0 // 设置初始状态0
  • is_terminated = False
  • step_counter = 0
  • update_env(S, episode, step_counter) // 这个只是为了可视化和为了最后到达之后显示episode信息
  • while not is_termitated:
    • A = choose_action(S, q_table) // 从Q表中运用epsilon-greedy选择一个策略
    • S_, R = get_env_feedback(S, A) // 查询如果我们在State S采用了这个动作，能到达的新的状态S’和得到的奖励R
    • q_predict = q_table.loc[S, A] // 老的Q(s,a)
    • if S_ != ‘terminal’:
      • q_target = R + GAMMA * q_table.iloc[S_, :].max() # next state is not terminal
    • else:
      • q_target = R # next state is terminal，我们就没有max Q(s’,a’)这项了
      • is_terminated = True # terminate this episode
    • q_table.loc[S, A] += ALPHA * (q_target - q_predict) # 更新Q表
    • S = S_ # 移动到下一个状态
    • update_env(S, episode, step_counter+1)
    • step_counter += 1
• 返回最后的Q表

如果去掉可视化，算法为：

• 初始化Q表 q_table
• for episode in range(MAX_Episode):
  • S = 0 // 设置初始状态0
  • is_terminated = False
  • while not is_termitated:
    • A = choose_action(S, q_table) // 从Q表中运用epsilon-greedy选择一个策略
    • S_, R = get_env_feedback(S, A) // 查询如果我们在State S采用了这个动作，能到达的新的状态S’和得到的奖励R
    • q_predict = q_table.loc[S, A] // 老的Q(s,a)
    • if S_ != ‘terminal’:
      • q_target = R + GAMMA * q_table.iloc[S_, :].max() # next state is not terminal
    • else:
      • q_target = R # next state is terminal，我们就没有max Q(s’,a’)这项了
      • is_terminated = True # terminate this episode
    • q_table.loc[S, A] += ALPHA * (q_target - q_predict) # 更新Q表
    • S = S_ # 移动到下一个状态
• 返回最后的Q表

1. 一维找宝藏

o----T
# T 就是宝藏的位置, o 是探索者的位置

import numpy as np
import pandas as pd
import time

##########
# 参数定义 #
##########

N_STATES = 6   # -----T 长度
ACTIONS = ['left','right']
FRESH_TIME = 0.01 # 用于显示探索每一步的结果
MAX_EPISODES = 13 # 最多的Episode数量
GAMMA = 0.9       # Discount Factor
alpha = 0.1       # Learning Rate
EPSILON = 0.9

############
## 环境搭建 ##
############

# S = [0,1,2,3,...,N_STATES-2,'terminal']
# A = ['left','right']
# 返回值是 S_,R
def get_env_feedback(S,A):
    if A == 'right':
        if S==N_STATES-2:
            S_ = 'terminal'
            R = 1 # 到达终点奖励
        else:
            S_ = S+1
            R = 0
    else:
        # 如果已经到底了，位置不变
        if S==0:
            S_ = 0
            R = 0
        else:
            S_ = S-1
            R = 0
    return S_,R

# 可视化与结果信息打印，真正状态定义实际上是在get_env_feedback
def update_env(S,episode,step_counter):
    env_list = ["-"]*(N_STATES-1) + ['T']
    if S == 'terminal':
        # 直接显示这个episode的信息
        msg = 'Episode