强化学习 Q-Learning 几个例子
参考:
例子1:
https://mofanpy.com/tutorials/machine-learning/reinforcement-learning/general-rl/
例子2:
https://mofanpy.com/tutorials/machine-learning/reinforcement-learning/tabular-q1/
Q-Learning 算法回顾
- 初始化Q表 q_table
- for episode in range(MAX_Episode):
- S = 0 // 设置初始状态0
- is_terminated = False
- step_counter = 0
- update_env(S, episode, step_counter) // 这个只是为了可视化和为了最后到达之后显示episode信息
- while not is_termitated:
- A = choose_action(S, q_table) // 从Q表中运用epsilon-greedy选择一个策略
- S_, R = get_env_feedback(S, A) // 查询如果我们在State S采用了这个动作,能到达的新的状态S’和得到的奖励R
- q_predict = q_table.loc[S, A] // 老的Q(s,a)
- if S_ != ‘terminal’:
- q_target = R + GAMMA * q_table.iloc[S_, :].max() # next state is not terminal
- else:
- q_target = R # next state is terminal,我们就没有max Q(s’,a’)这项了
- is_terminated = True # terminate this episode
- q_table.loc[S, A] += ALPHA * (q_target - q_predict) # 更新Q表
- S = S_ # 移动到下一个状态
- update_env(S, episode, step_counter+1)
- step_counter += 1
- 返回最后的Q表
如果去掉可视化,算法为:
- 初始化Q表 q_table
- for episode in range(MAX_Episode):
- S = 0 // 设置初始状态0
- is_terminated = False
- while not is_termitated:
- A = choose_action(S, q_table) // 从Q表中运用epsilon-greedy选择一个策略
- S_, R = get_env_feedback(S, A) // 查询如果我们在State S采用了这个动作,能到达的新的状态S’和得到的奖励R
- q_predict = q_table.loc[S, A] // 老的Q(s,a)
- if S_ != ‘terminal’:
- q_target = R + GAMMA * q_table.iloc[S_, :].max() # next state is not terminal
- else:
- q_target = R # next state is terminal,我们就没有max Q(s’,a’)这项了
- is_terminated = True # terminate this episode
- q_table.loc[S, A] += ALPHA * (q_target - q_predict) # 更新Q表
- S = S_ # 移动到下一个状态
- 返回最后的Q表
1. 一维找宝藏
o----T
# T 就是宝藏的位置, o 是探索者的位置
import numpy as np
import pandas as pd
import time
##########
# 参数定义 #
##########
N_STATES = 6 # -----T 长度
ACTIONS = ['left','right']
FRESH_TIME = 0.01 # 用于显示探索每一步的结果
MAX_EPISODES = 13 # 最多的Episode数量
GAMMA = 0.9 # Discount Factor
alpha = 0.1 # Learning Rate
EPSILON = 0.9
############
## 环境搭建 ##
############
# S = [0,1,2,3,...,N_STATES-2,'terminal']
# A = ['left','right']
# 返回值是 S_,R
def get_env_feedback(S,A):
if A == 'right':
if S==N_STATES-2:
S_ = 'terminal'
R = 1 # 到达终点奖励
else:
S_ = S+1
R = 0
else:
# 如果已经到底了,位置不变
if S==0:
S_ = 0
R = 0
else:
S_ = S-1
R = 0
return S_,R
# 可视化与结果信息打印,真正状态定义实际上是在get_env_feedback
def update_env(S,episode,step_counter):
env_list = ["-"]*(N_STATES-1) + ['T']
if S == 'terminal':
# 直接显示这个episode的信息
msg = 'Episode