欢迎来到强化学习系列的第3部分以及Q学习部分的第3部分。到目前为止,我们已经成功地制作了一个Q-learning算法来导航OpenAI山地车环境。现在的问题是,我们有很多需要调优的参数。能够打败游戏是一回事,但我们可能想要更快地打败它,甚至尝试探索更快地学习的方法。为了做到这一点,我们需要开始弄清楚我们到底在做什么。
首先,我们可以从程序中跟踪一些非常基本的度量标准。我们的启动脚本:
# objective is to get the cart to the flag.
# for now, let's just move randomly:
import gym
import numpy as np
env = gym.make("MountainCar-v0")
LEARNING_RATE = 0.1
DISCOUNT = 0.95
EPISODES = 25000
SHOW_EVERY = 3000
DISCRETE_OS_SIZE = [20] * len(env.observation_space.high)
discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/DISCRETE_OS_SIZE
# Exploration settings
epsilon = 1 # not a constant, qoing to be decayed
START_EPSILON_DECAYING = 1
END_EPSILON_DECAYING = EPISODES//2
epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)
q_table = np.random.uniform(low=-2, high=0, size=(DISCRETE_OS_SIZE + [env.action_space.n]))
def get_discrete_state(state):
discrete_state = (state - env.observation_space.low)/discrete_os_win_size
return tuple(discrete_state.astype(np.int)) # we use this tuple to look up the 3 Q values for the available actions in the q-table
for episode in range(EPISODES):
discrete_state = get_discrete_state(env.reset())
done = False
if episode % SHOW_EVERY == 0:
render = True
print(episode)
else:
render = False
while not done:
if np.random.random() > epsilon:
# Get action from Q table
action = np.argmax(q_table[discrete_state])
else:
# Get random action
action = np.random.randint(0, env.action_space.n)
new_state, reward, done, _ = env.step(action)
new_discrete_state = get_discrete_state(new_state)
if episode % SHOW_EVERY == 0:
env.render()
#new_q = (1 - LEARNING_RATE) * current_q + LEARNING_RATE * (reward + DISCOUNT * max_future_q)
# If simulation did not end yet after last step - update Q table
if not done:
# Maximum possible Q value in next step (for new state)
max_future_q = np.max(q_table[new_discrete_state])
# Current Q value (for current state and performed action)
current_q = q_table[discrete_state + (action,)]
# And here's our equation for a new Q value for current state and action
new_q = (1 - LEARNING_RATE) * current_q + LEARNING_RATE * (reward + DISCOUNT * max_future_q)
# Update Q table with new Q value
q_table[discrete_state + (action,)] = new_q
# Simulation ended (for any reson) - if goal position is achived - update Q value with reward directly
elif new_state[0] >= env.goal_position:
#q_table[discrete_state + (action,)] = reward
q_table[discrete_state + (action,)] = 0
discrete_state = new_discrete_state
# Decaying is being done every episode if episode number is within decaying range
if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:
epsilon -= epsilon_decay_value
env.close()
为了进行修改,让我们先将剧集更改为4000集,以保持迭代速度更快。然后我们将添加一个名为STATS_EVERY的新参数,并将其设置为100。
接下来,在我们的其他定义的顶部,让我们添加
# For stats
ep_rewards = []
aggr_ep_rewards = {'ep': [], 'avg': [], 'max': [], 'min': []}
我们将使用它们来跟踪不同的值,通过训练来绘制它们。
然后,我们将episode_reward = 0添加到我们的情节迭代中:
for episode in range(EPISODES):
episode_reward = 0
...
接下来,在我们收到奖励信息后,我们可以存储它:
new_state, reward, done, _ = env.step(action) # was already in our code
episode_reward += reward
然后在剧集循环的最后,我们可以加上:
ep_rewards.append(episode_reward)
if not episode % STATS_EVERY:
average_reward = sum(ep_rewards[-STATS_EVERY:])/STATS_EVERY
aggr_ep_rewards['ep'].append(episode)
aggr_ep_rewards['avg'].append(average_reward)
aggr_ep_rewards['max'].append(max(ep_rewards[-STATS_EVERY:]))
aggr_ep_rewards['min'].append(min(ep_rewards[-STATS_EVERY:]))
print(f'Episode: {episode:>5d}, average reward: {average_reward:>4.1f}, current epsilon: {epsilon:>1.2f}')
env.close() # this was already here, no need to add it again. Just here so you know where we are :)
最后,在我们的脚本的最后,我们可以看到:
plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label="average rewards")
plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label="max rewards")
plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label="min rewards")
plt.legend(loc=4)
plt.show()
不要忘记在顶部导入:import matplotlib.pyplot as plt
现在我们可以看到结果:
然后,我们可以调整某些事情,看看它是对我们有利还是有害。例如,我们可以尝试改变我们的Epsilon衰减策略。让我们把它设置为衰变到最后:
看起来第二个模型想要继续,让我们把剧集增加到10000集,然后增加plt.grid(True)
在plt.show()
前
回到END_EPSILON_DECAYING = EPISODES//2
这看起来很理想。调整观测空间怎么样?试40个桶。
DISCRETE_OS_SIZE = [40] * len(env.observation_space.high)
看起来它需要更多的训练。这是有意义的,因为我们显著地增加了表的大小。让我们做25000集。
看到这一点,我们似乎希望这个模式能有大约2万集的剧集,因为它有很高的总回报,但最低的回报也很高。另外,我们可能想训练模特…我不知道……使用它们?所以我们想要保存最终的表,但是,我建议我们把它们都保存起来!为什么?
所以我们可以画漂亮的图片,哇!
…以及在培训中使用模型。
首先,我们创建一个名为qtables
的新目录。在这里,我们将保存每一集的q表。
然后,我们可以在情节循环的末尾添加一个np.save()
:
for episode in range(EPISODES):
...
# AT THE END
np.save(f"qtables/{episode}-qtable.npy", q_table)
env.close()
这是每个q表。有很多Q表…因此,在我们的离散观察大小下,目录将是~ 1gb。如果您想将大小缩减到100mb左右,您可以做如下操作
if episode % 10 == 0:
np.save(f"qtables/{episode}-qtable.npy", q_table)
如果你想要10mb的空间,可以选择100。不管怎样,现在我们有了所有的q表。最后的q表看起来很好:
所以我们可以直接用它。所以现在,与其初始化一个随机qtable,不如直接用np。加载该文件。然后,您可以继续更新q值,或者使用这个表查找值。
终于!我我们没有白白保存所有这些Q表。漂亮的图片(和视频)时间!
为此,我们将打开一个新脚本。
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
style.use('ggplot')
def get_q_color(value, vals):
if value == max(vals):
return "green", 1.0
else:
return "red", 0.3
fig = plt.figure(figsize=(12, 9))
ax1 = fig.add_subplot(311)
ax2 = fig.add_subplot(312)
ax3 = fig.add_subplot(313)
i = 24999
q_table = np.load(f"qtables/{i}-qtable.npy")
for x, x_vals in enumerate(q_table):
for y, y_vals in enumerate(x_vals):
ax1.scatter(x, y, c=get_q_color(y_vals[0], y_vals)[0], marker="o", alpha=get_q_color(y_vals[0], y_vals)[1])
ax2.scatter(x, y, c=get_q_color(y_vals[1], y_vals)[0], marker="o", alpha=get_q_color(y_vals[1], y_vals)[1])
ax3.scatter(x, y, c=get_q_color(y_vals[2], y_vals)[0], marker="o", alpha=get_q_color(y_vals[2], y_vals)[1])
ax1.set_ylabel("Action 0")
ax2.set_ylabel("Action 1")
ax3.set_ylabel("Action 2")
plt.show()
所以这就是每个动作的Q表,给我们:
现在,我们可以画出所有,或者很多集。我建议把它们都画成图形。那需要25K帧,也就是以60fps拍摄7分钟的视频。可能毫无意义。如果我们每10帧画一张图,那就是41秒。这样,我们可以看到Q值随时间的变化,以及模型是如何“学习”的。
例如,如果我们设i = 1
你可以清楚地看到它是随机的。不是令人震惊的,我们是随机初始化的!
现在让我们遍历每10个q表,创建并保存图表。
代码是:
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
style.use('ggplot')
def get_q_color(value, vals):
if value == max(vals):
return "green", 1.0
else:
return "red", 0.3
fig = plt.figure(figsize=(12, 9))
for i in range(0, 25000, 10):
print(i)
ax1 = fig.add_subplot(311)
ax2 = fig.add_subplot(312)
ax3 = fig.add_subplot(313)
q_table = np.load(f"qtables/{i}-qtable.npy")
for x, x_vals in enumerate(q_table):
for y, y_vals in enumerate(x_vals):
ax1.scatter(x, y, c=get_q_color(y_vals[0], y_vals)[0], marker="o", alpha=get_q_color(y_vals[0], y_vals)[1])
ax2.scatter(x, y, c=get_q_color(y_vals[1], y_vals)[0], marker="o", alpha=get_q_color(y_vals[1], y_vals)[1])
ax3.scatter(x, y, c=get_q_color(y_vals[2], y_vals)[0], marker="o", alpha=get_q_color(y_vals[2], y_vals)[1])
ax1.set_ylabel("Action 0")
ax2.set_ylabel("Action 1")
ax3.set_ylabel("Action 2")
#plt.show()
plt.savefig(f"qtable_charts/{i}.png")
plt.clf()
这将使我们所有的图像,现在我们可以从他们的视频,与:
import cv2
import os
def make_video():
# windows:
fourcc = cv2.VideoWriter_fourcc(*'XVID')
# Linux:
#fourcc = cv2.VideoWriter_fourcc('M','J','P','G')
out = cv2.VideoWriter('qlearn.avi', fourcc, 60.0, (1200, 900))
for i in range(0, 14000, 10):
img_path = f"qtable_charts/{i}.png"
print(img_path)
frame = cv2.imread(img_path)
out.write(frame)
out.release()
make_video()
示例视频:
https://youtu.be/ObMsyrwVXTc
在下一个教程中,我们将创建我们自己的环境来使用Q-Learning。