强化学习:Reinforce with Baseline求解MountainCar-v0小车上山问题

赛亚茂

已于 2022-08-23 18:01:52 修改

阅读量1.2k

点赞数

分类专栏：集群机器人文章标签：机器学习 python 人工智能

于 2022-08-12 21:47:09 首次发布

本文链接：https://blog.csdn.net/shengzimao/article/details/126311397

版权

集群机器人专栏收录该内容

34 篇文章

订阅专栏

1.问题背景

小车上山问题的问题背景就不再赘述了，在实现过程中用到了python的gym库。导入该环境的过程代码如下:

import gym
# 环境类型
env = gym.make("MountainCar-v0")
env = env.unwrapped
print("初始状态{}".format(np.array(env.reset())))

而提前需要导入的库如下:

import sys
import numpy as np
import torch
# 导入torch的各种模块
import torch.nn as nn
from torch.nn import functional as F
from torch.distributions import Categorical

2.A2C原理

在这里插入图片描述

A2C的原理不过多赘述，只需要了解其策略网络 $\pi(a|s;\theta)$ 的梯度为:
$\nabla_{\theta}J(\theta)=E_{s_t,a_t\sim\pi(.|s_t;\theta)} [A(s_t,a_t;\omega)\nabla_{\theta}\ln\pi(a_t|s_t;\theta)] \\ \theta \leftarrow \theta +\alpha \nabla_{\theta}J(\theta)$ 其中： $A(s_t,a_t)=Q(s_t,a_t)-v(s_t;\omega)\approx G_t-v(s_t;\omega)$ 为优势函数。
而对于每一个轨迹 $\tau:s_0a_0r_0s_1,...s_{T-1}a_{T-1}r_{T-1}s_T$ 而言：
$\nabla_{\theta}J(\theta)=E_{\tau} [\nabla_{\theta}\sum_{i=0}^{T-1}\ln\pi(a_t|s_t;\theta)(R(\tau)-v(s_t;\omega))] \\$ 其中: $R(\tau)=\sum_{i=0}^{\infty} \gamma^{i}r_t$ 。
对于 $v(s;\omega)$ 其梯度采用MC算法计算:
$\nabla_{\omega}J(\omega)=(G_t-v(s_t;\omega)) \nabla_{\omega} v(s_t;\omega) \\ \omega \leftarrow \omega-\beta \nabla_{\omega}J(\omega)$ 其中: $G_t=\sum_{i=t}^T\gamma^ir_i$ 。
将初始化函数__init__;网络的前馈函数forward;根据 $\pi(a|s;\theta),v(s;\omega)$ 采样动作及输出 $\ln \pi(a_t|s_t;\theta),G_t$ 的函数为:select_action;更新网络参数的函数为:update_policy;网络的训练函数train_network封装到policyNet类中。其代码如下:

class policyNet(nn.Module):
    def __init__(self,state_num,action_num,hidden_num=50,lr=0.01):
        super(policyNet,self).__init__()
        self.fc1 = nn.Linear(state_num,hidden_num)
        self.action_fc = nn.Linear(hidden_num,action_num)
        self.state_fc = nn.Linear(hidden_num,1)
        self.optimzer = torch.optim.Adam(self.parameters(),lr=lr)
        # 动作与回报的缓存
        self.saved_states = []
        self.saved_log_pi = []
        self.saved_values = []
        self.saved_actions = []
        self.saved_rewards = []
    # 前馈
    def forward(self,state):
        hidden_output = self.fc1(state)
        x = F.relu(hidden_output)
        action_prob = self.action_fc(x)
        action_prob = F.softmax(action_prob,dim=-1)
        state_values = self.state_fc(x)
        return action_prob,state_values
    # 选择动作
    def select_action(self,state):
        # 输入state为numpy数据类型，要先进行数据转化
        # 输出lnpi(a|s),v(s),r
        state = torch.from_numpy(state).float()
        probs,state_value = self.forward(state)
        m = Categorical(probs)
        action = m.sample()
        # 存储数据
        self.saved_states.append(state)
        self.saved_log_pi.append(m.log_prob(action))
        self.saved_values.append(state_value)
        self.saved_actions.append(action)
        # 返回action
        return action.item()
    # 实现A2C策略梯度更新
    def update_policy(self,gamma=1.0):
        # 先计算Gt
        R = 0
        G = np.zeros_like(self.saved_rewards)
        for t in reversed(range(len(self.saved_rewards))):
            R = self.saved_rewards[t] + gamma*R
            G[t] = R
        # 对G标准化
        G = (G - np.mean(G))/np.std(G)
        G = torch.tensor(G)
        # 下面开始求policy_loss,value_loss
        policy_loss = []
        value_loss = []
        for t in range(len(self.saved_rewards)):
            log_prob = self.saved_log_pi[t]
            value = self.saved_values[t]
            Gt = G[t]
            advantage = Gt - value
            # 计算actor的loss
            policy_loss.append(-advantage*log_prob)
            value_loss.append(F.smooth_l1_loss(value,Gt))
        # 下面开始优化
        self.optimzer.zero_grad()
        loss = torch.stack(policy_loss).sum() + torch.stack(value_loss).sum()
        loss.backward()
        self.optimzer.step()
        # 更新完后重置缓存
        del self.saved_actions[:]
        del self.saved_log_pi[:]
        del self.saved_rewards[:]
        del self.saved_states[:]
        del self.saved_values[:]
    # 实现训练
    def train_network(self,env,epsiodes=1000,gamma=1.0):
        # 设置数组用以存储奖励
        epsiode_rewards_array = []
        rewards_mean_array = []
        # env是环境
        for epsiode in range(epsiodes):
            # 初始化环境状态
            state = np.array(env.reset())
            epsiode_reward = 0
            while True:
                action = self.select_action(state)
                next_state,reward,done,_ = env.step(action)
                self.saved_rewards.append(reward)
                epsiode_reward += reward
                if done:
                    #print("第{}次迭代的奖励值为{}".format(epsiode,epsiode_reward))
                    break
                state = next_state
            self.update_policy(gamma=gamma)
            epsiode_rewards_array.append(epsiode_reward)
            rewards_mean_array.append(np.mean(epsiode_rewards_array))
            print("第{}次迭代的奖励值为{},累计奖励{}".format(epsiode,epsiode_reward,int(rewards_mean_array[-1])))
        return epsiode_rewards_array,rewards_mean_array

3.训练网络

主函数调用如下:

policyNetWork = policyNet(2,env.action_space.n)
rewards_array,rewards_mean_array = policyNetWork.train_network(env,epsiodes=500)
# 下面是画图函数
import matplotlib.pyplot as plt
plt.plot(rewards_array)
plt.plot(rewards_mean_array)
plt.show()