强化学习_Actor Critic算法实现CartPole-v0

Actor Critic

Actor-Critic，其实是用了两个网络，两个网络有一个共同点，输入状态S：

• 一个输出策略，负责选择动作，我们把这个网络成为 Actor;
• 一个负责计算每个动作的分数，我们把这个网络成为 Critic.

再说一遍，Critic 网络负责估算 Q值; Actor 网络负责估算策略；

Actor通过Critic 给出的分数，去学习：如果 Critic 给的分数高，那么 Actor 会调整这个动作的输出概率；相反，如果 Critic 给的分数低，那么就减少这个动作输出的概率。

Actor (PG) 整合 Critic

回顾策略梯度 PG，它是利用带权重的梯度下降方法更新策略，而获得权重的方法是蒙地卡罗计算G值。

蒙地卡罗需要完成整个游戏过程，直到最终状态，才能通过回溯计算G 值。这使得 PG方法 的效率被限制。我们知道，MC 的效率是相对比较低的，因为需要一直走到最终状态。

所以我们希望用 TD 代替 MC，可以不用回湖，每一步立马估算出来G值，可以把 AC 理解成PG的TD版本。

这里Q值定义正好就是action 之后获得的奖励期望有多大，Baseline是Q的求平均，也就是Q的期望，那么Q的期望正好是V

Advantage Actor Critic

Actor 用 Q(s,a)-V(s) 去指导更新，但我们之前也说过 Q 和 V 都要估算太麻烦了。如果这么做需要两个网络，一个网络估算 Q，一个网络估算V。两倍的风险估测不准！能不能只统一成V呢？

Q(s,a)用 gamma * V(s’)＋r 来代替，如上图，于是整理后就可以得到：

gamma * V(s’) + r - V(s): 我们把这个差，叫做 TD-error

显而易见的，我们只需要用 value function 更新Actor 策略网络：

Trick 1 shared parameters

如果传入的 state是一张图片，则 Critic和Actor 网络都需要对图片进行特征提取，这两个网络可以共享特征提取层。

Trick 2 修改 reward

如果已经到达最终状态，那么奖励直接扣 20 点。

我们把 reward 减去 20，相当于是对濒死状态下，选择动作a的强烈不认同。通过 -20 大幅减少动作a出现的概率。

再进一步，reward 会向前传播，让智能体濒死状态之前状态时，不选择会进入濒死状态的动作，努力避免进入濒死状态。

代码实现

"""
Actor-Critic using TD-error as the Advantage, Reinforcement Learning.

The cart pole example. Policy is oscillated.

View more on my tutorial page: https://morvanzhou.github.io/tutorials/

Using:
tensorflow 1.0
gym 0.8.0
"""

import numpy as np
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

import gym

np.random.seed(2)


# Superparameters
OUTPUT_GRAPH = False
MAX_EPISODE = 3000
DISPLAY_REWARD_THRESHOLD = 200  # renders environment if total episode reward is greater then this threshold
MAX_EP_STEPS = 1000   # maximum time step in one episode
RENDER = False  # rendering wastes time
GAMMA = 0.9     # reward discount in TD error
LR_A = 0.001    # learning rate for actor
LR_C = 0.01     # learning rate for critic

env = gym.make('CartPole-v0',render_mode='human')

env = env.unwrapped

N_F = env.observation_space.shape[0]
N_A = env.action_space.n


class Actor(object):
    def __init__(self, sess, n_features, n_actions, lr=0.001):
        self.sess = sess

        self.s = tf.placeholder(tf.float32, [1, n_features], "state")
        self.a = tf.placeholder(tf.int32, None, "act")
        self.td_error = tf.placeholder(tf.float32, None, "td_error")  # TD_error

        with tf.variable_scope('Actor'):
            l1 = tf.layers.dense(
                inputs=self.s,
                units=20,    # number of hidden units
                activation=tf.nn.relu,
                kernel_initializer=tf.random_normal_initializer(0., .1),    # weights
                bias_initializer=tf.constant_initializer(0.1),  # biases
                name='l1'
            )

            self.acts_prob = tf.layers.dense(
                inputs=l1,
                units=n_actions,    # output units
                activation=tf.nn.softmax,   # get action probabilities
                kernel_initializer=tf.random_normal_initializer(0., .1),  # weights
                bias_initializer=tf.constant_initializer(0.1),  # biases
                name='acts_prob'
            )

        with tf.variable_scope('exp_v'):
            log_prob = tf.log(self.acts_prob[0, self.a])
            self.exp_v = tf.reduce_mean(log_prob * self.td_error)  # advantage (TD_error) guided loss

        with tf.variable_scope('train'):
            self.train_op = tf.train.AdamOptimizer(lr).minimize(-self.exp_v)  # minimize(-exp_v) = maximize(exp_v)

    def learn(self, s, a, td):
        s = s[np.newaxis, :]
        feed_dict = {self.s: s, self.a: a, self.td_error: td}
        _, exp_v = self.sess.run([self.train_op, self.exp_v], feed_dict)
        return exp_v

    def choose_action(self, s):
        s = s[np.newaxis, :]
        probs = self.sess.run(self.acts_prob, {self.s: s})   # get probabilities for all actions
        return np.random.choice(np.arange(probs.shape[1]), p=probs.ravel())   # return a int


class Critic(object):
    def __init__(self, sess, n_features, lr=0.01):
        self.sess = sess

        self.s = tf.placeholder(tf.float32, [1, n_features], "state")
        self.v_ = tf.placeholder(tf.float32, [1, 1], "v_next")
        self.r = tf.placeholder(tf.float32, None, 'r')

        with tf.variable_scope('Critic'):
            l1 = tf.layers.dense(
                inputs=self.s,
                units=20,  # number of hidden units
                activation=tf.nn.relu,  # None
                # have to be linear to make sure the convergence of actor.
                # But linear approximator seems hardly learns the correct Q.
                kernel_initializer=tf.random_normal_initializer(0., .1),  # weights
                bias_initializer=tf.constant_initializer(0.1),  # biases
                name='l1'
            )

            self.v = tf.layers.dense(
                inputs=l1,
                units=1,  # output units
                activation=None,
                kernel_initializer=tf.random_normal_initializer(0., .1),  # weights
                bias_initializer=tf.constant_initializer(0.1),  # biases
                name='V'
            )

        with tf.variable_scope('squared_TD_error'):
            self.td_error = self.r + GAMMA * self.v_ - self.v
            self.loss = tf.square(self.td_error)    # TD_error = (r+gamma*V_next) - V_eval
        with tf.variable_scope('train'):
            self.train_op = tf.train.AdamOptimizer(lr).minimize(self.loss)

    def learn(self, s, r, s_):
        s, s_ = s[np.newaxis, :], s_[np.newaxis, :]

        v_ = self.sess.run(self.v, {self.s: s_})
        td_error, _ = self.sess.run([self.td_error, self.train_op],
                                          {self.s: s, self.v_: v_, self.r: r})
        return td_error


sess = tf.Session()

actor = Actor(sess, n_features=N_F, n_actions=N_A, lr=LR_A)
critic = Critic(sess, n_features=N_F, lr=LR_C)     # we need a good teacher, so the teacher should learn faster than the actor

sess.run(tf.global_variables_initializer())

if OUTPUT_GRAPH:
    tf.summary.FileWriter("logs/", sess.graph)

for i_episode in range(MAX_EPISODE):
    s = env.reset()[0]
    t = 0
    track_r = []
    while True:
        if RENDER: env.render()

        a = actor.choose_action(s)

        s_, r, done, info, _ = env.step(a)

        if done: r = -20

        track_r.append(r)

        td_error = critic.learn(s, r, s_)  # gradient = grad[r + gamma * V(s_) - V(s)]
        actor.learn(s, a, td_error)     # true_gradient = grad[logPi(s,a) * td_error]

        s = s_
        t += 1

        if done or t >= MAX_EP_STEPS:
            ep_rs_sum = sum(track_r)

            if 'running_reward' not in globals():
                running_reward = ep_rs_sum
            else:
                running_reward = running_reward * 0.95 + ep_rs_sum * 0.05
            if running_reward > DISPLAY_REWARD_THRESHOLD: RENDER = True  # rendering
            print("episode:", i_episode, "  reward:", int(running_reward))
            break