MAML-RL Pytorch 代码解读 (9) -- maml_rl/envs/normalized_env.py

MAML-RL Pytorch 代码解读 (9) – maml_rl/envs/normalized_env.py

基本介绍

在网上看到的元学习 MAML 的代码大多是跟图像相关的，强化学习这边的代码比较少。

因为自己的思路跟 MAML-RL 相关，所以打算读一些源码。

MAML 的原始代码是基于 tensorflow 的，在 Github 上找到了基于 Pytorch 源码包，学习这个包。

源码链接

https://github.com/dragen1860/MAML-Pytorch-RL

文件路径

./maml_rl/envs/normalized_env.py

import 包

import numpy as np
import gym
from gym import spaces

NormalizedActionWrapper() 类

class NormalizedActionWrapper(gym.ActionWrapper):
	
    #### 这个类的工作是将归一化到[-1,1]上的动作信息扩展成自己环境的类型
    """
	Environment wrapper to normalize the action space to [-1, 1]. This
	wrapper is adapted from rllab's [1] wrapper `NormalizedEnv`
	https://github.com/rll/rllab/blob/b3a28992eca103cab3cb58363dd7a4bb07f250a0/rllab/envs/normalized_env.py

	[1] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel, 
		"Benchmarking Deep Reinforcement Learning for Continuous Control", 2016 
		(https://arxiv.org/abs/1604.06778)
	"""

	#### 初始化类和动作空间，动作空间的形状shape从gym.ActionWrapper类中取得。
    def __init__(self, env):
		super(NormalizedActionWrapper, self).__init__(env)
		self.action_space = spaces.Box(low=-1.0, high=1.0,
		                               shape=self.env.action_space.shape)

    #### 先将动作值归一化到[-1, 1]上，提取环境中的动作上界和下界，再对归一化的动作做扩展。
	def action(self, action):
		# Clip the action in [-1, 1]
		action = np.clip(action, -1.0, 1.0)
		# Map the normalized action to original action space
		lb, ub = self.env.action_space.low, self.env.action_space.high
		action = lb + 0.5 * (action + 1.0) * (ub - lb)
		return action

    #### 提取环境中的动作上界和下界，然后将具体的动作值映射到[-1, 1]上。
	def reverse_action(self, action):
		# Map the original action to normalized action space
		lb, ub = self.env.action_space.low, self.env.action_space.high
		action = 2.0 * (action - lb) / (ub - lb) - 1.0
		# Clip the action in [-1, 1]
		action = np.clip(action, -1.0, 1.0)
		return action

NormalizedObservationWrapper() 类

class NormalizedObservationWrapper(gym.ObservationWrapper):
	
    #### 这个类将观测observation信息转变为可变化的均值和标准差。但是不懂为啥这样做呢，参数很小说明并不是很明显的变化。
    """
	Environment wrapper to normalize the observations with a running mean
	and standard deviation. This wrapper is adapted from rllab's [1] 
	wrapper `NormalizedEnv`
	https://github.com/rll/rllab/blob/b3a28992eca103cab3cb58363dd7a4bb07f250a0/rllab/envs/normalized_env.py

	[1] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel, 
		"Benchmarking Deep Reinforcement Learning for Continuous Control", 2016 
		(https://arxiv.org/abs/1604.06778)
	"""

    #### 类初始化。self.alpha应该是更新变化率，self.epsilon是一个很小的数，用于避免分母为0的情况。shape和dtype获取的是观测空间的形状和数据结构。self._mean均值设置成全0,self._var方差设置成全1。
	def __init__(self, env, alpha=1e-3, epsilon=1e-8):
		super(NormalizedObservationWrapper, self).__init__(env)
		self.alpha = alpha
		self.epsilon = epsilon
		shape = self.observation_space.shape
		dtype = self.observation_space.dtype or np.float32
		self._mean = np.zeros(shape, dtype=dtype)
		self._var = np.ones(shape, dtype=dtype)

    #### 将观测信息通过中心化方法变成标准正态分布，但是其均值和方法会随着观测空间发生细微的变化。
	def observation(self, observation):
		self._mean = (1.0 - self.alpha) * self._mean + self.alpha * observation
		self._var = (1.0 - self.alpha) * self._var + self.alpha * np.square(observation, self._mean)
		return (observation - self._mean) / (np.sqrt(self._var) + self.epsilon)

NormalizedRewardWrapper() 类

class NormalizedRewardWrapper(gym.RewardWrapper):
    
    #### 这个类将奖励reward信息转变为可变化的均值和标准差。跟上面类似。
	"""
	Environment wrapper to normalize the rewards with a running mean
	and standard deviation. This wrapper is adapted from rllab's [1] 
	wrapper `NormalizedEnv`
	https://github.com/rll/rllab/blob/b3a28992eca103cab3cb58363dd7a4bb07f250a0/rllab/envs/normalized_env.py

	[1] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel, 
		"Benchmarking Deep Reinforcement Learning for Continuous Control", 2016 
		(https://arxiv.org/abs/1604.06778)
	"""

    #### 类初始化。self.alpha应该是更新变化率，self.epsilon是一个很小的数，用于避免分母为0的情况。self._mean均值设置成0,self._var方差设置成1，是单变量的标准正态分布。
	def __init__(self, env, alpha=1e-3, epsilon=1e-8):
		super(NormalizedRewardWrapper, self).__init__(env)
		self.alpha = alpha
		self.epsilon = epsilon
		self._mean = 0.0
		self._var = 1.0

    #### 将观测信息通过中心化方法变成标准正态分布，但是其均值和方法会随着观测空间发生细微的变化。
	def reward(self, reward):
		self._mean = (1.0 - self.alpha) * self._mean + self.alpha * reward
		self._var = (1.0 - self.alpha) * self._var + self.alpha * np.square(reward, self._mean)
		return (reward - self._mean) / (np.sqrt(self._var) + self.epsilon)