openai gym学习

openai gym学习

1. 环境的定义 Env类
   class Env(Generic[ObsType, ActType]):
   有如下抽象方法： step reset render close seed

   step

   @abstractmethod
   def step(self, action: ActType) -> Tuple[ObsType, float, bool, dict]:
       """Run one timestep of the environment's dynamics. When end of
       episode is reached, you are responsible for calling `reset()`
       to reset this environment's state.
       Accepts an action and returns a tuple (observation, reward, done, info).
       Args:
           action (object): an action provided by the agent
       Returns:
           observation (object): agent's observation of the current environment
           reward (float) : amount of reward returned after previous action
           done (bool): whether the episode has ended, in which case further step() calls will return undefined results
           info (dict): contains auxiliary diagnostic information (helpful for debugging, logging, and sometimes learning)
       """
       raise NotImplementedError
   

   reset

   @abstractmethod
   def reset(
       self,
       *,
       seed: Optional[int] = None,
       return_info: bool = False,
       options: Optional[dict] = None,
   ) -> Union[ObsType, tuple[ObsType, dict]]:
       """Resets the environment to an initial state and returns an initial
       observation.
       This method should also reset the environment's random number
       generator(s) if `seed` is an integer or if the environment has not
       yet initialized a random number generator. If the environment already
       has a random number generator and `reset` is called with `seed=None`,
       the RNG should not be reset.
       Moreover, `reset` should (in the typical use case) be called with an
       integer seed right after initialization and then never again.
       Returns:
           observation (object): the initial observation.
           info (optional dictionary): a dictionary containing extra information, this is only returned if return_info is set to true
       """
       # Initialize the RNG if the seed is manually passed
       if seed is not None:
           self._np_random, seed = seeding.np_random(seed)
   

   render

    @abstractmethod
    def render(self, mode="human"):
        """Renders the environment.
        The set of supported modes varies per environment. (And some
        third-party environments may not support rendering at all.)
        By convention, if mode is:
        - human: render to the current display or terminal and
          return nothing. Usually for human consumption.
        - rgb_array: Return an numpy.ndarray with shape (x, y, 3),
          representing RGB values for an x-by-y pixel image, suitable
          for turning into a video.
        - ansi: Return a string (str) or StringIO.StringIO containing a
          terminal-style text representation. The text can include newlines
          and ANSI escape sequences (e.g. for colors).
        Note:
            Make sure that your class's metadata 'render_modes' key includes
              the list of supported modes. It's recommended to call super()
              in implementations to use the functionality of this method.
        Args:
            mode (str): the mode to render with
        Example:
        class MyEnv(Env):
            metadata = {'render_modes': ['human', 'rgb_array']}
            def render(self, mode='human'):
                if mode == 'rgb_array':
                    return np.array(...) # return RGB frame suitable for video
                elif mode == 'human':
                    ... # pop up a window and render
                else:
                    super(MyEnv, self).render(mode=mode) # just raise an exception
        """
        raise NotImplementedError
   

   close

   def close(self):
       """Override close in your subclass to perform any necessary cleanup.
       Environments will automatically close() themselves when
       garbage collected or when the program exits.
       """
       pass
   

   seed

   def seed(self, seed=None):
       """Sets the seed for this env's random number generator(s).
       Note:
           Some environments use multiple pseudorandom number generators.
           We want to capture all such seeds used in order to ensure that
           there aren't accidental correlations between multiple generators.
       Returns:
           list<bigint>: Returns the list of seeds used in this env's random
             number generators. The first value in the list should be the
             "main" seed, or the value which a reproducer should pass to
             'seed'. Often, the main seed equals the provided 'seed', but
             this won't be true if seed=None, for example.
       """
       deprecation(
           "Function `env.seed(seed)` is marked as deprecated and will be removed in the future. "
           "Please use `env.reset(seed=seed) instead."
       )
       self._np_random, seed = seeding.np_random(seed)
       return [seed]
   

   有如下属性：
   action_space: The Space object corresponding to valid actions
   observation_space: The Space object corresponding to valid observations
   reward_range: A tuple corresponding to the min and max possible rewards

2. Wrapper类 class Wrapper(Env[ObsType, ActType]):
   装饰器模式，在不改变原有代码的基础上，添加或者更改一些功能。体现在修改原环境的类方法(reset step render等)或者类属性(action_space observation_space reward_range)，比如下面的CliffWalkingWrapper就是继承了Wrapper类，重写了用来可视化的render函数，部分代码如下：

   class Wrapper(Env[ObsType, ActType]):
       """Wraps the environment to allow a modular transformation.
       This class is the base class for all wrappers. The subclass could override
       some methods to change the behavior of the original environment without touching the
       original code.
       .. note::
           Don't forget to call ``super().__init__(env)`` if the subclass overrides :meth:`__init__`.
       """
   
       def __init__(self, env: Env):
           self.env = env
   
           self._action_space: spaces.Space | None = None
           self.