PettingZoo:多智能体强化学习的标准API

PettingZoo: A Standard API for Multi-Agent Reinforcement Learning

PettingZoo:多智能体强化学习的标准API

目录

Abstract  摘要

1 Introduction

1 介绍

2 Background and Related Works

2 背景及相关工作

2.1 Partially Observable Stochastic Games and RLlib

2.1部分可观察随机对策与RLlib

2.2 OpenSpiel and Extensive Form Games

2.2 OpenSpiel和Extensive Form游戏

3 PettingZoo Design Goals

3 PettingZoo设计目标

3.1 Be like Gym

3.2 Be a Universal API

3.2 成为通用API

4 Case Studies of Problems With The POSG Model in MARL

4 MARL中POSG模型问题的案例研究

4.1 POSGs Don’t Allow Access To Information You Should Have

4.1 posg不允许访问您应该拥有的信息

4.2 POSGs Based APIs Are Not Conceptually Clear For Games Implemented In Code

4.2 基于posg的api对于代码实现的游戏在概念上并不清晰

5 The Agent Environment Cycle Games Model

5 代理环境循环博弈模型

6 API Design

6.1 Basic API

6.1基本接口

6.2 The agent_iter Method

6.2 agent_iter方法

6.3 The last Metho

6.3 最后一种方法

6.4 Additional API Features

6.4 其他API特性

6.5 Environment Creation and the Parallel API

6.5 环境创建和并行API

7 Default Environments

7 默认环境

8 Adoption

9 Conclusion

9 结论

Abstract  摘要

        This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle (“AEC”) games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning (“MARL”), by making work more interchangeable, accessible and reproducible akin to what OpenAI’s Gym library did for single-agent reinforcement learning. PettingZoo’s API, while inheriting many features of Gym, is unique amongst MARL APIs in that it’s based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of games commonly used in MARL and accordingly can promote confusing bugs that are hard to detect, and that the AEC games model addresses these problems.

        本文介绍了PettingZoo库及其附带的Agent Environment Cycle(“AEC”)博弈模型。PettingZoo是一个包含多种多代理环境集的库，具有通用的、优雅的Python API。PettingZoo的开发目标是加速多智能体强化学习(“MARL”)的研究，通过使工作更具互换性、可访问性和可重复性，类似于OpenAI的Gym库为单智能体强化学习所做的工作。PettingZoo的API虽然继承了Gym的许多功能，但在MARL API中是独一无二的，因为它基于新颖的AEC游戏模型。我们认为，在某种程度上，通过对流行MARL环境中主要问题的案例研究，流行的游戏模型是MARL中常用的游戏的糟糕概念模型，因此可能导致难以检测的混淆错误，而AEC游戏模型解决了这些问题。

1 Introduction

1 介绍

        Multi-Agent Reinforcement Learning (MARL) has been behind many of the most publicized achievements of modern machine learning — AlphaGo Zero [Silver et al, 2017], OpenAI Five [OpenAI, 2018], AlphaStar [Vinyals et al, 2019]. These achievements motivated a boom in MARL research, with Google Scholar indexing 9,480 new papers discussing multi-agent reinforcement learning in 2020 alone. Despite this boom, conducting research in MARL remains a significant engineering challenge. A large part of this is because, unlike single agent reinforcement learning which has OpenAI’s Gym, no de facto standard API exists in MARL for how agents interface with environments.

        多智能体强化学习(MARL)是现代机器学习中许多最广为人知的成就的背后——AlphaGo Zero [Silver等人，2017]，OpenAI Five [OpenAI, 2018]， AlphaStar [Vinyals等人，2019]。这些成就推动了MARL研究的繁荣，仅在2020年，Google Scholar就索引了9480篇讨论多智能体强化学习的新论文。尽管如此，在MARL中进行研究仍然是一个重大的工程挑战。这在很大程度上是因为，与OpenAI的Gym的单代理强化学习不同，MARL中没有关于代理如何与环境交互的事实上的标准API。

        This makes the reuse of existing learning code for new purposes require substantial effort, consuming researchers’ time and preventing more thorough comparisons in research. This lack of a standardized API has also prevented the proliferation of learning libraries in MARL. While a massive number of Gym-based single-agent reinforcement learning libraries or code bases exist (as a rough measure 669 pip-installable packages depend on it at the time of writing GitHub [2021]), only 5 MARL libraries with large user bases exist [Lanctot et al, 2019, Weng et al, 2020, Liang et al, 2018, Samvelyan et al, 2019, Nota, 2020]. The proliferation of these Gym based learning libraries has pr