学会使用强化学习框架Acme(1)

ACME是一个分布式深度强化学习框架，可以在这个框架内创建、测试、调试新智能体。不用管RL 算法如何编写，因为Deepmind已经做好了。

框架论文：

https://arxiv.org/abs/2006.00979

跟着官方介绍一步一步来，框架的github：

https://github.com/deepmind/acmehttps://github.com/deepmind/acme

首先是这几行指令：

python3 -m venv acme
source acme/bin/activate
pip install --upgrade pip setuptools wheel

官方建议ACME安装在虚拟环境中，前两行代码就是做这个工作的。

创建虚拟环境的话我这里和官方不一样，用的anaconda创建：

(base) jw@Z8:~$ conda create -n acme python==3.8

(base) jw@Z8:~$ conda activate acme
(acme) jw@Z8:~$

这样一个用于安装acme的虚拟环境就创建好了， 并且进到环境中了。

第三行代码

pip install --upgrade pip setuptools wheel

看不太懂，搜到这两个解释

Python包工具setuptools、wheel、pip的详解 | 霍小强博客

【旁门Python 01】什么是wheel包，如何去用它？_深度学习深度近视的博客-CSDN博客_wheel包

 setupstools是构建和安装python模块用的，wheel是已编译发行版的一种格式。

运行代码：

(acme) jw@Z8:~$ pip install --upgrade pip setuptools wheel
Requirement already satisfied: pip in ./anaconda3/envs/acme/lib/python3.8/site-packages (21.2.4)
Collecting pip
  Downloading pip-22.0.4-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 1.1 MB/s 
Requirement already satisfied: setuptools in ./anaconda3/envs/acme/lib/python3.8/site-packages (61.2.0)
Collecting setuptools
  Using cached setuptools-62.1.0-py3-none-any.whl (1.1 MB)
Requirement already satisfied: wheel in ./anaconda3/envs/acme/lib/python3.8/site-packages (0.37.1)
Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 61.2.0
    Uninstalling setuptools-61.2.0:
      Successfully uninstalled setuptools-61.2.0
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-22.0.4 setuptools-62.1.0

从结果可以看出，这行代码是安装pip和setupstools的。程序检测到虚拟环境有pip setuptools wheel，卸载了原有的setuptools-61.2.0和pip-21.2.4，安装了setuptools-62.1.0和pip-22.0.4

上面那行指令使用

pip install --upgrade

来安装 pip、setuptools、wheel这三个包，通过研究代码运行结果搞懂了。

接下来要安装acme了

pip install dm-acme[jax,tensorflow]

 为了运行智能体，要安装jax或者tensorflow（毕竟是谷歌研究的框架）。

运行代码：

(acme) jw@Z8:~$ pip install dm-acme[tensorflow]
Collecting dm-acme[tensorflow]
  Using cached dm_acme-0.4.0-py3-none-any.whl
WARNING: dm-acme 0.4.0 does not provide the extra 'tensorflow'
Collecting dm-env
  Using cached dm_env-1.5-py3-none-any.whl (26 kB)
Collecting numpy
  Using cached numpy-1.22.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting dm-tree
  Using cached dm_tree-0.1.7-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (142 kB)
Collecting dm-launchpad==0.5.0
  Using cached dm_launchpad-0.5.0-cp38-cp38-manylinux2010_x86_64.whl (3.8 MB)
Collecting absl-py
  Using cached absl_py-1.0.0-py3-none-any.whl (126 kB)
Collecting pillow
  Using cached Pillow-9.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Collecting typing-extensions
  Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting cloudpickle
  Using cached cloudpickle-2.0.0-py3-none-any.whl (25 kB)
Collecting termcolor
  Using cached termcolor-1.1.0-py3-none-any.whl
Collecting mock
  Using cached mock-4.0.3-py3-none-any.whl (28 kB)
Collecting grpcio
  Using cached grpcio-1.44.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Collecting protobuf
  Using cached protobuf-3.20.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting portpicker
  Using cached portpicker-1.5.0-py3-none-any.whl (14 kB)
Collecting psutil
  Using cached psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: termcolor, dm-tree, typing-extensions, six, psutil, protobuf, pillow, numpy, mock, cloudpickle, portpicker, grpcio, absl-py, dm-launchpad, dm-env, dm-acme
Successfully installed absl-py-1.0.0 cloudpickle-2.0.0 dm-acme-0.4.0 dm-env-1.5 dm-launchpad-0.5.0 dm-tree-0.1.7 grpcio-1.44.0 mock-4.0.3 numpy-1.22.3 pillow-9.1.0 portpicker-1.5.0 protobuf-3.20.0 psutil-5.9.0 six-1.16.0 termcolor-1.1.0 typing-extensions-4.2.0

还是看一下运行结果，第二行收集dm-acme[tensorflow]，第三行的意思是使用本地缓存（因为我之前安装过acme），第四行的WARNING提醒我们安装acme0.4.0不会安装额外的tensorflow。