[复现] 逆强化学习 ICRL - Inverse Constrained Reinforcement Learning

Github - Inverse Constrained Reinforcement Learning

@InProceedings{pmlr-v139-malik21a,
  title = 	 {Inverse Constrained Reinforcement Learning},
  author =       {Shehryar Malik and Usman Anwar and Alireza Aghasi and Ali Ahmed},
  booktitle = 	 {Proceedings of the 38th International Conference on Machine Learning},
  pages = 	 {7390--7399},
  year = 	 {2021},
  editor = 	 {Meila, Marina and Zhang, Tong},
  volume = 	 {139},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {18--24 Jul},
  publisher =    {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v139/malik21a/malik21a.pdf},
  url = 	 {http://proceedings.mlr.press/v139/malik21a.html},
}

1. 安装mujoco200

下载：https://www.roboti.us/download.html

key：https://www.roboti.us/license.html

cp -r ~/Downloads/mujoco200_linux ~/.mujoco/
mv ~/.mujoco/mujoco200_linux ~/.mujoco/mujoco200
mv ~/Downloads/mjkey.txt ~/.mujoco/mujoco200/bin
mv ~/Downloads/mjkey.txt ~/.mujoco

gedit ~/.bashrc

export LD_LIBRARY_PATH=~/.mujoco/mujoco200/bin${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export MUJOCO_KEY_PATH=~/.mujoco${MUJOCO_KEY_PATH}

source ~/.bashrc

2. conda env

conda create -n icrl python=3.8
pip install 'mujoco-py<2.1,>=2.0'
pip install wandb==0.10.12 torch==1.5.0 gym==0.15.7 matplotlib==3.3.2 numpy==1.17.5 cloudpickle==1.2.2 tqdm pandas pillow psutil mpl-scatter-densit

报错：

Failed to build mujoco-py
ERROR: Could not build wheels for mujoco-py, which is required to install pyproject.toml-based projects

解决方法：

pip install mujoco_py==2.0.2.8

3. run

python run_me.py icrl -p ICRL-FE2 --group HC-ICRL -er 10 -ep icrl/expert_data/HCWithPos-New -tk 0.01 -cl 20 -bi 10 -ft 2e5 -ni 30 -tei HCWithPos-v0 -eei HCWithPosTest-v0 -clr 0.05 -aclr 0.9 -crc 0.5 -psis -ctkno 2.5

4. troubleshooting

（1）报错：

Cython.Compiler.Errors.CompileError: /home/rui/anaconda3/envs/icrl/lib/python3.8/site-packages/mujoco_py/cymj.pyx

原因：Cython 3.0.10 版本太高             

解决方法：

pip uninstall cython
pip install cython==0.29.21

（2）报错：

fatal error: GL/osmesa.h: No such file or directory
    1 | #include <GL/osmesa.h>
      |          ^~~~~~~~~~~~~
compilation terminated.

...

distutils.errors.CompileError: command '/usr/bin/gcc' failed with exit code 1

ubuntu 20.04 带 gcc 9

可以确认一下：

gcc --version

 解决：

sudo apt install libosmesa6-dev

（3）报错：

[Errno 2] No such file or directory: 'patchelf'

解决：

sudo apt-get install -y patchelf

（4）grid world这个实验

命令如下：

python run_me.py icrl -p ICRL-FE2 --group LapGrid-ICRL -er 20 -ep icrl/expert_data/LGW -tei LGW-v0 -eei CLGW-v0 -tk 0.01 -cl 20 -clr 0.003 -ft 0.5e5 -ni 10 -bi 20 -dno -dnr -dnc

报错：

FileNotFoundError: [Errno 2] No such file or directory: 'icrl/expert_data/LGW/files/EXPERT/rollouts/0.pkl'
Wrapping eval env in a VecNormalize.

github也有提这个issue

但是作者没有回复

可以试试其他的。。。