《Alleviating Matthew Effect of Offline Reinforcement Learningin Interactive Recommendation》代码复现笔记

目录

一、代码跑通情况

 第一部分：Examples to run the code

1： Run traditional DQN-based baselines (e.g., BCQ or SQN)

 step1：python run_worldModel_ensemble.py --env CoatEnv-v0  --cuda 0 --epoch 5 --tau 0 --loss "pointneg" --message "pointneg"

step2 ：python run_Policy_BCQ.py  --env CoatEnv-v0  --seed 0 --cuda 0    --which_tracker avg --reward_handle "cat"  --unlikely-action-threshold 0.6 --window_size 3 --read_message "pointneg"  --message "BCQ"

2：Run model-based method, i.e., DORL, MBPO, and MOPO.

 step1：python run_worldModel_ensemble.py --env KuaiEnv-v0  --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg"

 step2：

 第二部分：Reproduce the results of the paper

 Step 1: Re-running the Experiments

 1. Train the static user models as well as static baseline methods.

 For KuaiEnv:

 For KuaiRand:

  2. Learn the RL policies.

 For KuaiEnv:

 For KuaiRand:

 二、可视化结果

---

一、代码跑通情况

开源代码一般在git上都有使用教程，可以参考git上的教程，不要上来就去pycharm上运行。此论文的代码说明所在位置的链接为：https://github.com/chongminggao/DORL-codes?utm_source=catalyzex.com

 第一部分：Examples to run the code

1： Run traditional DQN-based baselines (e.g., BCQ or SQN)

 step1：python run_worldModel_ensemble.py --env CoatEnv-v0  --cuda 0 --epoch 5 --tau 0 --loss "pointneg" --message "pointneg"

step2 ：python run_Policy_BCQ.py  --env CoatEnv-v0  --seed 0 --cuda 0    --which_tracker avg --reward_handle "cat"  --unlikely-action-threshold 0.6 --window_size 3 --read_message "pointneg"  --message "BCQ"

 200轮完成后

2：Run model-based method, i.e., DORL, MBPO, and MOPO.

 step1：python run_worldModel_ensemble.py --env KuaiEnv-v0  --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg"

 五轮，每轮epoch-1-4完成

 step2：①②③

①Run DORL (Our proposed method)

python run_Policy_Main.py --env KuaiEnv-v0 --seed 0 --cuda 0 --num_leave_compute 1 --leave_threshold 0 --which_tracker avg --reward_handle "cat" --lambda_variance 0.01 --lambda_entropy 0.05 --window_size 3 --read_message "pointneg" --message "DORL" &

此处会报错，将&删除就可以正常运行了

 ②Run MBPO (the vanilla model-based offline RL method):

 python run_Policy_Main.py --env KuaiEnv-v0  --seed 0 --cuda 0   --num_leave_compute 1 --leave_threshold 0 --which_tracker avg --reward_handle "cat" --lambda_variance 0   --lambda_entropy 0    --window_size 3 --read_message "pointneg"  --message "MBPO" &

 ③Run MOPO (the model-based RL method considering distributional shift):

 python run_Policy_Main.py --env KuaiEnv-v0  --seed 0 --cuda 0   --num_leave_compute 1 --leave_threshold 0 --which_tracker avg --reward_handle "cat" --lambda_variance 0.05 --lambda_entropy 0    --window_size 3 --read_message "pointneg"  --message "MOPO" &

报错：内存分配出错，尝试同时执行两条指令，不大行

---

2024.3.17 再次连接算力云平台，连接到本地时下载包出错，暂时不管它。 

 已成功安装
Successfully installed absl-py-2.1.0 arrow-1.3.0 astunparse-1.6.3 async-lru-2.0.4 attrs-23.2.0 babel-2.14.0 bcrypt-4.1.2 beautifulsoup4-4.12.3 charset-normalizer-3.3.2 cloudpickle-3.0.0 fastjsonschema-2.19.1 filelock-3.13.1 flatbuffers-24.3.7 fqdn-1.5.1 fsspec-2024.3.0 gast-0.4.0 google-auth-2.28.2 google-auth-oauthlib-1.0.0 google-pasta-0.2.0 grpcio-1.62.1 gym-0.26.2 gym-notices-0.0.8 h11-0.14.0 h5py-3.10.0 httpcore-1.0.4 httpx-0.27.0 importlib-metadata-7.0.2 importlib-resources-6.3.1 isoduration-20.11.0 jax-0.4.13 jinja2-3.1.3 joblib-1.3.2 jsonpointer-2.4 jsonschema-4.21.1 jsonschema-specifications-2023.12.1 jupyter-client-7.4.9 jupyter-core-5.7.2 jupyter-events-0.9.1 jupyter-lsp-2.2.4 jupyter-server-2.13.0 jupyter-server-terminals-0.5.3 jupyterlab-4.1.5 jupyterlab-server-2.25.4 keras-2.12.0 libclang-16.0.6 llvmlite-0.41.1 logzero-1.7.0 mistune-3.0.2 ml-dtypes-0.2.0 mpmath-1.3.0 nbconvert-7.16.2 nbformat-5.10.3 nest-asyncio-1.6.0 networkx-3.1 notebook-shim-0.2.4 numba-0.58.1 numpy-1.23.5 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.4.99 nvidia-nvtx-cu12-12.1.105 opt-einsum-3.3.0 overrides-7.7.0 packaging-24.0 pandas-2.0.3 paramiko-3.4.0 pkgutil-resolve-name-1.3.10 platformdirs-4.2.0 protobuf-4.25.3 pynacl-1.5.0 python-json-logger-2.0.7 pyyaml-6.0.1 pyzmq-25.1.2 referencing-0.34.0 requests-2.31.0 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rpds-py-0.18.0 scikit-learn-1.3.2 scipy-1.10.1 seaborn-0.13.2 send2trash-1.8.2 soupsieve-2.5 sympy-1.12 tensorboard-2.12.3 tensorboard-data-server-0.7.2 tensorflow-2.12.0 tensorflow-estimator-2.12.0 tensorflow-io-gcs-filesystem-0.34.0 termcolor-2.4.0 threadpoolctl-3.3.0 tinycss2-1.2.1 tomli-2.0.1 torch-2.2.1 tornado-6.4 tqdm-4.66.2 traitlets-5.14.2 triton-2.2.0 types-python-dateutil-2.9.0.20240316 typing-extensions-4.10.0 tzdata-2024.1 uri-template-1.3.0 webcolors-1.13 wrapt-1.14.1 zipp-3.18.1
 

---

 第二部分：Reproduce the results of the paper

 Step 1: Re-running the Experiments

 1. Train the static user models as well as static baseline methods.

 For KuaiEnv:

 python run_worldModel_IPS.py --env KuaiEnv-v0  --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py         --env KuaiEnv-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 200 --seed 0 --cuda 0 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py  --env KuaiEnv-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 200 --seed 0 --cuda 1 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiEnv-v0  --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &

 因为设备问题（仅开了一个单个GPU的4090，很贵），此处在真正测试时使用的是

 python run_worldModel_IPS.py --env KuaiEnv-v0  --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py         --env KuaiEnv-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py  --env KuaiEnv-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiEnv-v0  --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &

下面分别是第1 2 3 4条指令执行完成后的截图以及其产生的文件

   user_model and its parameters have been saved in ./saved_models/KuaiEnv-v0/DeepFM-IPS

 user_model and its parameters have been saved in ./saved_models/KuaiEnv-v0/DeepFM

  此处的connect……（2）（3）（4）代表的是第一部分四条指令中的第几条指令

 For KuaiRand:

 python run_worldModel_IPS.py --env KuaiRand-v0  --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py         --env KuaiRand-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 200 --seed 0 --cuda 2 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py  --env KuaiRand-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 200 --seed 0 --cuda 3 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiRand-v0  --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &

依旧因为设备问题（仅开了一个单个GPU的4090，很贵），此处在真正测试时使用的是

python run_worldModel_IPS.py --env KuaiRand-v0  --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py         --env KuaiRand-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py  --env KuaiRand-v0  --num_leave_compute 4  --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiRand-v0  --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &

 下面分别是第1 2 3 4条指令执行完成后的截图以及其产生的文件

 user_model and its parameters have been saved in ./saved_models/KuaiRand-v0/DeepFM-IPS

 

user_model and its parameters have been saved in ./saved_models/KuaiRand-v0/DeepFM

  2. Learn the RL policies.

 For KuaiEnv:

因为只开了一个GPU，所以此处的cuda全部设为cuda 0

前5条指令生成了一下五个文件，该5个文件夹下是log日志文件

 

 后3条指令在A2C_with_emb下生成了三个log文件

 For KuaiRand:

因为只开了一个GPU，所以此处的cuda全部设为cuda 0

前5条指令生成了一下五个文件，该5个文件夹下是log日志文件

下面是后三条指令

  后3条指令在A2C_with_emb下生成了三个log文件

 有些指令在执行时会先创建文件夹，以便存放后续生成的日志文件或者是其他文件，第1、2、3、4、5条和第6条指令都有创建文件夹的操作。

二、代码执行完毕，可视化结果（论文中的图7）

整理代码执行后生成的log文件并进行可视化可以得到：