目录
1: Run traditional DQN-based baselines (e.g., BCQ or SQN)
2:Run model-based method, i.e., DORL, MBPO, and MOPO.
第二部分:Reproduce the results of the paper
Step 1: Re-running the Experiments
1. Train the static user models as well as static baseline methods.
一、代码跑通情况
开源代码一般在git上都有使用教程,可以参考git上的教程,不要上来就去pycharm上运行。此论文的代码说明所在位置的链接为:https://github.com/chongminggao/DORL-codes?utm_source=catalyzex.com
第一部分:Examples to run the code
1: Run traditional DQN-based baselines (e.g., BCQ or SQN)
step1:python run_worldModel_ensemble.py --env CoatEnv-v0 --cuda 0 --epoch 5 --tau 0 --loss "pointneg" --message "pointneg"
step2 :python run_Policy_BCQ.py --env CoatEnv-v0 --seed 0 --cuda 0 --which_tracker avg --reward_handle "cat" --unlikely-action-threshold 0.6 --window_size 3 --read_message "pointneg" --message "BCQ"
200轮完成后
2:Run model-based method, i.e., DORL, MBPO, and MOPO.
step1:python run_worldModel_ensemble.py --env KuaiEnv-v0 --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg"
五轮,每轮epoch-1-4完成
step2:①②③
①Run DORL (Our proposed method)
python run_Policy_Main.py --env KuaiEnv-v0 --seed 0 --cuda 0 --num_leave_compute 1 --leave_threshold 0 --which_tracker avg --reward_handle "cat" --lambda_variance 0.01 --lambda_entropy 0.05 --window_size 3 --read_message "pointneg" --message "DORL" &
此处会报错,将&删除就可以正常运行了
②Run MBPO (the vanilla model-based offline RL method):
python run_Policy_Main.py --env KuaiEnv-v0 --seed 0 --cuda 0 --num_leave_compute 1 --leave_threshold 0 --which_tracker avg --reward_handle "cat" --lambda_variance 0 --lambda_entropy 0 --window_size 3 --read_message "pointneg" --message "MBPO" &
③Run MOPO (the model-based RL method considering distributional shift):
python run_Policy_Main.py --env KuaiEnv-v0 --seed 0 --cuda 0 --num_leave_compute 1 --leave_threshold 0 --which_tracker avg --reward_handle "cat" --lambda_variance 0.05 --lambda_entropy 0 --window_size 3 --read_message "pointneg" --message "MOPO" &
报错:内存分配出错,尝试同时执行两条指令,不大行
2024.3.17 再次连接算力云平台,连接到本地时下载包出错,暂时不管它。
已成功安装
Successfully installed absl-py-2.1.0 arrow-1.3.0 astunparse-1.6.3 async-lru-2.0.4 attrs-23.2.0 babel-2.14.0 bcrypt-4.1.2 beautifulsoup4-4.12.3 charset-normalizer-3.3.2 cloudpickle-3.0.0 fastjsonschema-2.19.1 filelock-3.13.1 flatbuffers-24.3.7 fqdn-1.5.1 fsspec-2024.3.0 gast-0.4.0 google-auth-2.28.2 google-auth-oauthlib-1.0.0 google-pasta-0.2.0 grpcio-1.62.1 gym-0.26.2 gym-notices-0.0.8 h11-0.14.0 h5py-3.10.0 httpcore-1.0.4 httpx-0.27.0 importlib-metadata-7.0.2 importlib-resources-6.3.1 isoduration-20.11.0 jax-0.4.13 jinja2-3.1.3 joblib-1.3.2 jsonpointer-2.4 jsonschema-4.21.1 jsonschema-specifications-2023.12.1 jupyter-client-7.4.9 jupyter-core-5.7.2 jupyter-events-0.9.1 jupyter-lsp-2.2.4 jupyter-server-2.13.0 jupyter-server-terminals-0.5.3 jupyterlab-4.1.5 jupyterlab-server-2.25.4 keras-2.12.0 libclang-16.0.6 llvmlite-0.41.1 logzero-1.7.0 mistune-3.0.2 ml-dtypes-0.2.0 mpmath-1.3.0 nbconvert-7.16.2 nbformat-5.10.3 nest-asyncio-1.6.0 networkx-3.1 notebook-shim-0.2.4 numba-0.58.1 numpy-1.23.5 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.4.99 nvidia-nvtx-cu12-12.1.105 opt-einsum-3.3.0 overrides-7.7.0 packaging-24.0 pandas-2.0.3 paramiko-3.4.0 pkgutil-resolve-name-1.3.10 platformdirs-4.2.0 protobuf-4.25.3 pynacl-1.5.0 python-json-logger-2.0.7 pyyaml-6.0.1 pyzmq-25.1.2 referencing-0.34.0 requests-2.31.0 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rpds-py-0.18.0 scikit-learn-1.3.2 scipy-1.10.1 seaborn-0.13.2 send2trash-1.8.2 soupsieve-2.5 sympy-1.12 tensorboard-2.12.3 tensorboard-data-server-0.7.2 tensorflow-2.12.0 tensorflow-estimator-2.12.0 tensorflow-io-gcs-filesystem-0.34.0 termcolor-2.4.0 threadpoolctl-3.3.0 tinycss2-1.2.1 tomli-2.0.1 torch-2.2.1 tornado-6.4 tqdm-4.66.2 traitlets-5.14.2 triton-2.2.0 types-python-dateutil-2.9.0.20240316 typing-extensions-4.10.0 tzdata-2024.1 uri-template-1.3.0 webcolors-1.13 wrapt-1.14.1 zipp-3.18.1
第二部分:Reproduce the results of the paper
Step 1: Re-running the Experiments
1. Train the static user models as well as static baseline methods.
For KuaiEnv:
python run_worldModel_IPS.py --env KuaiEnv-v0 --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py --env KuaiEnv-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 200 --seed 0 --cuda 0 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py --env KuaiEnv-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 200 --seed 0 --cuda 1 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiEnv-v0 --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &
因为设备问题(仅开了一个单个GPU的4090,很贵),此处在真正测试时使用的是
python run_worldModel_IPS.py --env KuaiEnv-v0 --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py --env KuaiEnv-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py --env KuaiEnv-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiEnv-v0 --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &
下面分别是第1 2 3 4条指令执行完成后的截图以及其产生的文件
user_model and its parameters have been saved in ./saved_models/KuaiEnv-v0/DeepFM-IPS
user_model and its parameters have been saved in ./saved_models/KuaiEnv-v0/DeepFM
此处的connect……(2)(3)(4)代表的是第一部分四条指令中的第几条指令
For KuaiRand:
python run_worldModel_IPS.py --env KuaiRand-v0 --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py --env KuaiRand-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 200 --seed 0 --cuda 2 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py --env KuaiRand-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 200 --seed 0 --cuda 3 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiRand-v0 --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &
依旧因为设备问题(仅开了一个单个GPU的4090,很贵),此处在真正测试时使用的是
python run_worldModel_IPS.py --env KuaiRand-v0 --seed 0 --cuda 0 --loss "pointneg" --message "DeepFM-IPS" &
python run_linUCB.py --env KuaiRand-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "UCB" &
python run_epsilongreedy.py --env KuaiRand-v0 --num_leave_compute 4 --leave_threshold 0 --epoch 10 --seed 0 --cuda 0 --loss "pointneg" --message "epsilon-greedy" &
python run_worldModel_ensemble.py --env KuaiRand-v0 --cuda 0 --epoch 5 --loss "pointneg" --message "pointneg" &
下面分别是第1 2 3 4条指令执行完成后的截图以及其产生的文件
user_model and its parameters have been saved in ./saved_models/KuaiRand-v0/DeepFM-IPS
user_model and its parameters have been saved in ./saved_models/KuaiRand-v0/DeepFM
2. Learn the RL policies.
For KuaiEnv:
因为只开了一个GPU,所以此处的cuda全部设为cuda 0
前5条指令生成了一下五个文件,该5个文件夹下是log日志文件
后3条指令在A2C_with_emb下生成了三个log文件
For KuaiRand:
因为只开了一个GPU,所以此处的cuda全部设为cuda 0
前5条指令生成了一下五个文件,该5个文件夹下是log日志文件
下面是后三条指令
后3条指令在A2C_with_emb下生成了三个log文件
有些指令在执行时会先创建文件夹,以便存放后续生成的日志文件或者是其他文件,第1、2、3、4、5条和第6条指令都有创建文件夹的操作。
二、代码执行完毕,可视化结果(论文中的图7)
整理代码执行后生成的log文件并进行可视化可以得到: