一、下载评测工具lm-evaluation-harness-master
git clone https://github.com/EleutherAI/lm-evaluation-harness
git clone https://huggingface.co/datasets/ceval/ceval-exam/tree/main
获取评测工程及ceval-exam中文测试集保存至本地
二、工具及测试集拷贝至Llama2-Chinese的镜像内部
docker cp /root/lm-evaluation-harness container-id:/root/Llama2-Chinese/
路径如下图
三、lm-evaluation-harness-master对Ceval-exam测试集评估
CUDA_VISIBLE_DEVICES=1 python /root/lm-evaluation-harness-master/main.py --model hf --model_args pretrained=/root/Llama2-Chinese/Llama2-Chinese-7b-Chat --tasks Ceval-valid-accountant --output_base_path ./results/llama/ceval --write_out --output_path ./results/llama/ceval_result_5shot.json
3.1 报错raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
原因:根据报错信息,由于访问外网失败获取不到指定数据集路径
解决:
1)根据提示修改/root/Llama2-Chinese/lm-evaluation-harness-master/lm_eval/base.py 第542行,由注释可知,设置测试集为镜像本地路径时,需要修改传参
def download(self, data_dir="/root/Llama2-Chinese/ceval-exam", cache_dir=None, download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS):
2)修改/root/Llama2-Chinese/lm-evaluation-harness-master/lm_eval/tasks/ceval.py 第97行,设置dataset_path为测试集的绝对路径
至此3.1报错的获取本地数据集路径问题解决
3.2 报错raise DatasetGenerationError("An error occurred while generating the dataset") from e
解决中······