配置环境
重新搭建环境搞了两个多小时,真的无比慢。。。
注意要确保NumPy和Pandas版本兼容,一开始numpy搞了个2.1.2后面运行评测时报错了,修正后 我的环境是numpy1.26.4、pandas1.5.2
conda create -n opencompass python=3.10
conda activate opencompass
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia -y
# 注意:一定要先 cd /root
cd /root
git clone -b 0.2.4 https://github.com/open-compass/opencompass
cd opencompass
pip install -e .
apt-get update
apt-get install cmake
pip install -r requirements.txt
pip install protobuf
准备数据集
cp /share/temp/datasets/OpenCompassData-core-20231110.zip /root/opencompass/
unzip OpenCompassData-core-20231110.zip
查看相关模型配置文件
(opencompass) root@intern-studio-50006073:~/opencompass# python tools/list_configs.py internlm ceval
+----------------------------------------+----------------------------------------------------------------------+
| Model | Config Path |
|----------------------------------------+----------------------------------------------------------------------|
| hf_internlm2_1_8b | configs/models/hf_internlm/hf_internlm2_1_8b.py |
| hf_internlm2_20b | configs/models/hf_internlm/hf_internlm2_20b.py |
| hf_internlm2_7b | configs/models/hf_internlm/hf_internlm2_7b.py |
| hf_internlm2_base_20b | configs/models/hf_internlm/hf_internlm2_base_20b.py |
| hf_internlm2_base_7b | configs/models/hf_internlm/hf_internlm2_base_7b.py |
| hf_internlm2_chat_1_8b | configs/models/hf_internlm/hf_internlm2_chat_1_8b.py |
| hf_internlm2_chat_1_8b_sft | configs/models/hf_internlm/hf_internlm2_chat_1_8b_sft.py |
| hf_internlm2_chat_1_8b_test | configs/models/hf_internlm/hf_internlm2_chat_1_8b_test.py |
| hf_internlm2_chat_20b | configs/models/hf_internlm/hf_internlm2_chat_20b.py |
| hf_internlm2_chat_20b_sft | configs/models/hf_internlm/hf_internlm2_chat_20b_sft.py |
| hf_internlm2_chat_20b_with_system | configs/models/hf_internlm/hf_internlm2_chat_20b_with_system.py |
| hf_internlm2_chat_7b | configs/models/hf_internlm/hf_internlm2_chat_7b.py |
| hf_internlm2_chat_7b_sft | configs/models/hf_internlm/hf_internlm2_chat_7b_sft.py |
| hf_internlm2_chat_7b_with_system | configs/models/hf_internlm/hf_internlm2_chat_7b_with_system.py |
| hf_internlm2_chat_math_20b | configs/models/hf_internlm/hf_internlm2_chat_math_20b.py |
| hf_internlm2_chat_math_20b_with_system | configs/models/hf_internlm/hf_internlm2_chat_math_20b_with_system.py |
| hf_internlm2_chat_math_7b | configs/models/hf_internlm/hf_internlm2_chat_math_7b.py |
| hf_internlm2_chat_math_7b_with_system | configs/models/hf_internlm/hf_internlm2_chat_math_7b_with_system.py |
| hf_internlm_20b | configs/models/hf_internlm/hf_internlm_20b.py |
| hf_internlm_7b | configs/models/hf_internlm/hf_internlm_7b.py |
| hf_internlm_chat_20b | configs/models/hf_internlm/hf_internlm_chat_20b.py |
| hf_internlm_chat_7b | configs/models/hf_internlm/hf_internlm_chat_7b.py |
| hf_internlm_chat_7b_8k | configs/models/hf_internlm/hf_internlm_chat_7b_8k.py |
| hf_internlm_chat_7b_v1_1 | configs/models/hf_internlm/hf_internlm_chat_7b_v1_1.py |
| internlm_7b | configs/models/internlm/internlm_7b.py |
| lmdeploy_internlm2_chat_20b | configs/models/hf_internlm/lmdeploy_internlm2_chat_20b.py |
| lmdeploy_internlm2_chat_7b | configs/models/hf_internlm/lmdeploy_internlm2_chat_7b.py |
| ms_internlm_chat_7b_8k | configs/models/ms_internlm/ms_internlm_chat_7b_8k.py |
+----------------------------------------+----------------------------------------------------------------------+
+--------------------------------+------------------------------------------------------------------+
| Dataset | Config Path |
|--------------------------------+------------------------------------------------------------------|
| ceval_clean_ppl | configs/datasets/ceval/ceval_clean_ppl.py |
| ceval_contamination_ppl_810ec6 | configs/datasets/contamination/ceval_contamination_ppl_810ec6.py |
| ceval_gen | configs/datasets/ceval/ceval_gen.py |
| ceval_gen_2daf24 | configs/datasets/ceval/ceval_gen_2daf24.py |
| ceval_gen_5f30c7 | configs/datasets/ceval/ceval_gen_5f30c7.py |
| ceval_internal_ppl_1cd8bf | configs/datasets/ceval/ceval_internal_ppl_1cd8bf.py |
| ceval_ppl | configs/datasets/ceval/ceval_ppl.py |
| ceval_ppl_1cd8bf | configs/datasets/ceval/ceval_ppl_1cd8bf.py |
| ceval_ppl_578f8d | configs/datasets/ceval/ceval_ppl_578f8d.py |
| ceval_ppl_93e5ce | configs/datasets/ceval/ceval_ppl_93e5ce.py |
| ceval_zero_shot_gen_bd40ef | configs/datasets/ceval/ceval_zero_shot_gen_bd40ef.py |
+--------------------------------+------------------------------------------------------------------+
用命令行评测
配置以下代码到 /root/opencompass/configs/models/hf_internlm/的hf_internlm2_chat_1_8b.py
from opencompass.models import HuggingFaceCausalLM
models = [
dict(
type=HuggingFaceCausalLM,
abbr='internlm2-1.8b-hf',
path="/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b",
tokenizer_path='/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b',
model_kwargs=dict(
trust_remote_code=True,
device_map='auto',
),
tokenizer_kwargs=dict(
padding_side='left',
truncation_side='left',
use_fast=False,
trust_remote_code=True,
),
max_out_len=100,
min_out_len=1,
max_seq_len=2048,
batch_size=8,
run_cfg=dict(num_gpus=1, num_procs=1),
)
]
到opencompass文件夹下,运行命令
#环境变量配置
export MKL_SERVICE_FORCE_INTEL=1
#或
# export MKL_THREADING_LAYER=GNU
python run.py --datasets ceval_gen --models hf_internlm2_chat_1_8b --debug
看到进度条大概就是成功啦
(opencompass) root@intern-studio-50006073:~/opencompass# export MKL_SERVICE_FORCE_INTEL=1
(opencompass) root@intern-studio-50006073:~/opencompass# python run.py --datasets ceval_gen --models hf_internlm2_chat_1_8b_test --debug
09/12 23:49:47 - OpenCompass - INFO - Loading ceval_gen: configs/datasets/ceval/ceval_gen.py
09/12 23:49:47 - OpenCompass - INFO - Loading hf_internlm2_chat_1_8b_test: configs/models/hf_internlm/hf_internlm2_chat_1_8b_test.py
09/12 23:49:47 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
09/12 23:49:47 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
09/12 23:49:47 - OpenCompass - INFO - Partitioned into 1 tasks.
09/12 23:50:14 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-college_economics,internlm2-1.8b-hf/ceval-accountant,internlm2-1.8b-hf/ceval-tax_accountant,internlm2-1.8b-hf/ceval-physician,internlm2-1.8b-hf/ceval-civil_servant,internlm2-1.8b-hf/ceval-urban_and_rural_planner,internlm2-1.8b-hf/ceval-teacher_qualification,internlm2-1.8b-hf/ceval-college_programming,internlm2-1.8b-hf/ceval-electrical_engineer,internlm2-1.8b-hf/ceval-business_administration,internlm2-1.8b-hf/ceval-art_studies,internlm2-1.8b-hf/ceval-fire_engineer,internlm2-1.8b-hf/ceval-environmental_impact_assessment_engineer,internlm2-1.8b-hf/ceval-education_science,internlm2-1.8b-hf/ceval-professional_tour_guide,internlm2-1.8b-hf/ceval-college_chemistry,internlm2-1.8b-hf/ceval-metrology_engineer,internlm2-1.8b-hf/ceval-mao_zedong_thought,internlm2-1.8b-hf/ceval-law,internlm2-1.8b-hf/ceval-veterinary_medicine,internlm2-1.8b-hf/ceval-modern_chinese_history,internlm2-1.8b-hf/ceval-chinese_language_and_literature,internlm2-1.8b-hf/ceval-legal_professional,internlm2-1.8b-hf/ceval-logic,internlm2-1.8b-hf/ceval-middle_school_history,internlm2-1.8b-hf/ceval-plant_protection,internlm2-1.8b-hf/ceval-clinical_medicine,internlm2-1.8b-hf/ceval-computer_architecture,internlm2-1.8b-hf/ceval-middle_school_biology,internlm2-1.8b-hf/ceval-middle_school_politics,internlm2-1.8b-hf/ceval-middle_school_chemistry,internlm2-1.8b-hf/ceval-high_school_history,internlm2-1.8b-hf/ceval-computer_network,internlm2-1.8b-hf/ceval-operating_system,internlm2-1.8b-hf/ceval-college_physics,internlm2-1.8b-hf/ceval-advanced_mathematics,internlm2-1.8b-hf/ceval-high_school_physics,internlm2-1.8b-hf/ceval-high_school_chemistry,internlm2-1.8b-hf/ceval-high_school_biology,internlm2-1.8b-hf/ceval-middle_school_mathematics,internlm2-1.8b-hf/ceval-middle_school_physics,internlm2-1.8b-hf/ceval-marxism,internlm2-1.8b-hf/ceval-high_school_politics,internlm2-1.8b-hf/ceval-high_school_geography,internlm2-1.8b-hf/ceval-ideological_and_moral_cultivation,internlm2-1.8b-hf/ceval-high_school_chinese,internlm2-1.8b-hf/ceval-sports_science,internlm2-1.8b-hf/ceval-basic_medicine,internlm2-1.8b-hf/ceval-probability_and_statistics,internlm2-1.8b-hf/ceval-high_school_mathematics,internlm2-1.8b-hf/ceval-discrete_mathematics,internlm2-1.8b-hf/ceval-middle_school_geography]
Loading checkpoint shards: 100%|███████████████████████████████████████████| 2/2 [00:13<00:00, 6.90s/it]
09/12 23:51:00 - OpenCompass - INFO - Start inferencing [internlm2-1.8b-hf/ceval-college_economics]
100%|███████████████████████████████████████████████████████████████| 55/55 [00:00<00:00, 1469342.17it/s]
[2024-09-12 23:51:00,570] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
0%| | 0/7 [00:00<?, ?it/s]The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
71%|██████████████████████████████████████████████████ | 5/7 [02:15<00:53, 26.99s/it]
运行完毕后会看到1.8b模型在各种任务上的效果
20240912_132848
tabulate format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset version metric mode internlm2-1.8b-hf
---------------------------------------------- --------- ------------- ------ -------------------
ceval-computer_network db9ce2 accuracy gen 47.37
ceval-operating_system 1c2571 accuracy gen 47.37
ceval-computer_architecture a74dad accuracy gen 23.81
ceval-college_programming 4ca32a accuracy gen 27.03
ceval-college_physics 963fa8 accuracy gen 42.11
ceval-college_chemistry e78857 accuracy gen 37.5
ceval-advanced_mathematics ce03e2 accuracy gen 26.32
ceval-probability_and_statistics 65e812 accuracy gen 22.22
ceval-discrete_mathematics e894ae accuracy gen 25
ceval-electrical_engineer ae42b9 accuracy gen 27.03
ceval-metrology_engineer ee34ea accuracy gen 54.17
ceval-high_school_mathematics 1dc5bf accuracy gen 22.22
ceval-high_school_physics adf25f accuracy gen 42.11
ceval-high_school_chemistry 2ed27f accuracy gen 52.63
ceval-high_school_biology 8e2b9a accuracy gen 26.32
ceval-middle_school_mathematics bee8d5 accuracy gen 36.84
ceval-middle_school_biology 86817c accuracy gen 80.95
ceval-middle_school_physics 8accf6 accuracy gen 47.37
ceval-middle_school_chemistry 167a15 accuracy gen 80
ceval-veterinary_medicine b4e08d accuracy gen 43.48
ceval-college_economics f3f4e6 accuracy gen 32.73
ceval-business_administration c1614e accuracy gen 39.39
ceval-marxism cf874c accuracy gen 68.42
ceval-mao_zedong_thought 51c7a4 accuracy gen 70.83
ceval-education_science 591fee accuracy gen 55.17
ceval-teacher_qualification 4e4ced accuracy gen 59.09
ceval-high_school_politics 5c0de2 accuracy gen 57.89
ceval-high_school_geography 865461 accuracy gen 47.37
ceval-middle_school_politics 5be3e7 accuracy gen 76.19
ceval-middle_school_geography 8a63be accuracy gen 75
ceval-modern_chinese_history fc01af accuracy gen 52.17
ceval-ideological_and_moral_cultivation a2aa4a accuracy gen 73.68
ceval-logic f5b022 accuracy gen 31.82
ceval-law a110a1 accuracy gen 29.17
ceval-chinese_language_and_literature 0f8b68 accuracy gen 47.83
ceval-art_studies 2a1300 accuracy gen 42.42
ceval-professional_tour_guide 4e673e accuracy gen 51.72
ceval-legal_professional ce8787 accuracy gen 34.78
ceval-high_school_chinese 315705 accuracy gen 36.84
ceval-high_school_history 7eb30a accuracy gen 65
ceval-middle_school_history 48ab4a accuracy gen 86.36
ceval-civil_servant 87d061 accuracy gen 42.55
ceval-sports_science 70f27b accuracy gen 52.63
ceval-plant_protection 8941f9 accuracy gen 40.91
ceval-basic_medicine c409d6 accuracy gen 68.42
ceval-clinical_medicine 49e82d accuracy gen 36.36
ceval-urban_and_rural_planner 95b885 accuracy gen 52.17
ceval-accountant 002837 accuracy gen 36.73
ceval-fire_engineer bc23f5 accuracy gen 38.71
ceval-environmental_impact_assessment_engineer c64e2d accuracy gen 51.61
ceval-tax_accountant 3a5e3c accuracy gen 36.73
ceval-physician 6e277d accuracy gen 42.86
ceval-stem - naive_average gen 40.59
ceval-social-science - naive_average gen 58.21
ceval-humanities - naive_average gen 50.16
ceval-other - naive_average gen 45.43
ceval-hard - naive_average gen 33.76
ceval - naive_average gen 47.03
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
done✅