【书生浦语实战】opencompass评测internlm-chat1.8b模型

配置环境

重新搭建环境搞了两个多小时,真的无比慢。。。

注意要确保NumPy和Pandas版本兼容,一开始numpy搞了个2.1.2后面运行评测时报错了,修正后 我的环境是numpy1.26.4、pandas1.5.2

conda create -n opencompass python=3.10
conda activate opencompass
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia -y

# 注意:一定要先 cd /root
cd /root
git clone -b 0.2.4 https://github.com/open-compass/opencompass
cd opencompass
pip install -e .


apt-get update
apt-get install cmake
pip install -r requirements.txt
pip install protobuf

准备数据集

cp /share/temp/datasets/OpenCompassData-core-20231110.zip /root/opencompass/
unzip OpenCompassData-core-20231110.zip

查看相关模型配置文件

(opencompass) root@intern-studio-50006073:~/opencompass# python tools/list_configs.py internlm ceval
+----------------------------------------+----------------------------------------------------------------------+
| Model                                  | Config Path                                                          |
|----------------------------------------+----------------------------------------------------------------------|
| hf_internlm2_1_8b                      | configs/models/hf_internlm/hf_internlm2_1_8b.py                      |
| hf_internlm2_20b                       | configs/models/hf_internlm/hf_internlm2_20b.py                       |
| hf_internlm2_7b                        | configs/models/hf_internlm/hf_internlm2_7b.py                        |
| hf_internlm2_base_20b                  | configs/models/hf_internlm/hf_internlm2_base_20b.py                  |
| hf_internlm2_base_7b                   | configs/models/hf_internlm/hf_internlm2_base_7b.py                   |
| hf_internlm2_chat_1_8b                 | configs/models/hf_internlm/hf_internlm2_chat_1_8b.py                 |
| hf_internlm2_chat_1_8b_sft             | configs/models/hf_internlm/hf_internlm2_chat_1_8b_sft.py             |
| hf_internlm2_chat_1_8b_test            | configs/models/hf_internlm/hf_internlm2_chat_1_8b_test.py            |
| hf_internlm2_chat_20b                  | configs/models/hf_internlm/hf_internlm2_chat_20b.py                  |
| hf_internlm2_chat_20b_sft              | configs/models/hf_internlm/hf_internlm2_chat_20b_sft.py              |
| hf_internlm2_chat_20b_with_system      | configs/models/hf_internlm/hf_internlm2_chat_20b_with_system.py      |
| hf_internlm2_chat_7b                   | configs/models/hf_internlm/hf_internlm2_chat_7b.py                   |
| hf_internlm2_chat_7b_sft               | configs/models/hf_internlm/hf_internlm2_chat_7b_sft.py               |
| hf_internlm2_chat_7b_with_system       | configs/models/hf_internlm/hf_internlm2_chat_7b_with_system.py       |
| hf_internlm2_chat_math_20b             | configs/models/hf_internlm/hf_internlm2_chat_math_20b.py             |
| hf_internlm2_chat_math_20b_with_system | configs/models/hf_internlm/hf_internlm2_chat_math_20b_with_system.py |
| hf_internlm2_chat_math_7b              | configs/models/hf_internlm/hf_internlm2_chat_math_7b.py              |
| hf_internlm2_chat_math_7b_with_system  | configs/models/hf_internlm/hf_internlm2_chat_math_7b_with_system.py  |
| hf_internlm_20b                        | configs/models/hf_internlm/hf_internlm_20b.py                        |
| hf_internlm_7b                         | configs/models/hf_internlm/hf_internlm_7b.py                         |
| hf_internlm_chat_20b                   | configs/models/hf_internlm/hf_internlm_chat_20b.py                   |
| hf_internlm_chat_7b                    | configs/models/hf_internlm/hf_internlm_chat_7b.py                    |
| hf_internlm_chat_7b_8k                 | configs/models/hf_internlm/hf_internlm_chat_7b_8k.py                 |
| hf_internlm_chat_7b_v1_1               | configs/models/hf_internlm/hf_internlm_chat_7b_v1_1.py               |
| internlm_7b                            | configs/models/internlm/internlm_7b.py                               |
| lmdeploy_internlm2_chat_20b            | configs/models/hf_internlm/lmdeploy_internlm2_chat_20b.py            |
| lmdeploy_internlm2_chat_7b             | configs/models/hf_internlm/lmdeploy_internlm2_chat_7b.py             |
| ms_internlm_chat_7b_8k                 | configs/models/ms_internlm/ms_internlm_chat_7b_8k.py                 |
+----------------------------------------+----------------------------------------------------------------------+
+--------------------------------+------------------------------------------------------------------+
| Dataset                        | Config Path                                                      |
|--------------------------------+------------------------------------------------------------------|
| ceval_clean_ppl                | configs/datasets/ceval/ceval_clean_ppl.py                        |
| ceval_contamination_ppl_810ec6 | configs/datasets/contamination/ceval_contamination_ppl_810ec6.py |
| ceval_gen                      | configs/datasets/ceval/ceval_gen.py                              |
| ceval_gen_2daf24               | configs/datasets/ceval/ceval_gen_2daf24.py                       |
| ceval_gen_5f30c7               | configs/datasets/ceval/ceval_gen_5f30c7.py                       |
| ceval_internal_ppl_1cd8bf      | configs/datasets/ceval/ceval_internal_ppl_1cd8bf.py              |
| ceval_ppl                      | configs/datasets/ceval/ceval_ppl.py                              |
| ceval_ppl_1cd8bf               | configs/datasets/ceval/ceval_ppl_1cd8bf.py                       |
| ceval_ppl_578f8d               | configs/datasets/ceval/ceval_ppl_578f8d.py                       |
| ceval_ppl_93e5ce               | configs/datasets/ceval/ceval_ppl_93e5ce.py                       |
| ceval_zero_shot_gen_bd40ef     | configs/datasets/ceval/ceval_zero_shot_gen_bd40ef.py             |
+--------------------------------+------------------------------------------------------------------+

用命令行评测

配置以下代码到 /root/opencompass/configs/models/hf_internlm/的hf_internlm2_chat_1_8b.py

from opencompass.models import HuggingFaceCausalLM


models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='internlm2-1.8b-hf',
        path="/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b",
        tokenizer_path='/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b',
        model_kwargs=dict(
            trust_remote_code=True,
            device_map='auto',
        ),
        tokenizer_kwargs=dict(
            padding_side='left',
            truncation_side='left',
            use_fast=False,
            trust_remote_code=True,
        ),
        max_out_len=100,
        min_out_len=1,
        max_seq_len=2048,
        batch_size=8,
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]

到opencompass文件夹下,运行命令

#环境变量配置
export MKL_SERVICE_FORCE_INTEL=1
#或
# export MKL_THREADING_LAYER=GNU

python run.py --datasets ceval_gen --models hf_internlm2_chat_1_8b --debug

看到进度条大概就是成功啦

(opencompass) root@intern-studio-50006073:~/opencompass# export MKL_SERVICE_FORCE_INTEL=1
(opencompass) root@intern-studio-50006073:~/opencompass# python run.py --datasets ceval_gen --models hf_internlm2_chat_1_8b_test --debug
09/12 23:49:47 - OpenCompass - INFO - Loading ceval_gen: configs/datasets/ceval/ceval_gen.py
09/12 23:49:47 - OpenCompass - INFO - Loading hf_internlm2_chat_1_8b_test: configs/models/hf_internlm/hf_internlm2_chat_1_8b_test.py
09/12 23:49:47 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
09/12 23:49:47 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
09/12 23:49:47 - OpenCompass - INFO - Partitioned into 1 tasks.
09/12 23:50:14 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-college_economics,internlm2-1.8b-hf/ceval-accountant,internlm2-1.8b-hf/ceval-tax_accountant,internlm2-1.8b-hf/ceval-physician,internlm2-1.8b-hf/ceval-civil_servant,internlm2-1.8b-hf/ceval-urban_and_rural_planner,internlm2-1.8b-hf/ceval-teacher_qualification,internlm2-1.8b-hf/ceval-college_programming,internlm2-1.8b-hf/ceval-electrical_engineer,internlm2-1.8b-hf/ceval-business_administration,internlm2-1.8b-hf/ceval-art_studies,internlm2-1.8b-hf/ceval-fire_engineer,internlm2-1.8b-hf/ceval-environmental_impact_assessment_engineer,internlm2-1.8b-hf/ceval-education_science,internlm2-1.8b-hf/ceval-professional_tour_guide,internlm2-1.8b-hf/ceval-college_chemistry,internlm2-1.8b-hf/ceval-metrology_engineer,internlm2-1.8b-hf/ceval-mao_zedong_thought,internlm2-1.8b-hf/ceval-law,internlm2-1.8b-hf/ceval-veterinary_medicine,internlm2-1.8b-hf/ceval-modern_chinese_history,internlm2-1.8b-hf/ceval-chinese_language_and_literature,internlm2-1.8b-hf/ceval-legal_professional,internlm2-1.8b-hf/ceval-logic,internlm2-1.8b-hf/ceval-middle_school_history,internlm2-1.8b-hf/ceval-plant_protection,internlm2-1.8b-hf/ceval-clinical_medicine,internlm2-1.8b-hf/ceval-computer_architecture,internlm2-1.8b-hf/ceval-middle_school_biology,internlm2-1.8b-hf/ceval-middle_school_politics,internlm2-1.8b-hf/ceval-middle_school_chemistry,internlm2-1.8b-hf/ceval-high_school_history,internlm2-1.8b-hf/ceval-computer_network,internlm2-1.8b-hf/ceval-operating_system,internlm2-1.8b-hf/ceval-college_physics,internlm2-1.8b-hf/ceval-advanced_mathematics,internlm2-1.8b-hf/ceval-high_school_physics,internlm2-1.8b-hf/ceval-high_school_chemistry,internlm2-1.8b-hf/ceval-high_school_biology,internlm2-1.8b-hf/ceval-middle_school_mathematics,internlm2-1.8b-hf/ceval-middle_school_physics,internlm2-1.8b-hf/ceval-marxism,internlm2-1.8b-hf/ceval-high_school_politics,internlm2-1.8b-hf/ceval-high_school_geography,internlm2-1.8b-hf/ceval-ideological_and_moral_cultivation,internlm2-1.8b-hf/ceval-high_school_chinese,internlm2-1.8b-hf/ceval-sports_science,internlm2-1.8b-hf/ceval-basic_medicine,internlm2-1.8b-hf/ceval-probability_and_statistics,internlm2-1.8b-hf/ceval-high_school_mathematics,internlm2-1.8b-hf/ceval-discrete_mathematics,internlm2-1.8b-hf/ceval-middle_school_geography]
Loading checkpoint shards: 100%|███████████████████████████████████████████| 2/2 [00:13<00:00,  6.90s/it]
09/12 23:51:00 - OpenCompass - INFO - Start inferencing [internlm2-1.8b-hf/ceval-college_economics]
100%|███████████████████████████████████████████████████████████████| 55/55 [00:00<00:00, 1469342.17it/s]
[2024-09-12 23:51:00,570] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
  0%|                                                                              | 0/7 [00:00<?, ?it/s]The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
 71%|██████████████████████████████████████████████████                    | 5/7 [02:15<00:53, 26.99s/it]

运行完毕后会看到1.8b模型在各种任务上的效果

20240912_132848
tabulate format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset                                         version    metric         mode      internlm2-1.8b-hf
----------------------------------------------  ---------  -------------  ------  -------------------
ceval-computer_network                          db9ce2     accuracy       gen                   47.37
ceval-operating_system                          1c2571     accuracy       gen                   47.37
ceval-computer_architecture                     a74dad     accuracy       gen                   23.81
ceval-college_programming                       4ca32a     accuracy       gen                   27.03
ceval-college_physics                           963fa8     accuracy       gen                   42.11
ceval-college_chemistry                         e78857     accuracy       gen                   37.5
ceval-advanced_mathematics                      ce03e2     accuracy       gen                   26.32
ceval-probability_and_statistics                65e812     accuracy       gen                   22.22
ceval-discrete_mathematics                      e894ae     accuracy       gen                   25
ceval-electrical_engineer                       ae42b9     accuracy       gen                   27.03
ceval-metrology_engineer                        ee34ea     accuracy       gen                   54.17
ceval-high_school_mathematics                   1dc5bf     accuracy       gen                   22.22
ceval-high_school_physics                       adf25f     accuracy       gen                   42.11
ceval-high_school_chemistry                     2ed27f     accuracy       gen                   52.63
ceval-high_school_biology                       8e2b9a     accuracy       gen                   26.32
ceval-middle_school_mathematics                 bee8d5     accuracy       gen                   36.84
ceval-middle_school_biology                     86817c     accuracy       gen                   80.95
ceval-middle_school_physics                     8accf6     accuracy       gen                   47.37
ceval-middle_school_chemistry                   167a15     accuracy       gen                   80
ceval-veterinary_medicine                       b4e08d     accuracy       gen                   43.48
ceval-college_economics                         f3f4e6     accuracy       gen                   32.73
ceval-business_administration                   c1614e     accuracy       gen                   39.39
ceval-marxism                                   cf874c     accuracy       gen                   68.42
ceval-mao_zedong_thought                        51c7a4     accuracy       gen                   70.83
ceval-education_science                         591fee     accuracy       gen                   55.17
ceval-teacher_qualification                     4e4ced     accuracy       gen                   59.09
ceval-high_school_politics                      5c0de2     accuracy       gen                   57.89
ceval-high_school_geography                     865461     accuracy       gen                   47.37
ceval-middle_school_politics                    5be3e7     accuracy       gen                   76.19
ceval-middle_school_geography                   8a63be     accuracy       gen                   75
ceval-modern_chinese_history                    fc01af     accuracy       gen                   52.17
ceval-ideological_and_moral_cultivation         a2aa4a     accuracy       gen                   73.68
ceval-logic                                     f5b022     accuracy       gen                   31.82
ceval-law                                       a110a1     accuracy       gen                   29.17
ceval-chinese_language_and_literature           0f8b68     accuracy       gen                   47.83
ceval-art_studies                               2a1300     accuracy       gen                   42.42
ceval-professional_tour_guide                   4e673e     accuracy       gen                   51.72
ceval-legal_professional                        ce8787     accuracy       gen                   34.78
ceval-high_school_chinese                       315705     accuracy       gen                   36.84
ceval-high_school_history                       7eb30a     accuracy       gen                   65
ceval-middle_school_history                     48ab4a     accuracy       gen                   86.36
ceval-civil_servant                             87d061     accuracy       gen                   42.55
ceval-sports_science                            70f27b     accuracy       gen                   52.63
ceval-plant_protection                          8941f9     accuracy       gen                   40.91
ceval-basic_medicine                            c409d6     accuracy       gen                   68.42
ceval-clinical_medicine                         49e82d     accuracy       gen                   36.36
ceval-urban_and_rural_planner                   95b885     accuracy       gen                   52.17
ceval-accountant                                002837     accuracy       gen                   36.73
ceval-fire_engineer                             bc23f5     accuracy       gen                   38.71
ceval-environmental_impact_assessment_engineer  c64e2d     accuracy       gen                   51.61
ceval-tax_accountant                            3a5e3c     accuracy       gen                   36.73
ceval-physician                                 6e277d     accuracy       gen                   42.86
ceval-stem                                      -          naive_average  gen                   40.59
ceval-social-science                            -          naive_average  gen                   58.21
ceval-humanities                                -          naive_average  gen                   50.16
ceval-other                                     -          naive_average  gen                   45.43
ceval-hard                                      -          naive_average  gen                   33.76
ceval                                           -          naive_average  gen                   47.03
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

done✅

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值