书生大模型实战营-基础关-OpenCompass 评测 InternLM-1.8B 实践

最新推荐文章于 2024-08-31 20:18:08 发布
最新推荐文章于 2024-08-31 20:18:08 发布
阅读量179 收藏 5
点赞数 3
文章标签： 大模型
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/csdn1e/article/details/141220938
版权

OpenCompass 评测 InternLM-1.8B 实践

概览

在 OpenCompass 中评估一个模型通常包括以下几个阶段：配置 -> 推理 -> 评估 -> 可视化。

  • 配置：这是整个工作流的起点。您需要配置整个评估过程，选择要评估的模型和数据集。此外，还可以选择评估策略、计算后端等，并定义显示结果的方式。
  • 推理与评估：在这个阶段，OpenCompass 将会开始对模型和数据集进行并行推理和评估。推理阶段主要是让模型从数据集产生输出，而评估阶段则是衡量这些输出与标准答案的匹配程度。这两个过程会被拆分为多个同时运行的“任务”以提高效率。
  • 可视化：评估完成后，OpenCompass 将结果整理成易读的表格，并将其保存为 CSV 和 TXT 文件。

环境配置

conda create -n opencompass python=3.10
conda activate opencompass
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia -y

# 注意：一定要先 cd /root
cd /root
git clone -b 0.2.4 https://github.com/open-compass/opencompass
cd opencompass
pip install -e .


apt-get update
apt-get install cmake
pip install -r requirements.txt
pip install protobuf

数据准备

cp /share/temp/datasets/OpenCompassData-core-20231110.zip /root/opencompass/
unzip OpenCompassData-core-20231110.zip

配置文件

打开 opencompass文件夹下configs/models/hf_internlm/的hf_internlm2_chat_1_8b.py ,贴入以下代码：

from opencompass.models import HuggingFaceCausalLM


models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='internlm2-1.8b-hf',
        path="/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b",
        tokenizer_path='/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b',
        model_kwargs=dict(
            trust_remote_code=True,
            device_map='auto',
        ),
        tokenizer_kwargs=dict(
            padding_side='left',
            truncation_side='left',
            use_fast=False,
            trust_remote_code=True,
        ),
        max_out_len=100,
        min_out_len=1,
        max_seq_len=2048,
        batch_size=8,
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]

评测

使用命令行

#环境变量配置
export MKL_SERVICE_FORCE_INTEL=1
#或
export MKL_THREADING_LAYER=GNU

# 开始评测
python run.py --datasets ceval_gen --models hf_internlm2_chat_1_8b --debug

评测结果如下：

08/16 10:10:27 - OpenCompass - INFO - time elapsed: 4317.13s
08/16 10:10:31 - OpenCompass - INFO - Partitioned into 52 tasks.
08/16 10:10:34 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-computer_network]: {'accuracy': 47.368421052631575}
08/16 10:10:36 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-operating_system]: {'accuracy': 47.368421052631575}
08/16 10:10:39 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-computer_architecture]: {'accuracy': 23.809523809523807}
08/16 10:10:41 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-college_programming]: {'accuracy': 27.027027027027028}
08/16 10:10:43 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-college_physics]: {'accuracy': 42.10526315789473}
08/16 10:10:45 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-college_chemistry]: {'accuracy': 37.5}
08/16 10:10:47 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-advanced_mathematics]: {'accuracy': 26.31578947368421}
08/16 10:10:49 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-probability_and_statistics]: {'accuracy': 22.22222222222222}
08/16 10:10:51 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-discrete_mathematics]: {'accuracy': 25.0}
08/16 10:10:53 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-electrical_engineer]: {'accuracy': 27.027027027027028}
08/16 10:10:55 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-metrology_engineer]: {'accuracy': 54.166666666666664}
08/16 10:10:57 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_mathematics]: {'accuracy': 22.22222222222222}
08/16 10:11:00 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_physics]: {'accuracy': 42.10526315789473}
08/16 10:11:02 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_chemistry]: {'accuracy': 52.63157894736842}
08/16 10:11:04 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_biology]: {'accuracy': 26.31578947368421}
08/16 10:11:06 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_mathematics]: {'accuracy': 36.84210526315789}
08/16 10:11:08 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_biology]: {'accuracy': 80.95238095238095}
08/16 10:11:10 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_physics]: {'accuracy': 47.368421052631575}
08/16 10:11:12 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_chemistry]: {'accuracy': 80.0}
08/16 10:11:14 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-veterinary_medicine]: {'accuracy': 43.47826086956522}
08/16 10:11:17 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-college_economics]: {'accuracy': 32.72727272727273}
08/16 10:11:19 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-business_administration]: {'accuracy': 39.39393939393939}
08/16 10:11:21 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-marxism]: {'accuracy': 68.42105263157895}
08/16 10:11:23 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-mao_zedong_thought]: {'accuracy': 70.83333333333334}
08/16 10:11:25 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-education_science]: {'accuracy': 55.172413793103445}
08/16 10:11:27 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-teacher_qualification]: {'accuracy': 59.09090909090909}
08/16 10:11:29 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_politics]: {'accuracy': 57.89473684210527}
08/16 10:11:32 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_geography]: {'accuracy': 47.368421052631575}
08/16 10:11:34 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_politics]: {'accuracy': 76.19047619047619}
08/16 10:11:36 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_geography]: {'accuracy': 75.0}
08/16 10:11:38 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-modern_chinese_history]: {'accuracy': 52.17391304347826}
08/16 10:11:40 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-ideological_and_moral_cultivation]: {'accuracy': 73.68421052631578}
08/16 10:11:42 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-logic]: {'accuracy': 31.818181818181817}
08/16 10:11:44 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-law]: {'accuracy': 29.166666666666668}
08/16 10:11:47 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-chinese_language_and_literature]: {'accuracy': 47.82608695652174}
08/16 10:11:49 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-art_studies]: {'accuracy': 42.42424242424242}
08/16 10:11:51 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-professional_tour_guide]: {'accuracy': 51.724137931034484}
08/16 10:11:53 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-legal_professional]: {'accuracy': 34.78260869565217}
08/16 10:11:55 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_chinese]: {'accuracy': 36.84210526315789}
08/16 10:11:58 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-high_school_history]: {'accuracy': 65.0}
08/16 10:12:00 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-middle_school_history]: {'accuracy': 86.36363636363636}
08/16 10:12:02 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-civil_servant]: {'accuracy': 42.5531914893617}
08/16 10:12:04 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-sports_science]: {'accuracy': 52.63157894736842}
08/16 10:12:06 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-plant_protection]: {'accuracy': 40.909090909090914}
08/16 10:12:08 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-basic_medicine]: {'accuracy': 68.42105263157895}
08/16 10:12:10 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-clinical_medicine]: {'accuracy': 36.36363636363637}
08/16 10:12:12 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-urban_and_rural_planner]: {'accuracy': 52.17391304347826}
08/16 10:12:14 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-accountant]: {'accuracy': 36.734693877551024}
08/16 10:12:17 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-fire_engineer]: {'accuracy': 38.70967741935484}
08/16 10:12:19 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-environmental_impact_assessment_engineer]: {'accuracy': 51.61290322580645}
08/16 10:12:21 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-tax_accountant]: {'accuracy': 36.734693877551024}
08/16 10:12:23 - OpenCompass - INFO - Task [internlm2-1.8b-hf/ceval-physician]: {'accuracy': 42.857142857142854}
dataset                                         version    metric         mode      internlm2-1.8b-hf
----------------------------------------------  ---------  -------------  ------  -------------------
ceval-computer_network                          db9ce2     accuracy       gen                   47.37
ceval-operating_system                          1c2571     accuracy       gen                   47.37
ceval-computer_architecture                     a74dad     accuracy       gen                   23.81
ceval-college_programming                       4ca32a     accuracy       gen                   27.03
ceval-college_physics                           963fa8     accuracy       gen                   42.11
ceval-college_chemistry                         e78857     accuracy       gen                   37.5
ceval-advanced_mathematics                      ce03e2     accuracy       gen                   26.32
ceval-probability_and_statistics                65e812     accuracy       gen                   22.22
ceval-discrete_mathematics                      e894ae     accuracy       gen                   25
ceval-electrical_engineer                       ae42b9     accuracy       gen                   27.03
ceval-metrology_engineer                        ee34ea     accuracy       gen                   54.17
ceval-high_school_mathematics                   1dc5bf     accuracy       gen                   22.22
ceval-high_school_physics                       adf25f     accuracy       gen                   42.11
ceval-high_school_chemistry                     2ed27f     accuracy       gen                   52.63
ceval-high_school_biology                       8e2b9a     accuracy       gen                   26.32
ceval-middle_school_mathematics                 bee8d5     accuracy       gen                   36.84
ceval-middle_school_biology                     86817c     accuracy       gen                   80.95
ceval-middle_school_physics                     8accf6     accuracy       gen                   47.37
ceval-middle_school_chemistry                   167a15     accuracy       gen                   80
ceval-veterinary_medicine                       b4e08d     accuracy       gen                   43.48
ceval-college_economics                         f3f4e6     accuracy       gen                   32.73
ceval-business_administration                   c1614e     accuracy       gen                   39.39
ceval-marxism                                   cf874c     accuracy       gen                   68.42
ceval-mao_zedong_thought                        51c7a4     accuracy       gen                   70.83
ceval-education_science                         591fee     accuracy       gen                   55.17
ceval-teacher_qualification                     4e4ced     accuracy       gen                   59.09
ceval-high_school_politics                      5c0de2     accuracy       gen                   57.89
ceval-high_school_geography                     865461     accuracy       gen                   47.37
ceval-middle_school_politics                    5be3e7     accuracy       gen                   76.19
ceval-middle_school_geography                   8a63be     accuracy       gen                   75
ceval-modern_chinese_history                    fc01af     accuracy       gen                   52.17
ceval-ideological_and_moral_cultivation         a2aa4a     accuracy       gen                   73.68
ceval-logic                                     f5b022     accuracy       gen                   31.82
ceval-law                                       a110a1     accuracy       gen                   29.17
ceval-chinese_language_and_literature           0f8b68     accuracy       gen                   47.83
ceval-art_studies                               2a1300     accuracy       gen                   42.42
ceval-professional_tour_guide                   4e673e     accuracy       gen                   51.72
ceval-legal_professional                        ce8787     accuracy       gen                   34.78
ceval-high_school_chinese                       315705     accuracy       gen                   36.84
ceval-high_school_history                       7eb30a     accuracy       gen                   65
ceval-middle_school_history                     48ab4a     accuracy       gen                   86.36
ceval-civil_servant                             87d061     accuracy       gen                   42.55
ceval-sports_science                            70f27b     accuracy       gen                   52.63
ceval-plant_protection                          8941f9     accuracy       gen                   40.91
ceval-basic_medicine                            c409d6     accuracy       gen                   68.42
ceval-clinical_medicine                         49e82d     accuracy       gen                   36.36
ceval-urban_and_rural_planner                   95b885     accuracy       gen                   52.17
ceval-accountant                                002837     accuracy       gen                   36.73
ceval-fire_engineer                             bc23f5     accuracy       gen                   38.71
ceval-environmental_impact_assessment_engineer  c64e2d     accuracy       gen                   51.61
ceval-tax_accountant                            3a5e3c     accuracy       gen                   36.73
ceval-physician                                 6e277d     accuracy       gen                   42.86
ceval-stem                                      -          naive_average  gen                   40.59
ceval-social-science                            -          naive_average  gen                   58.21
ceval-humanities                                -          naive_average  gen                   50.16
ceval-other                                     -          naive_average  gen                   45.43
ceval-hard                                      -          naive_average  gen                   33.76
ceval                                           -          naive_average  gen                   47.03
08/16 10:12:23 - OpenCompass - INFO - write summary to /root/demo/opencompass/outputs/default/20240816_085805/summary/summary_20240816_085805.txt
08/16 10:12:23 - OpenCompass - INFO - write csv to /root/demo/opencompass/outputs/default/20240816_085805/summary/summary_20240816_085805.csv
great-wind
关注
书生大模型实战营--L1关卡-OpenCompass 评测 InternLM-1.8B 实践
cgq081616的专栏
07-27 1059
3、再里面的 opencompass/configs/eval_internlm_chat_lmdeploy_apiserver.py增加1.8b调用地址。一、使用 OpenCompass 评测 internlm2-chat-1.8b 模型在 MMLU 数据集上的性能。1、使用lmdeploy部署 internlm2-chat-1.8b模型。2、根据OpenCompass官网教程安装并下载数据集。注意：评测mmlu的时候注释掉其他数据集。4、运行评测脚本---使用API调用。
8.22 书生大模型实战营-基础岛6 - OpenCompass 评测 InternLM-1.8B 实践作业
zjzjsuus的博客
08-23 663
在 OpenCompass 中评估一个模型通常包括以下几个阶段：配置 -> 推理 -> 评估 -> 可视化。配置：这是整个工作流的起点。您需要配置整个评估过程，选择要评估的模型和数据集。此外，还可以选择评估策略、计算后端等，并定义显示结果的方式。推理与评估：在这个阶段，OpenCompass 将会开始对模型和数据集进行并行推理和评估。推理阶段主要是让模型从数据集产生输出，而评估阶段则是衡量这些输出与标准答案的匹配程度。这两个过程会被拆分为多个同时运行的“任务”以提高效率。
书生大模型实战营-基础关卡-6-OpenCompass 评测 InternLM-1.8B 实践
微风❤水墨
08-15 176
列出所有跟 InternLM 及 C-Eval 相关的配置 测试过程挺慢的，为了防止超时，测试结果直接导出到日志 08/14 20:33:06 - OpenCompass - INFO - Loading ceval_gen: configs/datasets/ceval/ceval_gen.py 08/14 20:33:07 - OpenCompass - INFO - Loading hf_internlm2_chat_1_8b: configs/models/hf_internlm/h
书生大模型实战营-基础岛第六关 OpenCompass 评测 InternLM-1.8B 实践
sty0213的博客
08-09 172
确保按照上述步骤正确安装 OpenCompass 并准备好数据集后，可以通过以下命令评测 InternLM2-Chat-1.8B 模型在 C-Eval 数据集上的性能。由于 OpenCompass 默认并行启动评估过程，我们可以在第一次运行时以 --debug 模式启动评估，并检查是否存在问题。在 --debug 模式下，任务将按顺序执行，并实时打印输出。打开 opencompass文件夹下configs/models/hf_internlm/的。解压评测数据集到 data/ 处。
书生大模型实战营闯关 - 基础岛 - OpenCompass 评测 InternLM-1.8B 实践
nachtmusik的博客
08-13 757
本文将进行使用 OpenCompass 来评测 InternLM2 1.8B实践。
第三期书生大模型实战营之OpenCompass 评测 InternLM-1.8B 实践
T_susan的博客
08-04 290
使用 OpenCompass 评测 internlm2-chat-1.8b 模型在 MMLU 数据集上的性能，记录复现过程并截图。
书生大模型实战--OpenCompass 评测 InternLM-1.8B 实践
m0_71853180的博客
08-31 768
本次学习，了解了 OpenCompass 的基础用法，分别用命令行方式和配置文件的方式评测InternLM2-Chat-1.8B，展示书生浦语在 `C-Eval` 基准任务上的评估。
书生大模型实战营 基础第6关 OpenCompass 评测 InternLM-1.8B 实践
qq_52700183的博客
08-23 211
Compass如何对大模型进行评测。查看支持的数据集和模型。
书生大模型实战营-OpenCompass 评测 InternLM-1.8B 实践
weixin_52114905的博客
08-07 791
PS：如果出现--num-gpus的错误，用错误提示中的tf--num-gpus替换即可。环境配置真的很难绷。创建了三次环境才搞好。
书生大模型实战营第三关浦语提示词工程实践
08-09
书生大模型实战营第三关浦语提示词工程实践
书生·浦语大模型 - 基础岛 - 第3关 书生浦语提示词工程实践
08-10
书生·浦语大模型 - 基础岛 - 第3关 书生浦语提示词工程实践
大模型-书生训练营第三期闯关指南-linux
07-19
大模型-书生训练营第三期闯关指南-linux
书生大模型实战营-入门关卡-Linux 前置基础
微风❤水墨
08-07 93
任务：Tutorial/docs/L0/Linux at camp3 · InternLM/Tutorial · GitHub完成：
report ETL .ffff
09-17
report ETL .ffff
基于ssm的共享充电宝管理系统设计与实现.docx
09-17
基于ssm的共享充电宝管理系统设计与实现.docx
操作系统大作业_Linux_命令行_OSWork.zip
最新发布
09-17
操作系统大作业_Linux_命令行_OSWork
Java项目-基于SSM+Jsp的社区管理与服务系统的设计与实现（源码+数据库脚本+部署视频+代码讲解视频+全套软件）
09-17
【基于SSM+Jsp的社区管理与服务系统的设计与实现】高分通过项目，已获导师指导。 本项目是一套基于SSM+Jsp的社区管理与服务系统，主要针对计算机相关专业的正在做毕设的学生和需要项目实战练习的Java学习者。也可作为课程设计、期末大作业 包含：项目源码、数据库脚本、开发说明文档、部署视频、代码讲解视频、全套软件等，该项目可以直接作为毕设使用。 项目都经过严格调试，确保可以运行！
吉他谱_Show Me How To Live - Audioslave.pdf
09-17
初级入门吉他谱 guitar tab
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

great-wind

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值