模型预测效果
用于测试的问题有一些模型未给出明确的答案,故需要人工校对
实例
"question": "根据《序卦传》,比卦位于六十四卦中的哪一位",
"options": [
"A:七",
"B:八",
"C:九",
"D:十"
],
"correct_answer": "B",
"type": "序卦传",
"kb_answer": "比卦位于六十四卦中的九位。"
正确答案未B。结果回答未九位,故答题错误。
结果统计与汇总
对所有文件进行结果分析,得出下表。x/y:x代表该类型题目回答正确,y代表该类型题目数量
模型类别 | 总计 | 卦辞 | 爻辞 | 彖传 | 象传 | 系辞传 | 卦间关系 | 爻间关系 | 上下卦 | 成语 | 人物著作 | 基础知识 | 序卦传 | 杂卦传 | 说卦传 | 文言传 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
kb_chat_chatflm3-6b | 2/4 | 0/1 | 1/1 | 1/2 | ||||||||||||
kb_chat_yizhou-ft-3 | 1/2 | 1/2 | ||||||||||||||
kb_chat_yizhou-ft-30 | 5/13 | 0/1 | 0/1 | 3/3 | 0/2 | 0/1 | 1/1 | 0/1 | 2/3 | |||||||
kb_chat_yizhou-ft-50 | 10/22 | 1/1 | 1/2 | 1/1 | 3/4 | 0/2 | 2/4 | 0/1 | 1/3 | 1/3 | 0/1 | |||||
kb_chat_yizhouft-100 | 6/20 | 1/3 | 0/1 | 0/1 | 1/2 | 1/2 | 1/2 | 2/6 | 0/1 | 0/1 | 0/1 | |||||
llm_chat_chatglm3-6b | 0/2 | 0/1 | 0/1 | |||||||||||||
online_chat_zhipu_ai | ||||||||||||||||
origin_llm_kb | 1/1 | 1/1 | ||||||||||||||
origin_llm | ||||||||||||||||
yizhou-ft-3-kb | 3/4 | 2/3 | 1/1 | |||||||||||||
yizhou-ft-30-kb | 4/14 | 0/1 | 2/2 | 1/1 | 0/1 | 1/6 | 0/1 | 0/2 | ||||||||
yizhou-ft-50-kb | 8/21 | 3/3 | 1/3 | 0/1 | 1/1 | 0/1 | 0/2 | 1/1 | 0/1 | 1/2 | 0/2 | 1/2 | 0/2 | |||
yizhou-ft-100-kb | 5/20 | 1/2 | 0/2 | 0/1 | 0/2 | 2/2 | 1/2 | 1/6 | 0/1 | 0/2 | ||||||
zhipu_ai |
添加未解答的数据,更新solve_rate文件。
[
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 80,
"Solve_rate": "0.4706",
"llm_name": "llm_chat_chatglm3-6b",
"type_stats": {
"卦辞": "9/15",
"爻辞": "8/16",
"彖传": "3/8",
"象传": "5/15",
"系辞传": "8/10",
"卦间关系": "8/16",
"爻间关系": "2/10",
"上下卦": "4/10",
"成语": "3/10",
"人物著作": "5/10",
"基础知识": "6/10",
"序卦传": "1/10",
"杂卦传": "5/10",
"说卦传": "4/10",
"文言传": "9/10"
}
},
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 108,
"Solve_rate": "0.6353",
"llm_name": "kb_chat_chatglm3-6b",
"type_stats": {
"卦辞": "13/15",
"爻辞": "9/16",
"彖传": "4/8",
"象传": "8/15",
"系辞传": "10/10",
"卦间关系": "12/16",
"爻间关系": "4/10",
"上下卦": "6/10",
"成语": "5/10",
"人物著作": "6/10",
"基础知识": "7/10",
"序卦传": "2/10",
"杂卦传": "5/10",
"说卦传": "7/10",
"文言传": "10/10"
}
},
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 97,
"Solve_rate": "0.5706",
"llm_name": "kb_chat_yizhou-ft-3",
"type_stats": {
"卦辞": "13/15",
"爻辞": "8/16",
"彖传": "5/8",
"象传": "7/15",
"系辞传": "10/10",
"卦间关系": "10/16",
"爻间关系": "4/10",
"上下卦": "5/10",
"成语": "3/10",
"人物著作": "6/10",
"基础知识": "7/10",
"序卦传": "2/10",
"杂卦传": "5/10",
"说卦传": "6/10",
"文言传": "6/10"
}
},
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 101,
"Solve_rate": "0.5941",
"llm_name": "kb_chat_yizhou-ft-30",
"type_stats": {
"卦辞": "11/15",
"爻辞": "10/16",
"彖传": "6/8",
"象传": "8/15",
"系辞传": "6/10",
"卦间关系": "11/16",
"爻间关系": "6/10",
"上下卦": "4/10",
"成语": "6/10",
"人物著作": "6/10",
"基础知识": "7/10",
"序卦传": "3/10",
"杂卦传": "4/10",
"说卦传": "6/10",
"文言传": "8/10"
}
},
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 107,
"Solve_rate": "0.6294",
"llm_name": "kb_chat_yizhou-ft-50",
"type_stats": {
"卦辞": "13/15",
"爻辞": "10/16",
"彖传": "6/8",
"象传": "5/15",
"系辞传": "7/10",
"卦间关系": "11/16",
"爻间关系": "5/10",
"上下卦": "7/10",
"成语": "7/10",
"人物著作": "6/10",
"基础知识": "8/10",
"序卦传": "4/10",
"杂卦传": "6/10",
"说卦传": "5/10",
"文言传": "7/10"
}
},
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 103,
"Solve_rate": "0.6059",
"llm_name": "kb_chat_yizhou-ft-100",
"type_stats": {
"卦辞": "11/15",
"爻辞": "8/16",
"彖传": "6/8",
"象传": "7/15",
"系辞传": "10/10",
"卦间关系": "11/16",
"爻间关系": "4/10",
"上下卦": "4/10",
"成语": "7/10",
"人物著作": "3/10",
"基础知识": "6/10",
"序卦传": "6/10",
"杂卦传": "7/10",
"说卦传": "5/10",
"文言传": "8/10"
}
},
{
"Tot_problem_nums": 170,
"Solved_problem_nums": 124,
"Solve_rate": "0.7294",
"llm_name": "online_chat_zhipu_ai",
"type_stats": {
"卦辞": "14/15",
"爻辞": "10/16",
"彖传": "8/8",
"象传": "8/15",
"系辞传": "10/10",
"卦间关系": "13/16",
"爻间关系": "8/10",
"上下卦": "5/10",
"成语": "7/10",
"人物著作": "9/10",
"基础知识": "8/10",
"序卦传": "3/10",
"杂卦传": "4/10",
"说卦传": "7/10",
"文言传": "10/10"
}
}
]