基于OpenCompass大模型评测
关于评测的三个问题Why/What/How
Why
What
有许多任务评测，包括垂直领域
How
包含客观评测和主观评测，其中主观评测分人工和模型来评估。
提示词工程
主流评测框架
OpenCompass 能力框架
- 模型层
- 能力层
- 方法层
- 工具层
支持丰富的模型
评测流水线设计，能切分多个独立执行的任务，最大化利用计算资源。
大模型能力对比结果输出
前言探索
探索性方向涵盖：
- 多模态
- 法律
- 医生
挑战
实践
创建开发环境和准备数据集
查看支持的数据集：
启动评测
客观评测
主要是run.py代码文件
- datasets：指定数据集
- hf-path：模型文件
- tokenizer-path：tokenizer路径
- max-seq-len：模型读入的最大长度
- max-out-len：模型输出的最大长度，客观题设置一般较小
- –debug：debug模式，打印出所有的过程
主观评测
主要是eval_sbujective_alignbench.py文件修改，需要注意model，max_out_len等处的修改。
最终结果：
python tools/list_configs.py internlm ceval
20240122_153109
tabulate format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset version metric mode opencompass.models.huggingface.HuggingFace_model_repos_internlm2-chat-7b
---------------------------------------------- --------- ------------- ------ --------------------------------------------------------------------------
ceval-computer_network db9ce2 accuracy gen 47.37
ceval-operating_system 1c2571 accuracy gen 57.89
ceval-computer_architecture a74dad accuracy gen 38.1
ceval-college_programming 4ca32a accuracy gen 18.92
ceval-college_physics 963fa8 accuracy gen 5.26
ceval-college_chemistry e78857 accuracy gen 0
ceval-advanced_mathematics ce03e2 accuracy gen 0
ceval-probability_and_statistics 65e812 accuracy gen 11.11
ceval-discrete_mathematics e894ae accuracy gen 18.75
ceval-electrical_engineer ae42b9 accuracy gen 18.92
ceval-metrology_engineer ee34ea accuracy gen 50
ceval-high_school_mathematics 1dc5bf accuracy gen 0
ceval-high_school_physics adf25f accuracy gen 31.58
ceval-high_school_chemistry 2ed27f accuracy gen 26.32
ceval-high_school_biology 8e2b9a accuracy gen 26.32
ceval-middle_school_mathematics bee8d5 accuracy gen 21.05
ceval-middle_school_biology 86817c accuracy gen 66.67
ceval-middle_school_physics 8accf6 accuracy gen 52.63
ceval-middle_school_chemistry 167a15 accuracy gen 80
ceval-veterinary_medicine b4e08d accuracy gen 39.13
ceval-college_economics f3f4e6 accuracy gen 29.09
ceval-business_administration c1614e accuracy gen 30.3
ceval-marxism cf874c accuracy gen 84.21
ceval-mao_zedong_thought 51c7a4 accuracy gen 70.83
ceval-education_science 591fee accuracy gen 62.07
ceval-teacher_qualification 4e4ced accuracy gen 77.27
ceval-high_school_politics 5c0de2 accuracy gen 21.05
ceval-high_school_geography 865461 accuracy gen 47.37
ceval-middle_school_politics 5be3e7 accuracy gen 38.1
ceval-middle_school_geography 8a63be accuracy gen 58.33
ceval-modern_chinese_history fc01af accuracy gen 65.22
ceval-ideological_and_moral_cultivation a2aa4a accuracy gen 89.47
ceval-logic f5b022 accuracy gen 13.64
ceval-law a110a1 accuracy gen 37.5
ceval-chinese_language_and_literature 0f8b68 accuracy gen 47.83
ceval-art_studies 2a1300 accuracy gen 66.67
ceval-professional_tour_guide 4e673e accuracy gen 82.76
ceval-legal_professional ce8787 accuracy gen 30.43
ceval-high_school_chinese 315705 accuracy gen 21.05
ceval-high_school_history 7eb30a accuracy gen 75
ceval-middle_school_history 48ab4a accuracy gen 68.18
ceval-civil_servant 87d061 accuracy gen 38.3
ceval-sports_science 70f27b accuracy gen 63.16
ceval-plant_protection 8941f9 accuracy gen 68.18
ceval-basic_medicine c409d6 accuracy gen 57.89
ceval-clinical_medicine 49e82d accuracy gen 45.45
ceval-urban_and_rural_planner 95b885 accuracy gen 58.7
ceval-accountant 002837 accuracy gen 34.69
ceval-fire_engineer bc23f5 accuracy gen 12.9
ceval-environmental_impact_assessment_engineer c64e2d accuracy gen 38.71
ceval-tax_accountant 3a5e3c accuracy gen 42.86
ceval-physician 6e277d accuracy gen 51.02
ceval-stem - naive_average gen 30.5
ceval-social-science - naive_average gen 51.86
ceval-humanities - naive_average gen 54.34
ceval-other - naive_average gen 46.53
ceval-hard - naive_average gen 11.63
ceval - naive_average gen 43.04
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$-------------------------------------------------------------------------------------------------------------------------------- THIS IS A DIVIDER --------------------------------------------------------------------------------------------------------------------------------csv format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset,version,metric,mode,opencompass.models.huggingface.HuggingFace_model_repos_internlm2-chat-7b
ceval-computer_network,db9ce2,accuracy,gen,47.37
ceval-operating_system,1c2571,accuracy,gen,57.89
ceval-computer_architecture,a74dad,accuracy,gen,38.10
ceval-college_programming,4ca32a,accuracy,gen,18.92
ceval-college_physics,963fa8,accuracy,gen,5.26
ceval-college_chemistry,e78857,accuracy,gen,0.00
ceval-advanced_mathematics,ce03e2,accuracy,gen,0.00
ceval-probability_and_statistics,65e812,accuracy,gen,11.11
ceval-discrete_mathematics,e894ae,accuracy,gen,18.75
ceval-electrical_engineer,ae42b9,accuracy,gen,18.92
ceval-metrology_engineer,ee34ea,accuracy,gen,50.00
ceval-high_school_mathematics,1dc5bf,accuracy,gen,0.00
ceval-high_school_physics,adf25f,accuracy,gen,31.58
ceval-high_school_chemistry,2ed27f,accuracy,gen,26.32
ceval-high_school_biology,8e2b9a,accuracy,gen,26.32
ceval-middle_school_mathematics,bee8d5,accuracy,gen,21.05
ceval-middle_school_biology,86817c,accuracy,gen,66.67
ceval-middle_school_physics,8accf6,accuracy,gen,52.63
ceval-middle_school_chemistry,167a15,accuracy,gen,80.00
ceval-veterinary_medicine,b4e08d,accuracy,gen,39.13
ceval-college_economics,f3f4e6,accuracy,gen,29.09
ceval-business_administration,c1614e,accuracy,gen,30.30
ceval-marxism,cf874c,accuracy,gen,84.21
ceval-mao_zedong_thought,51c7a4,accuracy,gen,70.83
ceval-education_science,591fee,accuracy,gen,62.07
ceval-teacher_qualification,4e4ced,accuracy,gen,77.27
ceval-high_school_politics,5c0de2,accuracy,gen,21.05
ceval-high_school_geography,865461,accuracy,gen,47.37
ceval-middle_school_politics,5be3e7,accuracy,gen,38.10
ceval-middle_school_geography,8a63be,accuracy,gen,58.33
ceval-modern_chinese_history,fc01af,accuracy,gen,65.22
ceval-ideological_and_moral_cultivation,a2aa4a,accuracy,gen,89.47
ceval-logic,f5b022,accuracy,gen,13.64
ceval-law,a110a1,accuracy,gen,37.50
ceval-chinese_language_and_literature,0f8b68,accuracy,gen,47.83
ceval-art_studies,2a1300,accuracy,gen,66.67
ceval-professional_tour_guide,4e673e,accuracy,gen,82.76
ceval-legal_professional,ce8787,accuracy,gen,30.43
ceval-high_school_chinese,315705,accuracy,gen,21.05
ceval-high_school_history,7eb30a,accuracy,gen,75.00
ceval-middle_school_history,48ab4a,accuracy,gen,68.18
ceval-civil_servant,87d061,accuracy,gen,38.30
ceval-sports_science,70f27b,accuracy,gen,63.16
ceval-plant_protection,8941f9,accuracy,gen,68.18
ceval-basic_medicine,c409d6,accuracy,gen,57.89
ceval-clinical_medicine,49e82d,accuracy,gen,45.45
ceval-urban_and_rural_planner,95b885,accuracy,gen,58.70
ceval-accountant,002837,accuracy,gen,34.69
ceval-fire_engineer,bc23f5,accuracy,gen,12.90
ceval-environmental_impact_assessment_engineer,c64e2d,accuracy,gen,38.71
ceval-tax_accountant,3a5e3c,accuracy,gen,42.86
ceval-physician,6e277d,accuracy,gen,51.02
ceval-stem,-,naive_average,gen,30.50
ceval-social-science,-,naive_average,gen,51.86
ceval-humanities,-,naive_average,gen,54.34
ceval-other,-,naive_average,gen,46.53
ceval-hard,-,naive_average,gen,11.63
ceval,-,naive_average,gen,43.04
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$-------------------------------------------------------------------------------------------------------------------------------- THIS IS A DIVIDER --------------------------------------------------------------------------------------------------------------------------------raw format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-------------------------------
Model: opencompass.models.huggingface.HuggingFace_model_repos_internlm2-chat-7b
ceval-computer_network: {'accuracy': 47.368421052631575}
ceval-operating_system: {'accuracy': 57.89473684210527}
ceval-computer_architecture: {'accuracy': 38.095238095238095}
ceval-college_programming: {'accuracy': 18.91891891891892}
ceval-college_physics: {'accuracy': 5.263157894736842}
ceval-college_chemistry: {'accuracy': 0.0}
ceval-advanced_mathematics: {'accuracy': 0.0}
ceval-probability_and_statistics: {'accuracy': 11.11111111111111}
ceval-discrete_mathematics: {'accuracy': 18.75}
ceval-electrical_engineer: {'accuracy': 18.91891891891892}
ceval-metrology_engineer: {'accuracy': 50.0}
ceval-high_school_mathematics: {'accuracy': 0.0}
ceval-high_school_physics: {'accuracy': 31.57894736842105}
ceval-high_school_chemistry: {'accuracy': 26.31578947368421}
ceval-high_school_biology: {'accuracy': 26.31578947368421}
ceval-middle_school_mathematics: {'accuracy': 21.052631578947366}
ceval-middle_school_biology: {'accuracy': 66.66666666666666}
ceval-middle_school_physics: {'accuracy': 52.63157894736842}
ceval-middle_school_chemistry: {'accuracy': 80.0}
ceval-veterinary_medicine: {'accuracy': 39.130434782608695}
ceval-college_economics: {'accuracy': 29.09090909090909}
ceval-business_administration: {'accuracy': 30.303030303030305}
ceval-marxism: {'accuracy': 84.21052631578947}
ceval-mao_zedong_thought: {'accuracy': 70.83333333333334}
ceval-education_science: {'accuracy': 62.06896551724138}
ceval-teacher_qualification: {'accuracy': 77.27272727272727}
ceval-high_school_politics: {'accuracy': 21.052631578947366}
ceval-high_school_geography: {'accuracy': 47.368421052631575}
ceval-middle_school_politics: {'accuracy': 38.095238095238095}
ceval-middle_school_geography: {'accuracy': 58.333333333333336}
ceval-modern_chinese_history: {'accuracy': 65.21739130434783}
ceval-ideological_and_moral_cultivation: {'accuracy': 89.47368421052632}
ceval-logic: {'accuracy': 13.636363636363635}
ceval-law: {'accuracy': 37.5}
ceval-chinese_language_and_literature: {'accuracy': 47.82608695652174}
ceval-art_studies: {'accuracy': 66.66666666666666}
ceval-professional_tour_guide: {'accuracy': 82.75862068965517}
ceval-legal_professional: {'accuracy': 30.434782608695656}
ceval-high_school_chinese: {'accuracy': 21.052631578947366}
ceval-high_school_history: {'accuracy': 75.0}
ceval-middle_school_history: {'accuracy': 68.18181818181817}
ceval-civil_servant: {'accuracy': 38.297872340425535}
ceval-sports_science: {'accuracy': 63.1578947368421}
ceval-plant_protection: {'accuracy': 68.18181818181817}
ceval-basic_medicine: {'accuracy': 57.89473684210527}
ceval-clinical_medicine: {'accuracy': 45.45454545454545}
ceval-urban_and_rural_planner: {'accuracy': 58.69565217391305}
ceval-accountant: {'accuracy': 34.69387755102041}
ceval-fire_engineer: {'accuracy': 12.903225806451612}
ceval-environmental_impact_assessment_engineer: {'accuracy': 38.70967741935484}
ceval-tax_accountant: {'accuracy': 42.857142857142854}
ceval-physician: {'accuracy': 51.02040816326531}
ceval-stem: {'ceval-computer_network': 47.368421052631575, 'ceval-operating_system': 57.89473684210527, 'ceval-computer_architecture': 38.095238095238095, 'ceval-college_programming': 18.91891891891892, 'ceval-college_physics': 5.263157894736842, 'ceval-college_chemistry': 0.0, 'ceval-advanced_mathematics': 0.0, 'ceval-probability_and_statistics': 11.11111111111111, 'ceval-discrete_mathematics': 18.75, 'ceval-electrical_engineer': 18.91891891891892, 'ceval-metrology_engineer': 50.0, 'ceval-high_school_mathematics': 0.0, 'ceval-high_school_physics': 31.57894736842105, 'ceval-high_school_chemistry': 26.31578947368421, 'ceval-high_school_biology': 26.31578947368421, 'ceval-middle_school_mathematics': 21.052631578947366, 'ceval-middle_school_biology': 66.66666666666666, 'ceval-middle_school_physics': 52.63157894736842, 'ceval-middle_school_chemistry': 80.0, 'ceval-veterinary_medicine': 39.130434782608695, 'naive_average': 30.50061705625207}
ceval-social-science: {'ceval-college_economics': 29.09090909090909, 'ceval-business_administration': 30.303030303030305, 'ceval-marxism': 84.21052631578947, 'ceval-mao_zedong_thought': 70.83333333333334, 'ceval-education_science': 62.06896551724138, 'ceval-teacher_qualification': 77.27272727272727, 'ceval-high_school_politics': 21.052631578947366, 'ceval-high_school_geography': 47.368421052631575, 'ceval-middle_school_politics': 38.095238095238095, 'ceval-middle_school_geography': 58.333333333333336, 'naive_average': 51.86291158931812}
ceval-humanities: {'ceval-modern_chinese_history': 65.21739130434783, 'ceval-ideological_and_moral_cultivation': 89.47368421052632, 'ceval-logic': 13.636363636363635, 'ceval-law': 37.5, 'ceval-chinese_language_and_literature': 47.82608695652174, 'ceval-art_studies': 66.66666666666666, 'ceval-professional_tour_guide': 82.75862068965517, 'ceval-legal_professional': 30.434782608695656, 'ceval-high_school_chinese': 21.052631578947366, 'ceval-high_school_history': 75.0, 'ceval-middle_school_history': 68.18181818181817, 'naive_average': 54.340731439412956}
ceval-other: {'ceval-civil_servant': 38.297872340425535, 'ceval-sports_science': 63.1578947368421, 'ceval-plant_protection': 68.18181818181817, 'ceval-basic_medicine': 57.89473684210527, 'ceval-clinical_medicine': 45.45454545454545, 'ceval-urban_and_rural_planner': 58.69565217391305, 'ceval-accountant': 34.69387755102041, 'ceval-fire_engineer': 12.903225806451612, 'ceval-environmental_impact_assessment_engineer': 38.70967741935484, 'ceval-tax_accountant': 42.857142857142854, 'ceval-physician': 51.02040816326531, 'naive_average': 46.533350138807684}
ceval-hard: {'ceval-advanced_mathematics': 0.0, 'ceval-discrete_mathematics': 18.75, 'ceval-probability_and_statistics': 11.11111111111111, 'ceval-college_chemistry': 0.0, 'ceval-college_physics': 5.263157894736842, 'ceval-high_school_mathematics': 0.0, 'ceval-high_school_chemistry': 26.31578947368421, 'ceval-high_school_physics': 31.57894736842105, 'naive_average': 11.627375730994151}
ceval: {'ceval-computer_network': 47.368421052631575, 'ceval-operating_system': 57.89473684210527, 'ceval-computer_architecture': 38.095238095238095, 'ceval-college_programming': 18.91891891891892, 'ceval-college_physics': 5.263157894736842, 'ceval-college_chemistry': 0.0, 'ceval-advanced_mathematics': 0.0, 'ceval-probability_and_statistics': 11.11111111111111, 'ceval-discrete_mathematics': 18.75, 'ceval-electrical_engineer': 18.91891891891892, 'ceval-metrology_engineer': 50.0, 'ceval-high_school_mathematics': 0.0, 'ceval-high_school_physics': 31.57894736842105, 'ceval-high_school_chemistry': 26.31578947368421, 'ceval-high_school_biology': 26.31578947368421, 'ceval-middle_school_mathematics': 21.052631578947366, 'ceval-middle_school_biology': 66.66666666666666, 'ceval-middle_school_physics': 52.63157894736842, 'ceval-middle_school_chemistry': 80.0, 'ceval-veterinary_medicine': 39.130434782608695, 'ceval-college_economics': 29.09090909090909, 'ceval-business_administration': 30.303030303030305, 'ceval-marxism': 84.21052631578947, 'ceval-mao_zedong_thought': 70.83333333333334, 'ceval-education_science': 62.06896551724138, 'ceval-teacher_qualification': 77.27272727272727, 'ceval-high_school_politics': 21.052631578947366, 'ceval-high_school_geography': 47.368421052631575, 'ceval-middle_school_politics': 38.095238095238095, 'ceval-middle_school_geography': 58.333333333333336, 'ceval-modern_chinese_history': 65.21739130434783, 'ceval-ideological_and_moral_cultivation': 89.47368421052632, 'ceval-logic': 13.636363636363635, 'ceval-law': 37.5, 'ceval-chinese_language_and_literature': 47.82608695652174, 'ceval-art_studies': 66.66666666666666, 'ceval-professional_tour_guide': 82.75862068965517, 'ceval-legal_professional': 30.434782608695656, 'ceval-high_school_chinese': 21.052631578947366, 'ceval-high_school_history': 75.0, 'ceval-middle_school_history': 68.18181818181817, 'ceval-civil_servant': 38.297872340425535, 'ceval-sports_science': 63.1578947368421, 'ceval-plant_protection': 68.18181818181817, 'ceval-basic_medicine': 57.89473684210527, 'ceval-clinical_medicine': 45.45454545454545, 'ceval-urban_and_rural_planner': 58.69565217391305, 'ceval-accountant': 34.69387755102041, 'ceval-fire_engineer': 12.903225806451612, 'ceval-environmental_impact_assessment_engineer': 38.70967741935484, 'ceval-tax_accountant': 42.857142857142854, 'ceval-physician': 51.02040816326531, 'naive_average': 43.043391430358646}
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
