OSMB測試結果: Testset:20240425-1 Basic Financial Q&A

在这里插入图片描述

测试计划详细信息

使用简单的问题来测试大语言模型的财务知识。

测试集详细信息

使用 GPT-4 生成的基本财务问答集。
发布日期2024 年 4 月 26 日

方法

请参阅我们的博客文章以了解有关我们的方法的更多信息。还有我们所有数据的GitHub 存储库

主要发现

  • Llama3:70b 表现最佳,其次是 Mixtral 和 WizardLM2 8x22b MoE,以及qwen:32b
  • llama3:70b 比其他 70b 型号有更好的性能
  • qwen:110b q4 量化的性能比 q8 稍好。 q2 较低
  • starling-lm:7b q4 量化比 fp16 和 q8 具有更好的性能
  • phi3:3.8b fp16量化比q4有更好的性能
  • phi3:3.8b 的性能比 phi2:2.7b 好得多
  • gemma:7b-v1.1 在 7b 到 13b 模型中性能最好,比 llama3:8b 好得多
  • qwen:32b q8和fp16量化比q4有更好的性能
  • gemma:2b-v1.1 比 v1.0 更好
  • gemma:2b-v1.1 fp16 量化优于 q8 和 q4
  • gemma:7b-v1.1 比 v1.0 性能更好
  • gemma:7b-v1.1 q8和fp16量化比q4好,q2低
  • dolphin-llama3:70b的性能低于原始 llama3:70b
  • dolphin-llama3:8b 的性能低于原始 llama3:8b
  • llama3-gradient 具有更大的上下文窗口,但性能显着降低
  • mixtral:8x22b MoE 的性能优于 WizardLM2:8x22b
  • Wizardlm2:7b 量化性能有所不同。 fp16 是最好的,q4 击败 q8 和 q6
  • WizardLM2:8x22b q4和q8量化具有相同的性能
  • mixtral:8x22b q8 比 q4 性能更好
  • mixtral:7b q8 和 fp16 量化具有相同的性能。 q2 和 q4 较低
  • llama3:8b q4、q8 和 fp16 量化具有相同的性能。 q2 降低
  • llama3:70b q4、q8 和 fp16 量化具有相同的性能。 q2 降低
  • 尺寸越大并不总是越好
  • qwen:32b 击败了110b 和 72b 版本

数据集

Baisc Financial Q&A v0.3
包含基本的财务问题和解答,重点关注股票和外汇交易。答案很简单,可以通过程序验证。由 ChatGPT 生成。

下載數據

提示模板

You are a financial analyst. Answer the question below based on your knowledge. If you don’t know the answer, just answer you don’t know.
Question: %question%
Response only in JSON format as {“answer”: “your answer”} without any other text. No explanation is required.
If answer is a number, use string type enclosed by double quotes, no $ sign, comma, or unit is needed. Like {“answer”:“1000”}

模型结果
Download Data

testplantestsetmodelcorrectnesscorrectincorrect
0Basic Financial Q&A20240425-1llama3:70b-instruct-q8_00.669322061090
1Basic Financial Q&A20240425-1llama3:70b-instruct-fp160.664522161119
2Basic Financial Q&A20240425-1llama3:70b0.662521951118
3Basic Financial Q&A20240425-1mixtral:8x22b-instruct-v0.1-q8_00.641921561203
4Basic Financial Q&A20240425-1mixtral:8x22b0.605120331327
5Basic Financial Q&A20240425-1wizardlm2:8x22b0.59419961364
6Basic Financial Q&A20240425-1wizardlm2:8x22b-q8_00.592519891368
7Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-q8_00.5649949731
8Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp160.55924756
9Basic Financial Q&A20240425-1qwen:32b0.530717831577
10Basic Financial Q&A20240425-1dbrx:132b0.525818051628
11Basic Financial Q&A20240425-1dolphin-llama3:70b-v2.90.5158865812
12Basic Financial Q&A20240425-1llama3:70b-instruct-q2_K0.5089853823
13Basic Financial Q&A20240425-1mixtral:8x7b-instruct-v0.1-fp160.4952832848
14Basic Financial Q&A20240425-1mixtral:8x7b-instruct-v0.1-q8_00.491216531712
15Basic Financial Q&A20240425-1qwen:110b0.4625777903
16Basic Financial Q&A20240425-1qwen:110b-chat-v1.5-q8_00.4565767913
17Basic Financial Q&A20240425-1mixtral:8x7b0.449515751929
18Basic Financial Q&A20240425-1gemma:7b-instruct-v1.1-q8_00.449115091851
19Basic Financial Q&A20240425-1gemma:7b-instruct-v1.1-fp160.448515071853
20Basic Financial Q&A20240425-1deepseek-llm:67b0.439916532105
21Basic Financial Q&A20240425-1qwen:110b-chat-v1.5-q2_K0.4276667893
22Basic Financial Q&A20240425-1gemma:7b-v1.10.416413991961
23Basic Financial Q&A20240425-1gemma:7b-instruct-v1.1-q2_K0.38336441036
24Basic Financial Q&A20240425-1command-r-plus:104b0.379612862102
25Basic Financial Q&A20240425-1llama3:8b-instruct-fp160.352312212245
26Basic Financial Q&A20240425-1llama3:8b-instruct-q6_K0.350712252268
27Basic Financial Q&A20240425-1llama3:8b-instruct-q8_00.348912142266
28Basic Financial Q&A20240425-1command-r:35b0.342435986911
29Basic Financial Q&A20240425-1qwen:72b0.341211462213
30Basic Financial Q&A20240425-1llama3:8b0.335311722323
31Basic Financial Q&A20240425-1llama2:70b0.332711182242
32Basic Financial Q&A20240425-1phi3:3.8b-mini-instruct-4k-fp160.319611582465
33Basic Financial Q&A20240425-1gemma:7b0.310110422318
34Basic Financial Q&A20240425-1mixtral:8x7b-instruct-v0.1-q2_K0.30925221166
35Basic Financial Q&A20240425-1deepseek-coder:33b0.301616903913
36Basic Financial Q&A20240425-1wizardlm2:7b-fp160.300612322867
37Basic Financial Q&A20240425-1phi3:3.8b0.297111162640
38Basic Financial Q&A20240425-1wizardlm2:7b0.293132127748
39Basic Financial Q&A20240425-1starling-lm:7b0.28849812421
40Basic Financial Q&A20240425-1wizardlm2:7b-q8_00.288426996658
41Basic Financial Q&A20240425-1solar:10.7b0.28189472413
42Basic Financial Q&A20240425-1qwen:14b0.28159462414
43Basic Financial Q&A20240425-1openchat:7b0.27929382422
44Basic Financial Q&A20240425-1wizardlm2:7b-q6_K0.2784671213
45Basic Financial Q&A20240425-1yi:34b0.27419072402
46Basic Financial Q&A20240425-1starling-lm:7b-beta-q8_00.25838682492
47Basic Financial Q&A20240425-1starling-lm:7b-beta-fp160.25714321248
48Basic Financial Q&A20240425-1mistral:7b-instruct-v0.2-fp160.2549202702
49Basic Financial Q&A20240425-1llama2-uncensored:70b0.25218472513
50Basic Financial Q&A20240425-1mistral:7b-instruct-v0.2-q8_00.25188462514
51Basic Financial Q&A20240425-1mistral:v0.20.24269242885
52Basic Financial Q&A20240425-1llama3-gradient:8b-instruct-1048k-q8_00.24174061274
53Basic Financial Q&A20240425-1mistral:7b0.2412203863
54Basic Financial Q&A20240425-1llama3-gradient:8b-instruct-1048k-fp160.23953931248
55Basic Financial Q&A20240425-1qwen:7b0.23767952551
56Basic Financial Q&A20240425-1zephyr:7b0.232623767839
57Basic Financial Q&A20240425-1orca2:13b0.22477552605
58Basic Financial Q&A20240425-1dolphin-llama3:8b-v2.90.20878523231
59Basic Financial Q&A20240425-1neural-chat:7b0.20546902670
60Basic Financial Q&A20240425-1mistral:7b-instruct-v0.2-q2_K0.20183391341
61Basic Financial Q&A20240425-1llama3:8b-instruct-q2_K0.1973311349
62Basic Financial Q&A20240425-1llama2:13b0.19616592701
63Basic Financial Q&A20240425-1codellama:13b0.19436532707
64Basic Financial Q&A20240425-1llama3-gradient:8b-instruct-1048k-q4_00.18683141367
65Basic Financial Q&A20240425-1wizardlm-uncensored:13b0.178914136486
66Basic Financial Q&A20240425-1deepseek-llm:7b0.168344377
67Basic Financial Q&A20240425-1wizardlm2:7b-q2_K0.15652631417
68Basic Financial Q&A20240425-1deepseek-coder:6.7b0.150510615989
69Basic Financial Q&A20240425-1orca2:7b0.1315123395
70Basic Financial Q&A20240425-1yi:6b0.08893043114
71Basic Financial Q&A20240425-1llama2:7b0.08793063174
72Basic Financial Q&A20240425-1gemma:2b-instruct-v1.1-fp160.08751471533
73Basic Financial Q&A20240425-1gemma:2b-instruct-v1.1-q8_00.08041351545
74Basic Financial Q&A20240425-1gemma:2b-v1.10.07622563104
75Basic Financial Q&A20240425-1llama2-uncensored:7b0.07592553105
76Basic Financial Q&A20240425-1gemma:2b0.06792283132
77Basic Financial Q&A20240425-1qwen:4b0.06362143149
78Basic Financial Q&A20240425-1phi:2.7b0.05732293767
79Basic Financial Q&A20240425-1llama2-chinese:7b0.05182755036
80Basic Financial Q&A20240425-1qwen:1.8b0.03721253235
81Basic Financial Q&A20240425-1deepseek-coder:1.3b0.0101343325
82Basic Financial Q&A20240425-1qwen:0.5b0.003103349
83Basic Financial Q&A20240425-1falcon:7b-instruct0.001863353
84Basic Financial Q&A20240425-1qwen-chat:7b003360

检测结果

显示前 100 行。完整数据可以从我们的GitHub 存储库下载

testplantestsetmodelcorrectcorrect_answeranswerquestion
0Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “NVDA”}{“answer”: “NVDA”}During multiple trading sessions, an investor buys 200 shares of NVIDIA (NVDA) at $510, sells 100 at $515, buys 50 more at $512. Buys 300 shares of AMD (AMD) at $85, sells 150 at $90, buys 75 more at $88. Buys 180 shares of Salesforce (CRM) at $250, sells 90 at $255, buys 45 more at $253. Buys 210 shares of PayPal (PYPL) at $190, sells 105 at $195, buys 52 more at $192. Which stock has the highest average selling price? use only stock ticker in answer.
1Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “PYPL”}{“answer”: “CRM”}A portfolio includes: Buys 220 shares of Salesforce (CRM) at $250, sells 110 at $255, buys 55 more at $252. Buys 240 shares of PayPal (PYPL) at $190, sells 120 at $195, buys 60 more at $192. Buys 250 shares of Meta (META) at $275, sells 125 at $280, buys 62 more at $278. Buys 200 shares of Visa (V) at $215, sells 100 at $220, buys 50 more at $217. Which stock has the lowest average purchase price? use only stock ticker in answer.
2Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “META”}{“answer”: “V”}A trader operates with: Buys 180 shares of Meta (META) at $270, sells 90 at $275, buys 45 more at $272. Buys 220 shares of Visa (V) at $210, sells 110 at $215, buys 55 more at $212. Buys 160 shares of Pfizer (PFE) at $40, sells 80 at $42, buys 40 more at $41. Buys 130 shares of IBM (IBM) at $130, sells 65 at $135, buys 30 more at $132. Which stock has the highest total sales cost? use only stock ticker in answer.
3Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “PFE”}{“answer”: “PFE”}Transactions for an investor include: Buys 150 shares of Meta (META) at $270, sells 75 at $275, buys 37 more at $272. Buys 180 shares of Visa (V) at $210, sells 90 at $215, buys 45 more at $212. Buys 160 shares of Pfizer (PFE) at $40, sells 80 at $42, buys 40 more at $41. Buys 130 shares of IBM (IBM) at $130, sells 65 at $135, buys 32 more at $132. Which stock has the lowest total purchase cost? use only stock ticker in answer.
4Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “SPOT”}{“answer”: “ORCL”}A trader engages in these transactions: Buys 100 shares of Oracle (ORCL) at $60, sells 50 at $62, buys 25 more at $61. Buys 120 shares of Spotify (SPOT) at $240, sells 60 at $245, buys 30 more at $242. Buys 80 shares of Coca-Cola (KO) at $50, sells 40 at $52, buys 20 more at $51. Buys 90 shares of Snap Inc. (SNAP) at $10, sells 45 at $12, buys 22 more at $11. Which stock has the highest total profit? use only stock ticker in answer.
5Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “AMD”}{“answer”: “AMD”}A portfolio manager handles these transactions: Buys 250 shares of AMD (AMD) at $90, sells 125 at $95, buys 60 more at $92. Buys 200 shares of Pfizer (PFE) at $45, sells 100 at $47, buys 50 more at $46. Buys 180 shares of IBM (IBM) at $135, sells 90 at $140, buys 40 more at $138. Buys 220 shares of Visa (V) at $210, sells 110 at $215, buys 55 more at $212. Which stock has the highest remaining position? use only stock ticker in answer.
6Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “SNAP”}{“answer”: “SNAP”}Transactions include: Buys 130 shares of Oracle (ORCL) at $65, sells 65 at $67, buys 30 more at $66. Buys 170 shares of Spotify (SPOT) at $245, sells 85 at $250, buys 40 more at $248. Buys 200 shares of Coca-Cola (KO) at $52, sells 100 at $54, buys 45 more at $53. Buys 150 shares of Snap Inc. (SNAP) at $12, sells 75 at $14, buys 35 more at $13. Which stock has the lowest average selling price? use only stock ticker in answer.
7Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “META”}{“answer”: “CRM”}An investor engages with several stocks: Buys 150 shares of Salesforce (CRM) at $250, sells 75 at $255, buys 35 more at $253. Buys 180 shares of PayPal (PYPL) at $190, sells 90 at $195, buys 40 more at $192. Buys 220 shares of Meta (META) at $275, sells 110 at $280, buys 50 more at $278. Buys 160 shares of Visa (V) at $215, sells 80 at $220, buys 30 more at $217. Which stock has the highest total sales revenue? use only stock ticker in answer.
8Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “INTC”}{“answer”: “INTC”}During multiple transactions, an investor handles: 200 shares of Google (GOOGL) at $2250, sells 100 at $2260, buys 50 more at $2255. Buys 300 shares of Intel (INTC) at $55, sells 150 at $57, buys 75 more at $56. Buys 150 shares of Netflix (NFLX) at $520, sells 75 at $525, buys 30 more at $522. Buys 250 shares of NVIDIA (NVDA) at $510, sells 125 at $515, buys 60 more at $512. Which stock has the lowest average cost per remaining share? use only stock ticker in answer.
9Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “AMZN”}{“answer”: “AMZN”}A trader conducts transactions with four stocks: Buys 120 shares of Apple (AAPL) at $160, sells 60 at $165, buys 30 more at $162. Buys 80 shares of Tesla (TSLA) at $620, sells 40 at $625, buys 20 more at $622. Buys 90 shares of Microsoft (MSFT) at $310, sells 45 at $315, buys 15 more at $312. Buys 100 shares of Amazon (AMZN) at $3300, sells 50 at $3325, buys 25 more at $3310. Which stock has the highest total purchase cost? use only stock ticker in answer.
10Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “SPOT”}{“answer”: “ORCL”}A trader manages transactions as follows: Buys 200 shares of Oracle (ORCL) at $60, sells 100 at $62, buys 50 more at $61. Buys 300 shares of Spotify (SPOT) at $240, sells 150 at $245, buys 75 more at $242. Buys 250 shares of Coca-Cola (KO) at $50, sells 125 at $52, buys 60 more at $51. Buys 100 shares of Snap Inc. (SNAP) at $10, sells 50 at $12, buys 25 more at $11. Which stock has the highest remaining position? use only stock ticker in answer.
11Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “PFE”}{“answer”: “PFE”}During a busy trading session, transactions include: 180 shares of Meta (META) bought at $270, 90 sold at $275, 45 bought at $272. 220 shares of Visa (V) bought at $210, 110 sold at $215, 55 bought at $212. 160 shares of Pfizer (PFE) bought at $40, 80 sold at $42, 40 bought at $41. 130 shares of IBM (IBM) bought at $130, 65 sold at $135, 30 bought at $132. Which stock has the lowest total sales revenue? use only stock ticker in answer.
12Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “NVDA”}{“answer”: “NVDA”}A portfolio includes: 150 shares of NVIDIA (NVDA) bought at $500, 75 sold at $510, 25 bought at $505. 200 shares of AMD (AMD) bought at $85, 100 sold at $90, 50 bought at $88. 120 shares of Salesforce (CRM) bought at $250, 60 sold at $255, 30 bought at $252. 90 shares of PayPal (PYPL) bought at $190, 45 sold at $195, 20 bought at $192. Which stock has the highest total profit? use only stock ticker in answer.
13Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “INTC”}{“answer”: “AAPL”}An investor makes these moves: Purchases 100 shares of Apple (AAPL) at $155, sells 50 at $160, buys 25 more at $158. Buys 150 shares of Google (GOOGL) at $2200, sells 70 at $2220, buys 40 more at $2210. Buys 200 shares of Intel (INTC) at $50, sells 100 at $52, buys 50 more at $51. Buys 120 shares of Netflix (NFLX) at $500, sells 60 at $510, buys 30 more at $505. Which stock has the lowest average selling price? use only stock ticker in answer.
14Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “AMZN”}{“answer”: “AMZN”}A trader conducts the following transactions: Buys 50 shares of Apple (AAPL) at $150, sells 30 at $155, buys 20 more at $152. Buys 100 shares of Tesla (TSLA) at $600, sells 50 at $610, buys 30 more at $605. Buys 80 shares of Microsoft (MSFT) at $300, sells 40 at $305, buys 10 more at $302. Buys 70 shares of Amazon (AMZN) at $3200, sells 30 at $3225, buys 15 more at $3210. Which stock has the highest total purchase cost? use only stock ticker in answer.
15Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “-400”}{“answer”: “-4000”}A trader engages in buying 200 shares of Pfizer (PFE) at $42 each and selling at $40 each. What is the total profit or loss from this trade?
16Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-4500”}{“answer”: “-4500”}If a trader buys 150 shares of Google (GOOGL) at $2250 each and later sells them at $2220 each, calculate the profit or loss.
17Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-500”}{“answer”: “-500”}An investor buys 250 shares of AMD (AMD) at $87 each and sells them at $85 each due to a sudden stock decline. What is the profit or loss from this transaction?
18Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-1800”}{“answer”: “-1800”}A trader purchases 180 shares of Netflix (NFLX) at $520 each and sells them at $510 each. What is the profit or loss from this sale?
19Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-1000”}{“answer”: “-1000”}After buying 200 shares of Intel (INTC) at $55 each, a trader sells them at $50 each due to an unexpected market correction. What is the profit or loss?
20Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-3000”}{“answer”: “-3000”}A trader buys 300 shares of NVIDIA (NVDA) at $410 each and sells them at $400 each due to a market downturn. Calculate the profit or loss.
21Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-12500”}{“answer”: “-12500”}During a trading session, a trader acquires 250 shares of Amazon (AMZN) at $3250 each and sells them at $3200 each. What is the profit or loss on this transaction?
22Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-1000”}{“answer”: “-1000”}A portfolio manager buys 200 shares of Microsoft (MSFT) at $305 each and sells them at $300 each. What is the profit or loss from these transactions?
23Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-1500”}{“answer”: “-1500”}An investor purchases 150 shares of Tesla (TSLA) at $620 each and sells them at $610 each. What is the profit or loss from this sale?
24Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “-500”}{“answer”: “-500”}A trader buys 100 shares of Apple (AAPL) at $155 each and sells them at $150 each. Calculate the profit or loss from this transaction.
25Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “400”}{“answer”: “400”}A trader engages in a transaction where 200 shares of Pfizer (PFE) are bought at $42 and sold at $40. How much total loss does the trader suffer from this trade?
26Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “4500”}{“answer”: “3000”}If a trader purchases 150 shares of Google (GOOGL) at $2250 and later sells them at $2220, what is the total loss?
27Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “500”}{“answer”: “500”}An investor buys 250 shares of AMD (AMD) at $87 and sells them at $85 due to a sudden stock decline. Calculate the loss incurred from this transaction.
28Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “1800”}{“answer”: “1800”}A trader acquires 180 shares of Netflix (NFLX) at $520 each and sells them at $510. What is the total loss from this sale?
29Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “1000”}{“answer”: “1000”}After purchasing 200 shares of Intel (INTC) at $55 each, a trader sells them at $50 each due to an unexpected market correction. How much money does the trader lose?
30Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “3000”}{“answer”: “3000”}A trader buys 300 shares of NVIDIA (NVDA) at $410, and the market price drops, leading to a sale at $400. What is the financial loss from this downturn?
31Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “12500”}{“answer”: “12500”}During a trading session, a trader buys 250 shares of Amazon (AMZN) at $3250 and is forced to sell them at $3200 due to a market dip. What is the total loss on this transaction?
32Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “1000”}{“answer”: “1000”}A portfolio manager purchases 200 shares of Microsoft (MSFT) at $305 and sells 200 shares at $300. What is the loss incurred from these Microsoft transactions?
33Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “1500”}{“answer”: “-1500”}An investor buys 150 shares of Tesla (TSLA) at $620 and sells them at $610. Calculate the total financial loss from selling these shares.
34Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “500”}{“answer”: “500”}A trader buys 100 shares of Apple (AAPL) at $155 and then sells them at $150. What is the total loss from this transaction?
35Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “100”}{“answer”: “100”}A trader engages in transactions with Pfizer (PFE), buying 200 shares at $40 and another 300 shares at $42, and sells 400 shares at $45. What is the number of Pfizer shares left after these transactions?
36Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “50”}{“answer”: “50”}After buying 90 shares of Google (GOOGL) at $2200 and 110 shares at $2220, then selling 150 shares at $2250, calculate the number of Google shares still owned by the trader.
37Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “100”}{“answer”: “100”}A trader purchases 120 shares of AMD (AMD) at $85, another 180 at $87, and then sells 200 shares at $90. How many AMD shares remain after the sale?
38Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “50”}{“answer”: “250”}An investor buys 150 shares of Netflix (NFLX) at $500, buys 200 more shares at $510, and sells 300 shares at $520. After these transactions, how many shares of Netflix does the investor have left?
39Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “55”}{“answer”: “54.2”}A trader buys 100 shares of Intel (INTC) at $50, another 200 shares at $55, and sells 250 shares at $60. What is the average purchase price of the remaining Intel shares?
40Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “150”}{“answer”: “250”}During multiple transactions, a trader acquires 300 shares of NVIDIA (NVDA) at $400, buys an additional 200 at $405, and sells 350 at $410. How many NVIDIA shares does the trader still own?
41Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “150”}{“answer”: “250”}A portfolio manager buys 180 shares of Amazon (AMZN) at $3200 and another 120 at $3225, then sells 150 shares at $3250. Calculate the total number of Amazon shares remaining.
42Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “100”}{“answer”: “400”}An investor purchases 250 shares of Tesla (TSLA) at $610, an additional 250 shares at $615, and then sells 400 shares at $620. What is the total number of Tesla shares left after the sale?
43Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “303”}{“answer”: “304.5”}A trader buys 200 shares of Microsoft (MSFT) at $300, 150 shares at $305, and sells 100 shares at $310. What is the average cost of the remaining Microsoft shares?
44Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “30”}{“answer”: “30”}After buying 50 shares of Apple (AAPL) at $150 and another 100 shares at $155, and then selling 120 shares at $160, how many shares of Apple does the trader have left?
45Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “90”}{“answer”: “90”}During a busy trading day, an investor buys 100 shares of AMD (AMD) at $85, another 150 shares at $87, and sells 200 shares at $90. They also buy shares of Pfizer (PFE) but do not sell any. What is the average selling price of the AMD shares sold?
46Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “109500”}{“answer”: “28500”}A trader engages in multiple transactions: Buys 300 shares of NVIDIA (NVDA) at $400, buys 200 more at $405, and sells 250 at $410. Additionally, they purchase 150 shares of Intel (INTC) at $48. Ignoring the Intel shares, what is the total cost of the remaining NVIDIA shares?
47Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “150”}{“answer”: “30”}A portfolio manager buys 120 shares of Netflix (NFLX) at $520, another 180 shares at $530, and sells 150 shares at $540. They also handle 200 shares of Amazon (AMZN) buying at $3100. Calculate the number of Netflix shares remaining after the sale.
48Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “610”}{“answer”: “605”}An investor purchases 200 shares of Tesla (TSLA) at $600, buys an additional 300 shares at $610, and sells 400 shares at $620. They then buy 100 shares of Google (GOOGL) at $2200. Without using the Google transactions, what is the average purchase price of the remaining Tesla shares?
49Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “50”}{“answer”: “50”}A trader buys 100 shares of Apple (AAPL) at $150, 150 shares at $155, and then sells 200 shares at $160. They also buy 100 shares of Microsoft (MSFT) at $220. After selling the Apple shares, how many shares of Apple does the trader still own?
50Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “2040”}{“answer”: “570”}A trader operates as follows: Buys 120 shares of AMD (AMD) at $85, buys another 180 shares at $87, and sells 250 shares at $90. They also engage in transactions with Pfizer (PFE), buying 100 shares at $40, buying another 200 shares at $42, and selling 220 shares at $45. Calculate the profit from all transactions.
51Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “9200”}{“answer”: “3000”}The following transactions are made by a trader: Buys 300 shares of NVIDIA (NVDA) at $400, buys an additional 200 shares at $410, and sells 400 shares at $420. The same trader buys 100 shares of Intel (INTC) at $50, purchases 150 more at $55, and sells 200 shares at $60. What is the total profit from these trades?
52Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “17800”}{“answer”: “45000”}A portfolio includes the following activities: 200 shares of Netflix (NFLX) are bought at $500 each, 100 more shares at $510, and 250 shares are sold at $520. Meanwhile, 150 shares of Google (GOOGL) are purchased at $2200, 200 shares are bought at $2220, and 300 shares are sold at $2250. Calculate the total profit.
53Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “13500”}{“answer”: “500”}An investor makes these moves: Purchases 50 shares of Microsoft (MSFT) at $300, buys 70 more shares at $305, and sells 100 shares at $310. Additionally, they buy 80 shares of Amazon (AMZN) at $3200, purchase another 120 shares at $3250, and sell 150 shares at $3300. What is the overall profit from all transactions?
54Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “2050”}{“answer”: “250”}A trader conducts the following transactions: Buys 100 shares of Apple (AAPL) at $150, buys another 150 shares at $155, and sells 200 shares at $160. The trader also buys 50 shares of Tesla (TSLA) at $600, buys an additional 100 shares at $610, and sells 120 shares at $620. What is the total profit from these transactions?
55Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “164”}{“answer”: “1500”}If a Forex trader purchases €15,000 worth of USD/EUR at 0.9100 and sells at 0.9200, what is the profit in EUR?
56Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “189”}{“answer”: “2000”}An investor buys 200,000 JPY with USD at a USD/JPY rate of 105.00 and sells the JPY when the rate changes to 106.00. What is the profit in USD?
57Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “500”}{“answer”: “5000”}A trader buys 100,000 units of NZD/USD at 0.6400 and sells at 0.6450. What is the total profit?
58Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “1600”}{“answer”: “200”}If a trader buys USD/CAD at 1.2500 and sells at 1.2600 with an investment of $20,000, what is the profit in CAD?
59Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “357”}{“answer”: “250”}A Forex trader invests $5,000 in AUD/USD at 0.7000 and sells at 0.7050. What is the total profit from this transaction?
60Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “59”}{“answer”: “500”}If a trader buys EUR/GBP at 0.8500 with €10,000 and the rate rises to 0.8550, how much profit is made in GBP?
61Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “227”}{“answer”: “2500”}A trader purchases 50,000 units of USD/JPY at 110.00 and sells them at 110.50. What is the profit in USD?
62Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “127”}{“answer”: “195.45”}An investor exchanges $15,000 USD into EUR/USD at a rate of 1.1800. If the rate later rises to 1.1900 and they convert back to USD, what is the profit?
63Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “500”}{“answer”: “5000”}If a trader buys 100,000 units of GBP/USD at 1.3000 and sells at 1.3050, what is the total profit?
64Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “440”}{“answer”: “500”}A trader buys EUR/USD at 1.1350 and sells it at 1.1400. If the trader invested $10,000, what is the profit in USD?
65Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “44100”}{“answer”: “22050”}An investor buys 200 shares of Visa (V) at $210 each and the stock’s value increases by 5%. What is the total value of the shares after the increase?
66Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “4000”}{“answer”: “4000”}A trader buys 500 shares of Meta (META) at $270 each and sells 400 shares at $280 each. What is the total profit from selling these 400 shares?
67Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “23”}{“answer”: “23”}If 450 shares are bought for $22 each and sold for a total of $10,350, what was the selling price per share?
68Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “250”}{“answer”: “250”}A trader purchases 200 shares of Spotify (SPOT) at $240 each. If the total investment is sold for $50,000, what was the selling price per share?
69Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “750”}{“answer”: “750”}If a person buys 150 shares of Oracle (ORCL) at $60 and the stock price increases by $5, what is the total increase in value of the investment?
70Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “180”}{“answer”: “1800”}A trader buys 90 shares of Snap Inc. (SNAP) at $12 each and sells them later at $10 each. What is the total loss on this transaction?
71Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “10725”}{“answer”: “14250”}If a trader buys 75 shares of IBM (IBM) at $130 each and the stock price increases by 10%, what is the total investment worth now?
72Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “80”}{“answer”: “80”}An investor purchases 200 shares of Disney (DIS) and sells 120 shares. How many shares remain?
73Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “1050”}{“answer”: “1750”}If 350 shares of AMD (AMD) are purchased at $85 each and sold at $82 each, what is the total financial loss?
74Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “500”}{“answer”: “5000”}A trader buys 250 shares of Coca-Cola (KO) at $45 each and sells them at $47 each. What is the total profit?
75Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “3450”}{“answer”: “3450”}If a stock portfolio containing shares of Pfizer (PFE) valued at $3000 gains 15% in value, what is the new total value of the portfolio?
76Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “300”}{“answer”: “3000”}A trader sells 150 shares of Intel (INTC) that were bought at $48 each for $50 each. What is the total profit from the transaction?
77Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “50”}{“answer”: “50”}A company’s stock in Boeing (BA) is bought for $10,000 total at $200 per share. If the price per share rises to $205, how many shares were originally purchased?
78Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “10”}{“answer”: “10”}During a trading day, an investor buys 120 shares of NVIDIA (NVDA) at 10 AM for $500 each and sells all at 3 PM for $510 each. What is the net profit per share?
79Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “1833”}{“answer”: “1820”}If 100 shares of Google (GOOGL) are bought at $1800 each and another 150 shares are bought at $1850 each, what is the average cost per share?
80Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “8000”}{“answer”: “80000”}A trader buys 80 shares of Amazon (AMZN) at $3100 each and sells them at $3000 each due to a market decline. What is the total loss?
81Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “52000”}{“answer”: “52000”}An investor buys 200 shares of Netflix (NFLX) at $250 each and sells them at $260 each. What is the total revenue from the sale?
82Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “350”}{“answer”: “350”}If a trader executes two separate transactions, buying 150 shares and then 200 shares of Microsoft (MSFT), what is the total number of shares purchased?
83Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “25”}{“answer”: “25”}A trader buys 100 shares of Tesla (TSLA) at $600 each and sells 75 of them at $610 each. How many shares does the trader have left?
84Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “250”}{“answer”: “2500”}If an investor buys 50 shares of Apple Inc. (AAPL) at $150 each and sells them at $155 each, what is the total profit?
85Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “5500”}{“answer”: “5500”}An investor buys 100 shares at $50 each and the stock’s value increases by 10%. What is the total value of the shares after the increase?
86Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “3000”}{“answer”: “5000”}A trader buys 400 shares at $40 each and sells 300 shares at $50 each. What is the total profit from selling these 300 shares?
87Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “16.5”}{“answer”: “16.5”}If 600 shares are bought for $15 each and sold for a total of $9900, what was the selling price per share?
88Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “27”}{“answer”: “27”}A trader purchases 200 shares at $25 each. If the total investment is sold for $5400, what was the selling price per share?
89Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “300”}{“answer”: “300”}If a person buys 100 shares at $30 and the stock price increases by $3, what is the total increase in value of the investment?
90Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “500”}{“answer”: “500”}A trader buys 100 shares at $75 each and sells them later at $70 each. What is the total loss on this transaction?
91Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “5250”}{“answer”: “5250”}If a trader buys 50 shares at $100 each and the stock price increases by 5%, what is the total investment worth now?
92Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “150”}{“answer”: “150”}An investor purchases 400 shares of a company and sells 250 shares. How many shares remain?
93Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “1000”}{“answer”: “1000”}If 500 shares are purchased at $22 each and sold at $20 each, what is the total financial loss?
94Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “900”}{“answer”: “900”}A trader buys 300 shares at $15 each and sells them at $18 each. What is the total profit?
95Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “2200”}{“answer”: “2200”}If a stock portfolio valued at $2000 gains 10% in value, what is the new total value of the portfolio?
96Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “600”}{“answer”: “600”}A trader sells 120 shares of a stock that were bought at $25 each for $30 each. What is the total profit from the transaction?
97Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “20”}{“answer”: “20”}A company’s stock is bought for $1000 total at $50 per share. If the price per share rises to $55, how many shares were originally purchased?
98Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16True{“answer”: “5”}{“answer”: “5”}During a trading day, an investor buys 100 shares at 10 AM for $50 each and sells all at 3 PM for $55 each. What is the net profit per share?
99Basic Financial Q&A20240425-1qwen:32b-chat-v1.5-fp16False{“answer”: “43”}{“answer”: “42.5”}If 150 shares are bought at $40 each and another 200 shares are bought at $45 each, what is the average cost per share?

Key Findings

Llama3:70b are top performers, followed by Mixtral and WizardLM2 8x22b MoE, and qwen:32b

modelcorrectnesscorrectincorrect
0llama3:70b-instruct-q8_066.9322061090
1llama3:70b-instruct-fp1666.4522161119
2llama3:70b66.2521951118
3mixtral:8x22b-instruct-v0.1-q8_064.1921561203
4mixtral:8x22b60.5120331327
5wizardlm2:8x22b59.419961364
6wizardlm2:8x22b-q8_059.2519891368
7qwen:32b-chat-v1.5-q8_056.49949731
8qwen:32b-chat-v1.5-fp1655924756
9qwen:32b53.0717831577

llama3:70b has much better performance than other 70b models

modelcorrectnesscorrectincorrect
2llama3:70b66.2521951118
11dolphin-llama3:70b-v2.951.58865812
20deepseek-llm:67b43.9916532105
29qwen:72b34.1211462213
31llama2:70b33.2711182242
49llama2-uncensored:70b25.218472513

qwen:110b q4 quantization has slightly better performance than q8. q2 is lower

modelcorrectnesscorrectincorrect
15qwen:110b46.25777903
16qwen:110b-chat-v1.5-q8_045.65767913
21qwen:110b-chat-v1.5-q2_K42.76667893

starling-lm:7b q4 quantization has better performance than fp16 and q8

modelcorrectnesscorrectincorrect
39starling-lm:7b28.849812421
46starling-lm:7b-beta-q8_025.838682492
47starling-lm:7b-beta-fp1625.714321248

phi3:3.8b fp16 quantization has better performance than q4

modelcorrectnesscorrectincorrect
32phi3:3.8b-mini-instruct-4k-fp1631.9611582465
37phi3:3.8b29.7111162640

phi3:3.8b has much better performance than phi2:2.7b

modelcorrectnesscorrectincorrect
37phi3:3.8b29.7111162640
78phi:2.7b5.732293767

gemma:7b-v1.1 has best performance among 7b to 13b models, much beter than llama3:8b

modelcorrectnesscorrectincorrect
22gemma:7b-v1.141.6413991961
30llama3:8b33.5311722323
37phi3:3.8b29.7111162640
38wizardlm2:7b29.3132127748
39starling-lm:7b28.849812421
41solar:10.7b28.189472413
43openchat:7b27.929382422
53mistral:7b2412203863
55qwen:7b23.767952551
56zephyr:7b23.2623767839
57orca2:13b22.477552605
58dolphin-llama3:8b-v2.920.878523231
59neural-chat:7b20.546902670
62llama2:13b19.616592701
63codellama:13b19.436532707
65wizardlm-uncensored:13b17.8914136486
66deepseek-llm:7b168344377
69orca2:7b13.15123395
71llama2:7b8.793063174
75llama2-uncensored:7b7.592553105
78phi:2.7b5.732293767
79llama2-chinese:7b5.182755036
83falcon:7b-instruct0.1863353

qwen:32b q8 and fp16 quantization has better performance than q4

modelcorrectnesscorrectincorrect
7qwen:32b-chat-v1.5-q8_056.49949731
8qwen:32b-chat-v1.5-fp1655924756
9qwen:32b53.0717831577

gemma:2b-v1.1 is better than v1.0

modelcorrectnesscorrectincorrect
74gemma:2b-v1.17.622563104
76gemma:2b6.792283132

gemma:2b-v1.1 fp16 quatization better than q8 and q4

modelcorrectnesscorrectincorrect
72gemma:2b-instruct-v1.1-fp168.751471533
73gemma:2b-instruct-v1.1-q8_08.041351545
74gemma:2b-v1.17.622563104

gemma:7b-v1.1 has better performance than v1.0

modelcorrectnesscorrectincorrect
22gemma:7b-v1.141.6413991961
33gemma:7b31.0110422318

gemma:7b-v1.1 q8 and fp16 quatization are better than q4, q2 lower

modelcorrectnesscorrectincorrect
18gemma:7b-instruct-v1.1-q8_044.9115091851
19gemma:7b-instruct-v1.1-fp1644.8515071853
22gemma:7b-v1.141.6413991961
23gemma:7b-instruct-v1.1-q2_K38.336441036

dolphin-llama3:70b lower its performance than original llama3:70b

modelcorrectnesscorrectincorrect
0llama3:70b-instruct-q8_066.9322061090
1llama3:70b-instruct-fp1666.4522161119
2llama3:70b66.2521951118
11dolphin-llama3:70b-v2.951.58865812
12llama3:70b-instruct-q2_K50.89853823

dolphin-llama3:8b lower its performance than original llama3:8b

modelcorrectnesscorrectincorrect
25llama3:8b-instruct-fp1635.2312212245
26llama3:8b-instruct-q6_K35.0712252268
27llama3:8b-instruct-q8_034.8912142266
30llama3:8b33.5311722323
58dolphin-llama3:8b-v2.920.878523231
61llama3:8b-instruct-q2_K19.73311349

llama3-gradient has larger context window but significantlly lower the performance

modelcorrectnesscorrectincorrect
25llama3:8b-instruct-fp1635.2312212245
26llama3:8b-instruct-q6_K35.0712252268
27llama3:8b-instruct-q8_034.8912142266
30llama3:8b33.5311722323
52llama3-gradient:8b-instruct-1048k-q8_024.174061274
54llama3-gradient:8b-instruct-1048k-fp1623.953931248
61llama3:8b-instruct-q2_K19.73311349
64llama3-gradient:8b-instruct-1048k-q4_018.683141367

mixtral:8x22b MoE has better performance than WizardLM2:8x22b

modelcorrectnesscorrectincorrect
3mixtral:8x22b-instruct-v0.1-q8_064.1921561203
4mixtral:8x22b60.5120331327
5wizardlm2:8x22b59.419961364
6wizardlm2:8x22b-q8_059.2519891368

wizardlm2:7b performance vary. fp16 is the best and q4 beats q8 and q6

modelcorrectnesscorrectincorrect
36wizardlm2:7b-fp1630.0612322867
38wizardlm2:7b29.3132127748
40wizardlm2:7b-q8_028.8426996658
44wizardlm2:7b-q6_K27.84671213
67wizardlm2:7b-q2_K15.652631417

wizardlm2:8x22b q4 and q8 quantization has the same performance

modelcorrectnesscorrectincorrect
5wizardlm2:8x22b59.419961364
6wizardlm2:8x22b-q8_059.2519891368

mixtral:8x22b q8 has better performance than q4

modelcorrectnesscorrectincorrect
3mixtral:8x22b-instruct-v0.1-q8_064.1921561203
4mixtral:8x22b60.5120331327

mixtral:7b q8, and fp16 quantization have the same performance. q2 and q4 lower

modelcorrectnesscorrectincorrect
48mistral:7b-instruct-v0.2-fp1625.49202702
50mistral:7b-instruct-v0.2-q8_025.188462514
51mistral:v0.224.269242885
53mistral:7b2412203863
60mistral:7b-instruct-v0.2-q2_K20.183391341

llama3:8b q4, q8, and fp16 quantization have the same performance. q2 lower

modelcorrectnesscorrectincorrect
25llama3:8b-instruct-fp1635.2312212245
26llama3:8b-instruct-q6_K35.0712252268
27llama3:8b-instruct-q8_034.8912142266
30llama3:8b33.5311722323
61llama3:8b-instruct-q2_K19.73311349

llama3:70b q4, q8, and fp16 quantization have the same performance. q2 lower

modelcorrectnesscorrectincorrect
0llama3:70b-instruct-q8_066.9322061090
1llama3:70b-instruct-fp1666.4522161119
2llama3:70b66.2521951118
12llama3:70b-instruct-q2_K50.89853823

Larger size not always better

modelcorrectnesscorrectincorrect
10dbrx:132b52.5818051628
15qwen:110b46.25777903
16qwen:110b-chat-v1.5-q8_045.65767913
21qwen:110b-chat-v1.5-q2_K42.76667893
24command-r-plus:104b37.9612862102

qwen:32b beats its 110b and 72b versions

modelcorrectnesscorrectincorrect
7qwen:32b-chat-v1.5-q8_056.49949731
8qwen:32b-chat-v1.5-fp1655924756
9qwen:32b53.0717831577
15qwen:110b46.25777903
16qwen:110b-chat-v1.5-q8_045.65767913
21qwen:110b-chat-v1.5-q2_K42.76667893
29qwen:72b34.1211462213
42qwen:14b28.159462414
55qwen:7b23.767952551
77qwen:4b6.362143149
80qwen:1.8b3.721253235
82qwen:0.5b0.3103349
84qwen-chat:7b003360
  • 19
    点赞
  • 28
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值