为什么ChatGPT/GPT在多位整数乘法、数数问题上容易出错 - 大语言模型LLM

之前我写过一篇文章分析为什么LLM连简单乘法的都算错以及如何解决： 为何简单的乘法ChatGPT会算错？

我尝试了新的技术路线来进行训练：不使用TransformerDecoder，而是改用TransformerEncoder+CTCLoss进行训练，在3位数整数乘法上达到了约99.994%的推理精度。

针对数数问题，为了防止任务过于简单，我将训练数据集改复杂了一些：

How many letters are there in the following string: "3xwdggagrxd4tm670gfs7n"?

模型在数数时，需要过滤掉“数字”，只统计“字母”的数量。单卡训练1小时左右，测试结果如下：

How many letters are there in the following string: "13hp899sb0k1cph37jk91xubjkqatbcj7q7vzou9b0mv5ub50oa36pt89999vlq7t6u7b7z59dx3dqdbdksl5snoszorzuvbh"?
 63 != 64
How many letters are there in the following string: "324qsa4m2t0610919u24ihqso07gua9k03d06htyv2a4339q0ngkonnw9706grzs4t1"?
 35 != 34
How many letters are there in the following string: "br3elr726fnhga53qdrxj8rsvh28usj62u708sv1xo7e2or1ra75389pbt36gnmt0xmowe0jl5o0crmibnvo6845z0lu16v913"?
 69 != 59
How many letters are there in the following string: "14812962hco6c3685b7r2x0olerv2pszmsq71aj85bbin6t2j1rgv27g11x9970exf8iufmonxt2fosd5fg8cb338jeus13ag"?
 57 != 56
How many letters are there in the following string: "x385whs4cri12093110r4dik9433q6af7mv00pb31m1q00iba6y4zx071wie"?
 30 != 29
How many letters are there in the following string: "p4tjrrsxp341300jbv34z950s624fv58uum8jm5t4mzfwd311579sfd8q6ce8ym02zggon7coti4w0s12443868u1vmr"?
 41 != 49
How many letters are there in the following string: "0vvoxjwcevlz7twkpgbrd5ld2wtzrpgfi5vegwlfaqotszitywpadqq13fnebgs8wj01hskytg7hxeqzj7tpaz4j0izsaywpq"?
 82 != 83
How many letters are there in the following string: "sz6eylx0a4jigeqmsmtqqomhpa0fjaldyrnuci4hs87urfnuxoohhql1uii4r36752uooc3syfcvysmv0ardltzfmlwrbali"?
 89 != 80
How many letters are there in the following string: "5365m7omri4mpi8l77o39op8d8u7pr1g6m9o4bmai9fai1q6k0x93j86hz09i28qbmmybzrc4m61l6ixfkj27n3f6ldz7nx67v"?
 58 != 57
How many letters are there in the following string: "qnr6idr7nmmdmrjbpfvredsmap9c7zw4wz9v0ki30wi568pe6ouuhz438849782xendoq7kna2akjr8tafnfqflzxhhvoysrb"?
 71 != 72
How many letters are there in the following string: "wvup3oj933yxfpfpsuqphgspauk6jcfrnvvhdq26zxw3y2xnxzrxqvt982k93kqkdpvupwzlcsfjklojbgkp4"?
 60 != 70
How many letters are there in the following string: "xvrdz6jzd3qnrpyttaemtftfmjpmkukf9y8oo2ahsm97wcbe8rbaqb5opvr7xbnsvw8qasr3qbomkgdkrxyzwsrkij6oa7rouvz"?
 84 != 85
How many letters are there in the following string: "ms13yxoy54rvmmeb6ywip7b463m69p86u8ct201812t2is7839mqfqb85kbg4vh3gr1g49lwow555lygdwivihdaf3boxn3w5qj"?
 62 != 61
How many letters are there in the following string: "uibt7lrzln9xd259av3zqfgbjvf9l3ag0o71p4eq8eo88zrq82j746v6p1xwz7xd4p873r1t5y915z5dc3n7p56x3"?
 50 != 51
How many letters are there in the following string: "22um5065f00g1m173ve4l60n8t9ohmc8yzy5xb0i9rat431id219ev3929kvdt81nbigxvid0rhuuoy"?
 46 != 45
How many letters are there in the following string: "fj233k1jw1pn9ae8e1f8xlssb9803b28ln8m88gje524d079x7j383h2s3ch"?
 31 != 30
How many letters are there in the following string: "6xrih6sjjx9gvumexlsynvnulgsqrkcvg2ujeo02m30dc2namuhk7u5mkbfwixsj7qiqfnpfrhpfsqaf4rv6hdrsykjvzz6ejn"?
 82 != 83
How many letters are there in the following string: "p951xs83vp9h0t192m1138174jtv85otze63kjigv0pb6dlbr0v3jhkydrqcv5qr2zljjq2cltkdgfp6jre7gfybd42n3"?
 69 != 60
How many letters are there in the following string: "tw159948g5a248fmxf4uqzck5379s0l0rtai17782q5vz3213z1y8l5edg8412mnwrn27emv9u3"?
 39 != 37
19/100000  99.98100%

随机生成100000条样本，错了19个，测试准确率约为99.98100%

针对问题中提到的xVal论文，其实可以将论文中的Number head替换成CTCLoss，这样正常的token按标准GPT的方式进行训练，碰到NUM token就用CTCLoss来解码。因为CTCLoss在语音识别、文字识别等场景中已经得到了充分的验证，相对而言比较可靠。还有就是CTCLoss可以解码任意形式的不定长序列，而不仅仅是数字。另外xVal论文中用的是Adjusted R2 score，具体在3、4位整数乘法上准确率是多少，论文中都没写。

---

2024.04.15更新

这几天关注到Cohere公司开源的大模型Command R+，通过对Command R+的测试，我意识到Transformer Decoder-Only远比我之前认识的强大。受Command R+的启发，我重新设计了基于Transformer Decoder-Only的GPT模型方案，通过合成大量的100以内数数训练样本，使用CrossEntropyLoss损失函数进行梯度更新。单卡训练约2个小时后，训练损失几乎为0，测试结果如下：

[
'How many letters are there in the following string: "jzlbkuvlf4luvj947wlijcsgnss1jsiygm4vvmzx7kyqsfqlxht5muhbgzzrxy5tyethoqdvjypufl2x"?  69, 70(expected)', 'How many letters are there in the following string: "wbxn47zcmvyjllto6u2hrphnytc0qogqfj9ch8iu7yarkbgc2erbgvjqhyelbch8r7vwncbvpmsxyjonh"?  69, 70(expected)', 
'How many letters are there in the following string: "nm783ooc0va45s71z5s1wu6cno14x5f0wl1f0v8ke1oh7v35wnihnfmxkelk6j5113ybt6mp677tx0v8kl574y9tsytbwc957"?  56, 57(expected)', 
'How many letters are there in the following string: "honxr0jvfhwcnctm9jf4qifdsbrrjvfbzg1aag1u8feijgpossl8yh9eshbknrwpu96m4kxebfskgi06zkdtkauqvvljfakroi"?  85999999, 85(expected)', 
'How many letters are there in the following string: "10lo3fx7p29570wzr9158oa3ky6ij7ra7b979p6j9e11b441967kvgvoeg3yvuyyvf6265qgjf"?  40, 39(expected)', 
'How many letters are there in the following string: "vs2ebarz7ou9ta4ukcuqlvnas3ksf0sngn1kwjyipvijxortcspvsswwqzrxzeqjbbgft5pwzzbay1t9canve2csxmfwaazmgh"?  99, 87(expected)', 
'How many letters are there in the following string: "mj2divjsqz79tdbuzrdrirev8w80op8dqkpfjbuioptk6olyddsasl6l0mavmhmgdhfibiqlrvhdqebbuapgewpucfac5mj"?  85, 84(expected)', 
'How many letters are there in the following string: "b5a6gvn3m1zd7lfz7h2683w26gl8z14q2ico33ggqf115x13t2q94g8v29yjc77l3l58"?  35, 34(expected)', 
'How many letters are there in the following string: "x5kbkdhbmhf2zga5wpjlb2vfx523u59685k371h823gdf3t767l2do666mgs1ecos7va4pa46vhn5yg09275r3i894hih34d4b"?  55, 54(expected)', 
'How many letters are there in the following string: "p8d6e7hj7imr0se4482jz3mz12y4mqq261c26f8l4299ji3y6t8r6"?  25, 26(expected)', 
'How many letters are there in the following string: "i8nb7vi996fnd6iltr5p0fivtqfgjqtel7dmxmu0pm3rbqihhgikwxaigh0xopwwpgbxvlqn1prtupkllo4o"?  69, 70(expected)', 
'How many letters are there in the following string: "hacpg3imt1oncfavx4boeksvv6krxzryms13gkgsp7vuvjgakznvy3ogyzlevg0rqymtj3ufbxsfk5gaz"?  69, 70(expected)', 
'How many letters are there in the following string: "8l72108j49ou677q5d4p93bzuk14h045atkvs3o0m6ak1pwpc2q94r1wjrtf6lfj9rk7ayb8o880zajvbe6a666v8yyu3d2em0"?  56, 55(expected)', 
'How many letters are there in the following string: "dcnytn4cp97638v048nb27efc8uho22qw15er65476rp2295czhxr5s85uqjzrbfz320pq"?  40, 39(expected)', 
'How many letters are there in the following string: "p3992g92638s98za5u3cluj98l015i377nclh096w0psrwun"?  24, 23(expected)', 
'How many letters are there in the following string: "ijdskcdhzixppk7odzdwtdnaktchkcvgte"?  34, 33(expected)', 
'How many letters are there in the following string: "mt2kdrxougoo0srgdbjpcddxxhcayfzlksvgspxpqsrxqnhr"?  45, 46(expected)', 
'How many letters are there in the following string: "xh2ds9ayhioarancokuvicnyjhipvqrngjaysxzda"?  41, 39(expected)', 
'How many letters are there in the following string: "juoitopv1swkwssvdimynopjmcwbkxubq4anvnhxjflchxsfdauunngmsfs"?  59, 57(expected)', 
'How many letters are there in the following string: "syr1nao6aavgofoqodoorpvzlujvfsdy7l2ip189gefdo3dmpitnceikoncjqzxt7zfilbgfjwrqry5nltfm1r7kx9mha4rwddn"?  84, 85(expected)', 
'How many letters are there in the following string: "jlyce2776d8vfju2d51wgl9xh491dg83obw15z8fkgy2sf1vf0t2fra99p342xh7bt8vt8mkr0f78rx6hs99r47uhi4cw8c3f1"?  56, 57(expected)', 
'How many letters are there in the following string: "6tnjn61d4v06x1j922vw566w9nasyw2759406rr8893c"?  20, 19(expected)', 
'How many letters are there in the following string: "cs1eztcsd0sizbtont7uxceumilwrkulyxd1b3dei2dtiewljkpxue13lfy4jmbsfbebcm64ruazaskvq"?  69, 70(expected)', 
'How many letters are there in the following string: "205507fss10114r8iam08j0rxkthl1yfdhud9uvyt8muc53vhh"?  29, 30(expected)', 
'How many letters are there in the following string: "nwwor64nh2h3yrodfcxsk4717majg107781a2pi1bl049z27o06po477d83x9ptb68up7wb"?  40, 39(expected)', 
'How many letters are there in the following string: "y3mjzndgroqzqdkqjjckjm9csougt2ivtvlmfugr0okdfes4idakltvzzyoth1nsziviut0ioheak"?  69, 70(expected)'
]
26/100000  99.97400%

随机生成100000条样本，错了26个，测试准确率约为99.97400%

总结：对于100以内的数数问题，若按“single character”分词，其实就是数一共有多少个符合要求的token，使用Decoder-Only模型是可以训练到99.9%以上测试准确率的。但实际上在训练大语言模型之前，为了降低token个数，一般不会按“single character”分词。我们以按“single word”分词为例，对LLM来说，这个问题已经不是数数问题了，需要先学会vocabulary中每个word所包含字母的个数，然后再做加法。

从模型角度而言，看到的训练样本类似于：

How many letters are there in the following string: "W2101 W5 W5 W32 W450"?
  The answer is 50.
How many letters are there in the following string: "W700 W51 W1203 W2 W1203 W990"?
  The answer is 61.
How many letters are there in the following string: "W3200 W927 W5 W2340 W21 W7"?
  The answer is 72.
......

$W_i(0<i<=N)$表示一个单词，N通常大于1000，其难度在于单词$W_i$包含多少个字母是未知的，需要模型从训练样本中去学习，另外题目中要求不用CoT，也就是说必须用“心算”，不能够先将单词$W_1、W_2、W_3、......、W_N$所包含的字母个数写下来以方便后续计算。

---

上文提到如果不按“single character”分词会导致数数问题复杂化，那按“single word”分词的话，LLM还能学会“数数”吗？为了回答这个问题，我合成了新的训练数据集：

how many letters are there in the following string: "lock over constitution smart boil superior patient teenager graduation drop speaker pronounce contribution boring step carpet realize format surprise disappoint promote track thick rank affect nurse preparation armchair data warn pint construction tale organization tank wear understand vast tremble"? 261
how many letters are there in the following string: "european contradictory"? 21
how many letters are there in the following string: "letter tower land rather someone least attitude tease access decide twice bone rot mother laboratory translator apologize fountain game wrist sex speaker view obvious joke pocket fall tv quit microscope suffering prison hope defeat pork choke category cost honour package multiply tobacco interview institute anything messy due deed tv reform ours saying want geometry however turning register textbook paint power dinosaur cup climb construct as character see meanwhile beyond concentrate stamp concentrate"? 436
how many letters are there in the following string: "pack keyboard anyone dress"? 23
how many letters are there in the following string: "spread high"? 10
how many letters are there in the following string: "chat shoot occur population lab praise sneaker nursery chairman courtyard satisfaction violent many plane snow underground association school vertical rescue chicken comfort oval stocking feast symphony primitive private board media weep web several cousin red upstairs yellow ash support out weakness broken pill absolute arrange announce surprise fence feast palace electricity schoolbag existence thus spade herb punish rain skeptical toothpaste pan weak vacant protect earth evolution relax vain tissue dull attach common drawer cyclist struggle violinist their habit once youth programme countryside clock fiber"? 533

主要改进就是按“single word”分词，共3400多个单词。模型方案跟上文中的Decoder-Only模型几乎相同（仅做必要的修改以支持更大的vocab_size）。单卡训练超过12个小时，训练损失才接近于0，并且训练过程很不稳定，每训练一段时间后训练损失就会变成NaN（原因未知），需要多次人工调整参数以保证模型能够正常收敛。测试结果如下：

how many letters are there in the following string: "tortoise equip root gift due like grain ad breath reality term thunder brown that bottle cozy bakery overcome permit blow guide equality split south cousin ambiguous tear crime religious heel alone signature god law north go jet leak wise ruin specific compare skateboard unite cry granny ink meanwhile measure precious sometimes famous modest giraffe unlike rate meat remove level handy flashlight sideways ambition bag admire trend truth audience expert wind translator prove sugar meat dash local reality deal grocery film mat"?  450, 449(expected)
how many letters are there in the following string: "adore skip unconditional somehow by nose weep boot allowance embarrass merry heel govern agency attract into majority rice noble bare lorry labour service observe meat cheek growth inch himself tyre full speaker prefer finance fill table ash till line mad asleep freezing net sweat knock father fault sort theft navy rock franc feast accurate spaceship soup novelist hole gymnasium attend dot crop chorus claw daddy sightseeing apron representative beside normal nowadays boy sort envy stainless window page gain abstract classify pay seal punctuation cast rob way ownership fantasy constant tomorrow bomb clarify arise that rot"?  533, 534(expected)
how many letters are there in the following string: "vain invite bargain hopeless dirty wrong meet grocery pregnant judgement memory wrestle labour job exit dull cheers ambition vinegar pc neck popular urge surround rose why brake brother hear voyage joke hot permit remain barber file atom long guest origin until adaptation embarrass concern ourselves much learn juice carrier settle homework day notice during pest marry post carbon aim campaign diary early lively golf rude appear pair roll packet cheat dilemma near smart postage link tiger large identify angry brother calm different maths difficulty apology without student team cubic sail those castle give"?  518, 519(expected)
how many letters are there in the following string: "rough assume friendship limit labour stable acid lazy bye bishop warmth snow outing squirrel graph wage gift dvd access adjustment hatch atlantic coffee upstairs brick group university there present hurt brush commercial herb tail alongside base along actual bed yard dirt clumsy him security luck quiz rat swift quilt home recipe rank cozy steep chef safe security rent collision mile television hate expect beddings try long actual glory professor grocery royal justice requirement goal session decide arm swap stupid their mental skyscraper across arrow sock effect depth guide barbecue transform salty time coast lonely noon"?  533, 534(expected)
how many letters are there in the following string: "various westwards stick organisation drink explicit automatic tradition academy spray vcd mop medicine experience apologize variety algebra mixture leak government area always beneficial yet entrance operator motherland outstanding cruel admission merciful flood harvest considerate acquaintance delicious pepper fancy achieve essay centimetre ham park comedy nephew extraordinary combine decade public crayon grocery its officer retell shrink else mistake carve alphabet similar peaceful cosy consequence position sacrifice earthquake surrounding pc grandma avoid baseball reputation terminal illness imagine destination boss"?  549, 550(expected)
how many letters are there in the following string: "send abnormal together area muddy repeat oh seem moral queen laughter development nose busy medicine rope fair greet scholar disc submit common silk used hero passage kilometer bar acquaintance cube football sense second zero fool manner rat diary bus corn ad boundary arrive latter antarctic dull theft attack affect pleasure clean take outing broken joy latter scream down pull as are angle biscuit smog vote sight worm scan outside vase basket guest lid water own dirty madam peaceful underline serve relax security knock aluminum may general disturb globe myself confuse nationwide rank"?  400, 499(expected)
how many letters are there in the following string: "whatever off can bomb clothes manage nearby painting duck timetable sex mailbox nursery biology cloudy junior refer ache important argue whom cruel settle that second modern club beer director aware clone bare gun copy surplus to confident measure barbershop forbid eager necessary origin degree lonely fail challenge hardly breathe evaluate soap jewellery watch petrol this talk shoot beg disaster hope condition profit digest former cheese slide wise strange front debt aids sleepy however conclusion rail flow pocket cold applaud scare hers athletic cheer minimum gun vivid album ton"?  498, 499(expected)
how many letters are there in the following string: "healthy guidance campaign educate toward absolute tutor preserve sympathy period role sculpture appoint ride cake fancy tremble each uncomfortable appoint clothing category speaker small order straw drag crayon autumn cafeteria musical direction blouse repeat irrigation fragile clock basket cloth donate regardless bored ancient glory yours finish necessary feeling badminton schoolboy atlantic explain anecdote cattle anybody hate pub electronic approve cheer dictionary apartment carve strange circuit television consideration bravery expression relation fellow afternoon discussion neighbour punishment tiny fever now toy contradictory administration merry interest manner consistent tonight table watch found motherland harm education"?  649, 648(expected)
how many letters are there in the following string: "sing unwilling campaign bonus wrong lay poet below database stream remote employ value repair compare yellow physical listen sneeze quit ski crazy rat angle cure quality decide down outwards worth ministry roast shopkeeper jeep weekday event yard more soft asleep major actual piece bent ash advise bring aunt toy expose depth network widow pineapple suitcase apartment paragraph must example relate bother waste service parking heat top roof abolish menu acre pick reservation law gift realise born wait finance so today warm fluent merry express"?  463, 464(expected)
how many letters are there in the following string: "jar place tall paper home hundred zoo doll jeep housework sorry unfit arrival wrist elect type example rid wage garbage wish nest cost bright am rely flat symptom chest arrangement cage onto birthplace export suite cheat where cheek album independence sad middle nowhere biography hand shower physician tell chef behind dear pond award metre cash skip adequate assessment admire forgive bedroom sell next wear bag keep whom anyhow whenever move dark visa botanical tear betray oilfield routine probably ourselves"?  433, 434(expected)
how many letters are there in the following string: "mass person risk more handle howl allow anecdote rude boring young capsule gay wealth show bishop dash platform accompany pipe pleasant official ecology report choir harbor percentage personal swell mouth illegal sightseeing toy concentrate aluminum either smog gun nature annoy quiet flash output aid deed radio moral dizzy refuse sad typical fog single same ink whale drive upwards silver wool person alongside warehouse pc teacher mankind harbor handwriting wet that fool import roast freeze excite island harmony windy his native carry novel letter theater meal cookie seminar virus until learn"?  508, 509(expected)
how many letters are there in the following string: "comfort glance injure concern gold daily grill visit punish fact album lie leader himself beyond biology big take friend meanwhile spread already panic complex trick tire pain reach cousin them dollar educator official familiar gather eager him bush tasty fly rent haircut skill border dare muddy behave cautious ant intend panic embassy panda product know blue cruel mutton aim stain bounce howl your date bite vivid benefit giraffe glass camera spirit twice mom"?  392, 391(expected)
how many letters are there in the following string: "bond range motto dizzy pond preparation junior exercise practise brake painting draft fly head today coat himself minus bandage candle within lorry middle pulse prefer mild fingernail key she affect actor fellow forever what are duck grand truly ham onto breath weed park money about large fierce neighbor city awesome awkward garden type export brush quality fog big awake delay prejudice active fence door agenda output blank allowance host trade ample greengrocer cheap stand shrink coat remote clock pence honey national african yours pleased beard weight yard egg bend nowhere clever traditional approximately head menu glare"?  533, 535(expected)
how many letters are there in the following string: "permit coat skip relation trade biochemistry jet acquisition software notebook drunk heel congratulation wine useful hard native evidence recite glance smile strength fuel bench save arrange enough stomach pest fast sex import meat opinion maths come poster tool all chat crime connection private curd rabbit flee buy bell navy card yellow mess quality forbid headache beat pocket wool scholar stay technique for arise considerate golf member tissue harbor found top novelist everyday guarantee dvd bit nose large list aspect gain clone choice appendix bay freeze mankind shot grasp power lemon cautious kilometer theater week leg turning purse accelerate"?  557, 558(expected)
how many letters are there in the following string: "come police tram exist aggressive branch brief marathon easter wood whale swap die boring fork ox drug asleep mountain museum tape far pavement movie chalk rewind pale incident decision automatic camel pepper nearby farm tight port time kind finish attend reasonable shout valid grow although rescue round independence sun apple system review acute clever click flag bee adjust to plot center eat fair narrow oilfield tear chain conventional pool maple decision hate absurd nephew hang spoken shade irrigation blanket bone so raincoat oil"?  455, 456(expected)
how many letters are there in the following string: "alcoholic by neither wipe jaw rule mess pipe hear likely confidential text pea opinion virtue northeast jacket comb hate opera towel poem fall cheek ink hers finish garden strength dinner pint achieve debt hope scholar cruel clone properly apron repeat maple not bed circumstance almost helmet seem humor league numb peace brilliant jar deed fortunate heavy custom coat disturb fast primary barber stay attention fibre dot europe tourist sponsor glance replace helmet leaf distinction jewellery lawyer last he starvation mop tree ache voluntary yummy dinner abundant eager liquid possess god flash pen thought aboard after fragrant programme resemble thread"?  560, 559(expected)
how many letters are there in the following string: "oil chef here paper merely abandon price assist rob chat secret ignore mad theme pig routine labor shelf argue contribution early need rough sacred occur golf spell flat succeed finance web cloud crazy our goal taste rob hunt birth express energy aboard suggestion help ahead anything welfare oral"?  249, 250(expected)
how many letters are there in the following string: "curriculum salary silent tomb helicopter delighted flower librarian importance satisfy insect distance headmaster sniff engineer erupt holy millimeter appropriate wheel book direct stubborn time bungalow editor tasteless gay rough associate crew childhood soap in advance signature prescription journalist blame flame systematic mess famous next flesh aggressive carpet boycott youth technique cigarette assumption homework chemistry focus centimetre acknowledge international departure necessary goat acknowledge actual army convey razor triangle cure chemical lawyer credit stick desperate tonight bench sense judge tremble greengrocer humour dialog"?  572, 571(expected)
how many letters are there in the following string: "video boss memory favour subject detective recover still appointment seek tear dozen precise bath sow staff classmate organise millimeter cup stress tire atmosphere litre injure cubic outside bride forward distinguish uncomfortable dawn yummy center bond punctuation dance finance do database finance scar punctuation medal plot your button muddy mouth drink hate ok afford ambulance negotiate free pile hate college amusement questionnaire missile elephant butterfly fragile actress basement letters upon aside headline asia probably balloon bathtub accompany compete chocolate police novelist liter neat beyond title couple relevant slight put people consult butterfly nose able are identification acknowledge material alone"?  630, 629(expected)
how many letters are there in the following string: "schoolbag exercise worthy media graduation false neighbor energy punishment bridge curriculum by something medicine death surprise equipment altitude period level bother professor splendid waiter qualification outwards example comment forgive baby representative bowl yet distribute spare impress behind fare like partly dusty barbershop thief underline truth off doubt ready should pet pot born pie spear fundamental react honey damp today balloon clone black"?  400, 399(expected)
how many letters are there in the following string: "law crossroads pioneer want hen hard not astonish clap broad leak magic gym kill voyage nail forgetful technical which palace survive biscuit lively end grill rectangle barrier bag cosy fool crayon harvest wag bend allergic focus water important length sofa report wag buy spin forest chest pure male slide ticket approximately relative surround metal consist used selfish always plenty situation deserve real quantity please staff bell square forward bite this now chairman degree universal bonus truth possess metre compensate obtain taste ocean vote rope trade"?  480, 479(expected)
how many letters are there in the following string: "inside presentation balance book puzzle scare aggressive administration enthusiastic practice somehow taste international update hang secure tractor chance another frighten function wild kite organise tournament frontier vinegar over ankle automatic wrestle maid christian downtown order aggressive challenging prefer only division newspaper center postcard troublesome conflict westward hundred float mature pig motorcycle waitress fibre fortunate about its owner principle blanket twice reference nationwide eastern shave sceptical random"?  476, 475(expected)
how many letters are there in the following string: "connect headline with slight run beg sweet wild opinion festival fare cup here forehead injury farm brave die ever such nose fragile free creature none clinic puzzle near avenue pleased behavior dinosaur afterwards want walk heat ignore me post farm amateur remember serve shade pour interrupt free person indicate gallon split cheers oval shirt father choose trick facial town absent yard"?  330, 329(expected)
how many letters are there in the following string: "gallery light proper saying support practical ordinary across roll province reply hug fade right man past advertisement dust pack wisdom healthy extra rare vital golf shut case moon grain smile sink room straightforward buy apology duck circle disadvantage review sailor drag lay success entrance representative dismiss nurse appreciation flower envelope zip precise deliberately database anxious lesson humorous portable departure ship prohibit handkerchief barber grandparents bureaucratic tea portable painter simplify nervous symphony bishop franc reasonable style web up certificate handle fit shorts respect purchase every pillow path obey unit motorcycle"?  572, 573(expected)
how many letters are there in the following string: "witness camp gallon heat register bread suppose album permanent wire bathtub salute fiber fare duck shallow or ahead face folk nature socialist these oh smart seal nearly characteristic tail relay moral wag worried ice foot dull thirst hair trip late grow explicit rope soon cheese pot ripe trend storage tutor type meal rich fill therefore those glare tease glory tap least attack prepare silence disappointed continent"?  354, 355(expected)
how many letters are there in the following string: "supper dynamic amateur own catholic acquaintance overcome platform bend misunderstand deliberately regulation description drop condemn suite paddle garage pink theatre garlic accommodation teamwork certain seed shaver poster library conventional consequence astronomer careless grateful porridge shopkeeper anchor obtain language science olympics pole machine immigration copy distant medal leak flashlight suffering quake sparrow final tomorrow local officer likely fresh handsome background telephone tradition pioneer direct media rainbow natural export grain heat arrange active percentage allow deaf clay changeable corporation sweater basket probably traffic ancestor"?  591, 592(expected)
how many letters are there in the following string: "together skip introduction silly desk expression adequate generous foggy contradict almost division sightseeing studio arrive seaside nationality accident vest arm substitute rail terrible thread damp spot transform melon assessment arrest arithmetic possession reasonable department equal villager poison center lonely collision candy hostess downstairs flow surplus exhibition track discover capsule extraordinary occur yellow army grey comprehension mourn humorous revision post balance comfort paragraph bother favourite citizen deliver umbrella pound several sit recognize chairman ant nowhere pace chest accomplish explanation administration"?  570, 569(expected)
how many letters are there in the following string: "every policeman thunderstorm how"?  30, 29(expected)
how many letters are there in the following string: "discuss steal pack absolute world electronic pattern loaf positive catch sceptical acquaintance noble smell labour video electricity gray annual institution separation shy around experience plenty hair ticket performance ash grey volleyball sidewalk must suspension laundry evening altogether beef class disturbing shoe broken teach sightseeing champion nuclear saying capsule until enter disc income seaweed finish board rest commercial sharpener settler bottom regardless percentage smooth southwest jacket broken buddhism refrigerator collision office notebook influence instrument spoonful entertainment entrance section positive triangle box fool escape cruel reliable"?  589, 590(expected)
how many letters are there in the following string: "avenue howl grandparents receive should menu desperate gram dozen nephew over leg middle assess awesome say respond target storm ahead somewhere useless invite chest least kick play cut near porridge themselves except shave sport ordinary pack part combine basin education strong ought rely essay loose actual disgusting luck majority captain sit analysis hate hard violence pig ship loss near check noisy mutton educate box millimeter catch admirable ash rhyme overcoat become terror meet clothing maybe link reach dull apparent scar hunt book heaven special helpful bless burden hill dash expert moustache vivid look bond win belt madam"?  541, 542(expected)
how many letters are there in the following string: "mum permission it cautious breathe tv weakness chef variety embarrass necessary possible dog comment mother shopkeeper instrument sunlight are ant preview section welfare within town scarf departure dot mustard factory constitution usual name fridge cheap straightforward throughout arbitrary recent taxpayer middle bacteria master curd to invention aircraft radiation ant well motivation merciful disappointed assistant cause harbor holiday through neat musician culture honor southern caption stainless comfortable thermos church arrival burglar strict combine summary outer pronounce funeral worth air electricity actress register enjoy independence ambassadress assume antique alcoholic bunch generation pick manager cinema ash letters representative"?  659, 660(expected)
how many letters are there in the following string: "merely mushroom bingo role sharpener dip break lunch electronic super fill abolish trap reception insist inspect kick string boom suitable bright net hostess host lid carry also hat zone pipe organization nor hire dawn defeat degree army ever king appear live monitor museum entry opening let zero passive talk enterprise traveller being wing tortoise butcher apply agree purple recorder crew trade abundant wheat cost along instead coin spirit adolescence let gallery rat straw liter delete ray farm thank duty my come meal convince sour green play riddle cottage trust ceremony teacher corporation satellite shower tendency soldier resign"?  543, 544(expected)
how many letters are there in the following string: "tournament fill worn agency highway presentation conclude concern chaos dictionary example poster unfortunate sugar pocket boom restriction need hurt discrimination nothing universe salary essay offshore host grandparents violence cupboard disturbing till illness from squirrel withdraw remember servant paperwork toast colleague goods twist protection problem tractor beauty everyone adjustment poem stable vacant liberation acquisition present unite sister precious favourite hesitate dimension sleep subject win brick adaptation gymnasium proper joy circulate jump population hammer insurance accommodation river nationality swell train sweat theoretical"?  580, 578(expected)
how many letters are there in the following string: "saleswoman carrier sailor complex cash bakery load how in calm our dig voluntary authority advantage job spoken inn glue wolf pepper oneself pound flexible admission remind probably sneeze patient ham worm daughter look bathtub back recorder dream predict visa blouse join tube sword impress dynasty from melon hurt succeed strait however quarter bingo twist annual error forever finish broadcast achieve silent there who extra project silk gentleman pity budget sailor"?  399, 400(expected)
how many letters are there in the following string: "another urgent match entrance darkness niece excellent everywhere kangaroo stare rainfall occupy trouble circulate zip soldier distance league humorous exhibition desert instruction algebra master administration airmail biology production apple donate amount possess newspaper airspace examine accept single enterprise prescription literary bottom departure melon necessary ground tend signature time chopsticks sand part prevent and premier spokeswoman virus point expensive disagreement convenient standard videophone dad centre motto across song either celebration stream wealth rid sympathy cheerful hard embassy chemical loud aboard example navy liberation hurt allowance trend me postcode paragraph significance busy middle program give branch"?  657, 656(expected)
how many letters are there in the following string: "man role pop farmer grammar pig fierce pine those diamond rubber mind hi small feeling translate dentist disease pool cost into process sailor bus work hungry shake polite sky ground chairman shot deliver dictation aids jungle basket ferry sugar brewery practical personnel this one compass stadium tooth fade pencil whose pan airmail asia musical anything caption crop aunt guard abstract space greedy thrill event natural successful accuse month also sceptical theft ship zebra exam expand upward deaf typewriter obvious squeeze neighborhood itself"?  470, 469(expected)
how many letters are there in the following string: "marathon fireworks medical challenge string raise digest arrow decide steak everybody beneath terminal bungalow relief physical invent heaven crayon demand pleased way view trouble government computer jump soccer knee elegant energy hundred classify govern while intention envy amount wheel unconditional lunch signature turkey production practice impression pocket fantastic proper existence enquiry lie accustomed accuracy salesman recite arithmetic shade everyone beyond tale poison approximately oneself analyse biochemistry sorrow short fundamental neighbourhood outspoken regulation twist bare believe barbershop suitcase policeman know find forward around statue pence ordinary realise soup tomorrow west feed everything farm whom mat bond"?  653, 652(expected)
how many letters are there in the following string: "rely pea channel say wrinkle buy other consult stage stone music fat middle scene attach scan shirt storage wash acquire air widespread crossing plenty snake modest internet marble taste classic acre north future much of acid analyse wealth polite reply abnormal cell future atom patience candle exit indeed nervous male pie delay achieve dizzy human behaviour your farmer hungry ham means pity joy private club jewelry member pineapple praise recent riddle food cake policeman weekday habit effort stone emperor destination late native she luggage skin cheerful sir bare accent grocery doll belong nut resist more parallel benefit conclude"?  542, 543(expected)
how many letters are there in the following string: "fool office citizen cube situation recreation trial favor volcano read song beat violent typist distribute connect whatever season bridge lawyer pear roof star reply bottle station telephone crew brown biology appreciate angry fruit poem mercy month steel old centre affection head shower left via qualification literary cost bicycle bottle report hesitate east nearly umbrella urge leader union dam leak freeze decide sweet unfair cast stair single up cup globe spoon casual event sleepy silk land dish or switch unite huge arch cover"?  453, 454(expected)
how many letters are there in the following string: "many method achieve diet ever doll intention shower gesture bargain murder wait sell character operation royal stove bill balance recommend satisfy ambassador scholarship thick description shape darkness salesman illness same acute teapot birth academy southeast rag urban hero ambassador pass beauty relative videophone studio christian conduct hurricane respond feed weed urge appendix consultant afternoon changeable translation vase spaceship dust granddaughter memory suffer westwards acquisition congratulation tobacco grasp washroom itself mummy mud agency half gentle edition"?  510, 509(expected)
how many letters are there in the following string: "development modest suffering slip fingernail eager immigration respect difference association conclusion comb reply regret conference spirit recover arrangement weak oh together ray candidate maybe painting disease rare thus feast silk material battery introduce spray battery behaviour wildlife transport information cap rate divorce cabbage soldier foot flexible performance summary glass entertainment tip immediately grey characteristic policy happy decision attractive declare construct pepper sometimes allocate natural cigar anxious sponsor noisy admire gather backward teenager fortnight appreciate stadium cheque nowhere gather empty advise silly insure conscience strict buy salad shut clerk outspoken pure responsibility room protect relate"?  659, 658(expected)
how many letters are there in the following string: "glare motivation allow rainy dialog child same class hall difficult curious pianist forget grandma prefer abrupt sincerely systematic repair worth sunshine flame centre lid ability worm education lung sex straightforward assistant spoken preview trick tense advocate responsibility preparation tyre significance geography behavior mention see actual identification several sniff market evaluate enthusiastic story pill full loss because cattle grandparents famous dollar mercy plant ambition skill maid textbook phrase discuss earth biscuit powder we laboratory freeway profit cd signature simple winter jungle thrill swift extreme string captain attack bury salt could avoid hers luck junior shall after"?  609, 610(expected)
how many letters are there in the following string: "assumption hope animal an pretty empty beg unlike seal videophone outwards its calculate valley weak relay surplus otherwise put throw magic sense game dioxide slow hook policy standard jog edition reception rag opening needle sex skilful smile zero deliver harm any uncomfortable overhead mask pie neither balcony bush double scar sideroad bat clone mean note scan surrounding granny talent pleasure observe combine goods remove shelter much symptom hour condemn king he muddy listen wolf honour tent young word dorm hammer bay abrupt wish minute grateful permission quite note lemon gun shoot admit skirt pear access relationship side least deliberately"?  556, 557(expected)
how many letters are there in the following string: "thunderstorm steep population cinema trip contribution show zip application survive surround steal tension approximately conservative relation mummy village astronomy teacher frontier beer neighbour express afraid millimeter performer wolf withdraw waste forbid devote concrete ear pray fall camp greedy pacific adult wander criterion sunny jam violin ski buy anxiety tip pity unbearable noon news scratch downstairs rude windy coach enjoyable distinction friend supper rainbow typical glue form month vegetable grandchild exhibition unable dumpling flexible active tongue cup gesture remain rank whatever authority further refresh in dorm wine southwest hurt accommodation bureaucratic pill ruler common energetic oval view league place disability"?  649, 650(expected)
how many letters are there in the following string: "sight plot invite choice liter communicate large automatic medical performance introduction abrupt wall starvation gentleman shelf indeed sport operate product olympics profession contain pleasure guilty temperature worker position cigarette homework abrupt conductor search fun container substitute unfortunate rooster approve punish turkey fault passenger coal fist deadline goods fantasy face exercise villager pioneer brush weak tomb existence medicine swear drop replace strike subject practical observe separation above collar foster secretary exhibition grain crop officer remove telephone navy backwards troublesome shortly weep catastrophe gymnasium western brother recover gesture"?  606, 605(expected)
how many letters are there in the following string: "frontier prevent net spread balloon someone accuse final clear directory sentence overhead idea preserve throat pineapple endless little short origin servant african blanket lake suggest millimeter librarian weekend guarantee piano note pence recipe cushion dismiss personal result himself discovery psychology host sharp illness unconscious advantage eraser quiz kite attitude atmosphere winner innocent offer lecture announce rule she autonomous approval justice dismiss pleasant may rice exhibition hang porridge officer award pilot bride butcher chocolate dorm hall escape catastrophe medical destination girl resemble content trouble metre connection applicant midnight hear souvenir systematic determine hug click alcoholic grain idiom which"?  652, 651(expected)
how many letters are there in the following string: "unbelievable empty appeal culture pedestrian slave overcome always splendid possession bag lonely condition disgusting real experience chairman struggle young style pineapple fragrant voice bored sculpture construction alarm pineapple greedy affection thankful captain turn mainland wound afford blind eager kitchen transport effect shut affair religious radium happiness greet detective physicist ignore welcome heart mankind plan false practice bravery title anybody contribute whistle marriage permanent substitute safely material skateboard control relationship washroom steak practice stable treasure different mind jungle suck failure wherever forehead"?  579, 578(expected)
how many letters are there in the following string: "poison theme electrical skate atmosphere reference invent am peach incident guest organize noble drill tool steep rescue bush male value pillow rough book angle intention office twice ahead construction damp spade flame weather towel text provide victory alley in motorcycle pond shelf block oval monkey wander deer notice sun fibre tool yours lung breathless nursery fog contradict stove money fork unbearable gift juice finance busy brown outwards noisy handle upon great sadness remove spy mineral criteria rain nice just cassette assumption valuable gift programme puzzle hotel nut"?  500, 499(expected)
how many letters are there in the following string: "steam eraser postage affair pavement principle bottle stove visit mountain sacrifice live boss lock outing gate law specialist insure accumulate skin forehead questionnaire newspaper incident medal hurricane conscience conservative entrance neighbour percent store graduation judgment hungry postcard avoid northeast greedy whatever earth upset yesterday countryside boycott ambassador quality weakness weekend truth fox rewind population yourself winter rescue scientific pile hot peach metal family autonomous personal framework terminal withdraw parallel there mass contradictory straightforward dimension attach bee banana receive apologize drink glad below president position cotton indicate succeed whose function conscience cheek create pity record widow lamp now"?  675, 674(expected)
how many letters are there in the following string: "amount course pardon corn height afford wing volcano voluntary thread steward mention equipment keyboard sideroad mail electronic stream hit ride triangle uncertain place guitar library before super weekday farm illegal public elephant persuade representative register project often agent gallery approve robot south jeep stewardess moustache hearing defence make buddhism trunk insure teenager appropriate experiment helicopter selfish beneficial clone accident voyage conservative outward traffic pacific decide disappear carrier franc faith home excellent decoration equal inside liquid ready carpenter award strike satellite granddaughter bite starve equipment artificial skyscraper"?  602, 601(expected)
how many letters are there in the following string: "restaurant patience supermarket pyramid eggplant advance sparrow atmosphere share postpone alcohol sponsor rabbit middle dial command dirty entrance suppose lab just flower cabbage parallel dimension downtown history guess fireworks advance kick represent along trunk tower merciful initial market astronomer sightseeing beside object absence your owe translation thin disappear reception properly unlike canteen knowledge northern mailbox head vocabulary circuit elephant enjoy ordinary toothache woollen ordinary marble appreciation principle jungle door will scarf guidance fence prescription ripen tourism toothpaste society survival round retell can appropriate affection write sneeze relay"?  610, 609(expected)
how many letters are there in the following string: "cushion overcome act year goods accelerate be disagree addicted grasp hear representative until reputation distance postman alcohol athlete purse aloud ad labor zebra shut up admission girl routine trick dash brand salty skilful breast broken socket database counter sleep heavy ray well troop rush blow settle suck immediately shine biscuit last lid loud beast girl yours sunny ban long lazy stupid production grocer develop strike rich woman destroy absurd tale skate star near stream pot olympic examine freeze limit disturb grey crop write systematic group string facial visual contain difficult fortnight classroom impress we create spring obtain risk strawberry"?  568, 569(expected)
how many letters are there in the following string: "pale altogether iron reward bid equip mummy stress terror charge menu bike slow used toothpaste office circulate shark theft citizen ton die carry fan police am union divorce impossible heel twice music flag thrill warn arch towel kite nor to pair forget wide being unbelievable tape soup purpose sale notice arise tiny purple ridiculous nationality kite addition customs eagle roast grandparents fat listen lung strawberry instrument up major quiet delighted tomato educate feed yours worldwide heel disturbing impression sharp origin quiz awful explore medal thread mobile pretend airplane pain golf elect he"?  518, 519(expected)
how many letters are there in the following string: "shape fantasy uncomfortable typist share we blouse table asleep biscuit review restriction match graduation ache subject daughter movement main contain shower backward carriage ancient instruct comfortable selfish adequate bay elephant motorcycle wing conversation international anything offence immediately furnished future neighbour pulse merciful whale abrupt tasteless husband invent musical absolute apparent instruction drag purple patient opposite approximately humorous chant reject invention bar grateful achievement unless appointment lack one evolution frighten even centigrade easy experiment family telephone alphabet sudden jazz salad basin bother everywhere him"?  595, 594(expected)
how many letters are there in the following string: "fist consultant catholic rule reservation separation soon awful authentic physicist choir weekly farm squirrel businesswoman cater discrimination enough traveller procedure college fill dinosaur relay absorb westwards airport scenery visitor upset affect mat slow temperature available worthwhile schedule minority neither doubt tortoise percentage foresee publish article sneaker interesting ink salary lawyer pound digest home useless federal flag student local conductor reference swim undertake plane educate chance energetic warmth accumulate sometimes swim left pan own arithmetic therefore european adult mistake stay actor businessman as together button valuable skilful disaster ceremony occur confidential civil universal"?  639, 640(expected)
how many letters are there in the following string: "positive shock across incident heart jet album bacon major holiday predict bottle defence in audience super admire cage excuse cast twin chairwoman hide way wolf voyage voice humor beauty artist can quake double tooth click card treat railway fiction row actress abroad moustache office platform dash museum away honor cooker spokesman socket ham experiment slim fog text kiss cheat"?  323, 324(expected)
how many letters are there in the following string: "carriage acknowledge agricultural those battery following finance superb grandfather troop hen erupt olympics sex television favour heavy quality contribute worth total strait student snake perform academic marry promote protection around ceremony fail quiet federal educator university possession parent union and garment practice desperate butcher generous skeptical cigarette lazy river especially grandpa combine identification saying regulation bureaucratic kid enthusiastic southwest advertisement yellow metre oceania depend blow seize rebuild extraordinary weekly snake practical technique citizen education woman silver goods wind explicit compete novelist youth agency profit beach porter relative consequence jewelry memorial average guitar"?  659, 660(expected)
how many letters are there in the following string: "year able rigid bare seem agricultural labor pill eager solar how dance"?  59, 60(expected)
how many letters are there in the following string: "throughout actress department independent fibre cushion gymnastics fragile whistle convince sunshine season ball earth tire easy essay commercial rich apology cash murder preference cigarette appeal worker alcoholic exhibition embassy pray forbid settler swear corporation horrible carve nursery your wonderful consequence africa innocent translator geometry success bowl practical translate particular judgement sharp villager teacher moment careless specialist toilet notice stranger cause digital tram all enjoyable brief premier progress bunch cigar consume kilo education sea fragrant weep obvious scare challenging headache awake copy brilliant altogether passive wait quality any hydrogen tolerate thought different classroom president healthy while embassy"?  670, 669(expected)
how many letters are there in the following string: "want mourn flee cow visitor popcorn adult clone upset distant quake deed fancy fan comfortable standard group diploma tough service ride sneeze delicious couple explode home shortly compensate cat policy herb slow dog motto tank account possibility hurry consult brain set glare steady violinist forget although action bathtub outing race reality deed liquid report but frighten drink pancake look zero translator artist you roll kill at offer company extra appear lion calm rhyme draft within recipe exhibition hand note pupil truth avoid statistics wolf grow curtain care ash fat rag concentrate him suffer"?  515, 516(expected)
how many letters are there in the following string: "scold outing celebrate museum nutrition escape justice prize speech pillow little oxygen block website congratulation judgement bonus ridiculous hammer discrimination fill member evolution weed along hobby microscope hardly interval particular available exchange classroom answer insect hunger attitude tease absolute graduate dictation lion adventure term buy afraid insist interest receipt recognize premier respond able republic construct sniff automatic aircraft principle tour review read acknowledge ill much sorrow container atom determine soil example respond strengthen defence pity bureaucratic suit clinic function discuss merciful annual single pace painful application waiter finance geometry representative hers sorrow hair shake"?  649, 650(expected)
how many letters are there in the following string: "fierce breath summer zipper honour melon greet eggplant look chef ice mental style belt prize cousin seldom terror people defence load start fond persuade memory reputation touch mean abnormal abuse coke kindness madam kite umbrella convince alone razor hesitate bay edge earth upper electronic bone follow pretend seaside postpone everyone nearby dvd abrupt pattern hour thing gram prejudice wife smoke gay amateur respond camp dynasty lot southern apple hers tower me ending mild hometown tennis container potential harm percentage salesman patience asian drum act environment mum panic tank west pin spirit reasonable mind shelter"?  539, 540(expected)
how many letters are there in the following string: "sunny secretary scan defend administration prisoner generation outdoors underline shrink principle expect brewery counter representative hospital crazy survival explain insurance police headline bad coincidence granddaughter inch symphony professor carbon fish translate technology danger sunburnt spend awesome tortoise grow mirror succeed fancy subjective trap diamond language catastrophe colour litre adjust address generous hide document emergency seaweed starve slavery fit fat assistance heart performer aim kindness unbelievable expose mustard simple across skyscraper loss evident injure shake ancestor bureaucratic perform librarian burden sight spokesman circuit climb universal sharpener dedicate conflict skate today"?  639, 641(expected)
63/100000  99.93700%

随机生成100000条样本，错了63个，测试准确率约为99.93700%。即使预测错误，跟正确答案也是非常接近的。

总结：对于100个单词以内的统计字母个数问题，按“single word”分词，使用足够多的样本，训练足够多的轮数，使用Decoder-Only模型也是可以训练到99.9%以上测试准确率的。但现实情况是，大模型的训练成本太高，一般也就训练一轮，导致在数数问题上准确率偏低。

原文链接 请勿转载