2007年的一个星期,两位朋友(迪恩和比尔)独立告诉我,他们对谷歌的拼写纠正感到惊讶。输入类似[speling]的搜索,Google会立即显示结果: spelling。我认为Dean和Bill是高度成熟的工程师和数学家,他们对这个过程的运作方式有很好的直觉。但他们没有,并且想到它,为什么他们应该知道迄今为止他们的专长?
我认为他们和其他人可以从解释中受益。工业强度的纠正器的全部细节非常复杂(你可以在这里或这里阅读一些关于它的内容)。但我认为,在横贯大陆的飞机旅行过程中,我可以编写和解释一个玩具拼写校正器,在大约半页代码中以每秒至少10个字的处理速度达到80%或90%的准确度。
这里是(或参见spell.py):
import re
from collections import Counter
def words(text): return re.findall(r'\w+', text.lower())
WORDS = Counter(words(open('big.txt').read()))
def P(word, N=sum(WORDS.values())):
"Probability of `word`."
return WORDS[word] / N
def correction(word):
"Most probable spelling correction for word."
return max(candidates(word), key=P)
def candidates(word):
"Generate possible spelling corrections for word."
return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])
def known(words):
"The subset of `words` that appear in the dictionary of WORDS."
return set(w for w in words if w in WORDS)
def edits1(word):
"All edits that are one edit away from `word`."
letters = 'abcdefghijklmnopqrstuvwxyz'
splits = [(word[:i], word[i:]) for i in range(len(word) + 1)]
deletes = [L + R[1:] for L, R in splits if R]
transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
replaces = [L + c + R[1:] for L, R in splits if R for c in letters]
inserts = [L + c + R for L, R in splits for c in letters]
return set(deletes + transposes + replaces + inserts)
def edits2(word):
"All edits that are two edits away from `word`."
return (e2 for e1 in edits1(word) for e2 in edits1(e1))
注:edits1() 函数写的太简洁了。后边还有很多进一步分析,我不想翻译了。