Part 1: 搭建一个分词工具
Part 1.1 基于枚举方法来搭建中文分词工具
此项目需要的数据:
- 综合类中文词库.xlsx: 包含了中文词,当做词典来用
- 以变量的方式提供了部分unigram概率 word_prob
举个例子: 给定词典=[我们 学习 人工 智能 人工智能 未来 是], 另外我们给定unigram概率:p(我们)=0.25, p(学习)=0.15, p(人工)=0.05, p(智能)=0.1, p(人工智能)=0.2, p(未来)=0.1, p(是)=0.15
Step 1: 对于给定字符串:”我们学习人工智能,人工智能是未来“, 找出所有可能的分割方式
- [我们,学习,人工智能,人工智能,是,未来]
- [我们,学习,人工,智能,人工智能,是,未来]
- [我们,学习,人工,智能,人工,智能,是,未来]
- [我们,学习,人工智能,人工,智能,是,未来] …
Step 2: 我们也可以计算出每一个切分之后句子的概率
- p(我们,学习,人工智能,人工智能,是,未来)= -log p(我们)-log p(学习)-log p(人工智能)-log p(人工智能)-log p(是)-log p(未来)
- p(我们,学习,人工,智能,人工智能,是,未来)=-log p(我们)-log p(学习)-log p(人工)-log p(智能)-log p(人工智能)-log p(是)-log p(未来)
- p(我们,学习,人工,智能,人工,智能,是,未来)=-log p(我们)-log p(学习)-log p(人工)-log p(智能)-log p(人工)-log p(智能)-log p(是)-log p(未来)
- p(我们,学习,人工智能,人工,智能,是,未来)=-log p(我们)-log p(学习)-log p(人工智能)-log p(人工)-log p(智能)-log(是)-log p(未来) …
Step 3: 返回第二步中概率最大的结果
程序代码:
import xlrd
import numpy as np
def create_dic(file_path):
workbook = xlrd.open_workbook(file_path)
booksheet = workbook.sheet_by_index(0)
col_values = booksheet.col_values(0)
dic_words = {}
max_len_word = 0
for word in col_values:
dic_words[word] = 0.00001
len_word = len(word)
if len_word > max_len_word:
max_len_word = len_word
print(len(dic_words))
print(max_len_word)
return dic_words, max_len_word
dic_words, max_len_word = create_dic('./data/综合类中文词库.xlsx')
word_prob = {"北京": 0.03, "的": 0.08, "天": 0.005, "气": 0.005, "天气": 0.06, "真": 0.04, "好": 0.05, "真好": 0.04, "啊": 0.01,
"真好啊": 0.02,
"今": 0.01, "今天": 0.07, "课程": 0.06, "内容": 0.06, "有": 0.05, "很": 0.03, "很有": 0.04, "意思": 0.06, "有意思": 0.005,
"课": 0.01,
"程": 0.005, "经常": 0.08, "意见": 0.08, "意": 0.01, "见": 0.005, "有意见": 0.02, "分歧": 0.04, "分": 0.02, "歧": 0.005}
for key, value in word_prob.items():
dic_words[key] = value
def word_segementation(input_str):
segments = []
if len(input_str) == 0:
return segments
max_split = min(len(input_str), max_len_word) + 1
for idx in range(1, max_split):
word = input_str[0:idx]
if word in dic_words:
segments_substr = word_segementation(input_str[idx:])
if (segments_substr == []) and (len(input_str[idx:]) == 0):
segments.append([word])
else:
for seg in segments_substr:
seg = [word] + seg
segments.append(seg)
return segments
def word_segment_naive(input_str):
segments = word_segementation(input_str)
best_segment = []
best_score = np.inf
for seg in segments:
log_prob = -1 * np.sum(np.log([dic_words[word] for word in seg]))
if log_prob < best_score:
best_segment = seg
best_score = log_prob
return best_segment
print(word_segment_naive("北京的天气真好啊"))
print(word_segment_naive("今天的课程内容很有意思"))
print(word_segment_naive("经常有意见分歧"))
Part 1.2 基于维特比算法来优化上述流程
此项目需要的数据:
- 综合类中文词库.xlsx: 包含了中文词,当做词典来用
- 以变量的方式提供了部分unigram概率word_prob
举个例子: 给定词典=[我们 学习 人工 智能 人工智能 未来 是], 另外我们给定unigram概率:p(我们)=0.25, p(学习)=0.15, p(人工)=0.05, p(智能)=0.1, p(人工智能)=0.2, p(未来)=0.1, p(是)=0.15
Step 1: 根据词典,输入的句子和 word_prob来创建带权重的有向图(Directed Graph) 参考:课程内容
有向图的每一条边是一个单词的概率(只要存在于词典里的都可以作为一个合法的单词),这些概率已经给出(存放在word_prob)。 注意:思考用什么方式来存储这种有向图比较合适? 不一定只有一种方式来存储这种结构。
Step 2: 编写维特比算法(viterebi)算法来找出其中最好的PATH, 也就是最好的句子切分
具体算法参考课程中讲过的内容
Step 3: 返回结果
跟PART 1.1的要求一致
程序代码:
import xlrd
import numpy as np
def create_dic(file_path):
workbook = xlrd.open_workbook(file_path)
booksheet = workbook.sheet_by_index(0)
col_values = booksheet.col_values(0)
dic_words = {}
max_len_word = 0
for word in col_values:
dic_words[word] = 0.00001
len_word = len(word)
if len_word > max_len_word:
max_len_word = len_word
print(len(dic_words))
print(max_len_word)
return dic_words, max_len_word
dic_words, max_len_word = create_dic('./data/综合类中文词库.xlsx')
word_prob = {"北京": 0.03, "的": 0.08, "天": 0.005, "气": 0.005, "天气": 0.06, "真": 0.04, "好": 0.05, "真好": 0.04, "啊": 0.01,
"真好啊": 0.02,
"今": 0.01, "今天": 0.07, "课程": 0.06, "内容": 0.06, "有": 0.05, "很": 0.03, "很有": 0.04, "意思": 0.06, "有意思": 0.005,
"课": 0.01,
"程": 0.005, "经常": 0.08, "意见": 0.08, "意": 0.01, "见": 0.005, "有意见": 0.02, "分歧": 0.04, "分": 0.02, "歧": 0.005}
for key, value in word_prob.items():
dic_words[key] = value
def create_graph(input_str):
N = len(input_str)
graph = {}
for idx_end in range(1, N + 1):
temp_list = []
max_split = min(idx_end, max_len_word)
for idx_start in range(idx_end - max_split, idx_end):
word = input_str[idx_start:idx_end]
if word in dic_words:
temp_list.append(idx_start)
graph[idx_end] = temp_list
return graph
def word_segment_viterbi(input_str):
graph = create_graph(input_str)
N = len(input_str)
m = [np.inf] * (N + 1)
m[0] = 0
last_index = [0] * (N + 1)
for idx_end in range(1, N + 1):
for idx_start in graph[idx_end]:
log_prob = round(-1 * np.log(dic_words[input_str[idx_start:idx_end]])) + m[idx_start]
if log_prob < m[idx_end]:
m[idx_end] = log_prob
last_index[idx_end] = idx_start
best_segment = []
i = N
while True:
best_segment.insert(0, input_str[last_index[i]:i])
i = last_index[i]
if i == 0:
break
return best_segment
print(word_segment_viterbi("北京的天气真好啊"))
print(word_segment_viterbi("今天的课程内容很有意思"))
print(word_segment_viterbi("经常有意见分歧"))