目录
实验目的:
加深对汉语文本信息处理基础理论及方法的认识和了解,锻炼和提高分析问题、解决问题的能力。通过对具体项目的任务分析、数据准备、算法设计和编码实现以及测试评价几个环节的练习,基本掌握实现一个自然语言处理系统的基本过程。
实验要求:
1.抽取词表
2.统计总词数、不同词的个数并输出
3.统计每个词出现的次数,并按照词频从小到大排序
4.去除语料中的分词和词性标记,形成未加工的语料(原始文本)
参考代码:
ll=[]
word_count=dict()
with open("199801.txt") as f1:
f1.seek(0)
str0 = f1.read()
list1 = str0.split(" ")
with open("词表.txt",'w+',encoding='utf-8') as f2:
for i in range(len(list1)):
str1 =list1[i].strip('/abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
'[]《》,。;:、?!19980131-04-013-0045672“”()『』①②③④⑤⑥').strip().replace('\n','')#获得词
if str1 != '':
ll.append(str1)
set1=set(ll)
for word in set1:
f2.write(word+'\n')
sum0 = 0
sum1 = 0
ll=[]
word_count=dict()
with open("199801.txt") as f1:
f1.seek(0)
str0 = f1.read()
list1 = str0.split(" ")
for i in range(len(list1)):
str1 =list1[i].strip('/abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
'[]《》,。;:、?!19980131-04-013-0045672“”()『』①②③④⑤⑥').strip().replace('\n','')#获得词
if str1 !='':
ll.append(str1)
for i in ll:
word_count.setdefault(i,0)
word_count[i]=word_count[i]+1
l0=list(word_count.items())
l0.sort(key=lambda x:x[1])
sum1=len(l0)
sum0=len(ll)
with open("统计.txt",'w+',encoding='utf-8')as f2:
f2.write('总词数:'+str(sum0)+'\n不同词的个数:'+str(sum1))
ll=[]
word_count=dict()
with open("199801.txt") as f1:
f1.seek(0)
str0 = f1.read()
list1 = str0.split(" ")
for i in range(len(list1)):
str1 = list1[i].strip('/abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
'[]《》,。;:、?!19980131-04-013-0045672“”()『』①②③④⑤⑥').strip().replace('\n', '') # 获得词
if str1 != '':
ll.append(str1)
for i in ll:
word_count.setdefault(i, 0)
word_count[i] = word_count[i] + 1
l0 = list(word_count.items())
l0.sort(key=lambda x: x[1])
# 统计不同词出现次数:
with open("词频.txt", 'w+',encoding='utf-8') as f:
for i in l0:
word, count = i
f.write(word + ':' + str(count) + '\n')
with open("199801.txt") as f1:
f1.seek(0)
str0 = f1.read()
list1 = str0.split(" ")
with open("原始文本.txt", 'w+',encoding='utf-8') as f3:
for i in range(len(list1)):
ss=list1[i].strip('/abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')#获得原始语料
f3.write(ss)
实验结果:
1.词表
2.总词数,不同词的个数
3.词频
4.原始文本(由于完整图片放上去审批不通过,所以打了马赛克)