python文本分词及去除停用词

对于文本分词，此处使用的是python自带的jieba包进行，首先我们要先读取我们所需要分词的文章，然后使用jieba.cut进行分词，注意分词时要将这些段落归并成同一个字符串，然后输出的是一个列表。最后写入一个文件中

import jieba.analyse
test1 =""
fencilist=[]
with open(r"testtxt",'r',encoding="UTF-8") as test:
    for line in test:
        line.strip()
        test1+=line
fencilist=jieba.cut(test1)
fencilist=list(set(fencilist))
with open(r"fenciescult",'w',encoding="UTF-8") as f:
    for i in fencilist:
        f.write(i+'\n')

在去除停用词时，我们可以将停用词进行提取，并存放在一个列表中，然后将分好的词存放在一个列表中，用for循环进行一个一个判断是否在停用词库中，如果不在，就存放在一个新的列表中，得到最终结果。

stopwordlist=[]
fencilist=[]
resultlist=[]

with open(r"stopwords",'r',encoding="UTF-8") as f:
    for i in f:
        stopwordlist.append(i)
with open(r"fenciescult",'r',encoding="UTF-8") as test:
    for line in test:
        fencilist.append(line.strip())
for i in fencilist:
    if(i not in stopwordlist):
        resultlist.append(i)
with open(r"result",'w',encoding="UTF-8") as xx:
    for x in resultlist:
        xx.write(x+'\n')