python使用jieba实现tf-idf

最新推荐文章于 2024-06-13 18:38:02 发布

kiki.try

最新推荐文章于 2024-06-13 18:38:02 发布

阅读量1.8k

点赞数

分类专栏： python 文章标签： jieba tf-idf

本文链接：https://blog.csdn.net/m0_38071676/article/details/90261797

版权

分词实现：

import pandas as pd
import jieba
import jieba.analyse
# 数据源
df_news = pd.read_table('C:/Users/Shirley/Desktop/python/article.txt',names=['id','content'],encoding='utf-8')
# 存在缺失值，drop掉
df_news = df_news.dropna()
content = df_news.content.values.tolist()
content_S = []
for line in content:
 	current_segment = jieba.lcut(line)
  	if len(current_segment)>1 and current_segment != '\r\n':
  		content_S.append(current_segment)
df_content = pd.DataFrame({'content_S':content_S})
print(df_content.head())

加入停用词：

stopwords= pd.read_csv("C:/Users/Shirley/Desktop/python/stopwords_3.txt",index_col=False,sep='\t',quoting=3,names=['stopwords'],encoding='utf-8')
stopwords.head()
def drop_stopwords(contents,stopwords):
	contents_clean = []
 	all_words = []
 	for line in conten