数据源:http://www.sogou.com/labs/resource/ca.php
数据源是一个搜狗新闻数据集,里面包含新闻的种类、主题、URL与内容,现在需要将新闻的内容进行清洗、去掉杂乱的字符串,并实现关键词提取、构造关键词云图,最后将整合后的内容进行分类。
首先导入数据:
import pandas as pd
import numpy as np
import jieba
import matplotlib.pyplot as plt
file_path = 'F:\\python_learing\\news_data\\val.txt'
def load_data(filename):
df_news = pd.read_table(filename,sep='\t',names=['category','theme','URL','content'])
df_news = df_news.dropna()
return df_news
df_news = load_data(file_path)
df_news.head()
接下来列用jieba对内容进行分词:
content = df_news.content.values.tolist()
def jieba_cut(dataSet):
content_S = []
for line in dataSet:
current = jieba.lcut(line)
if len(current) > 1 and current != '\r\t':
content_S.append(current)
df_content = pd.DataFrame({'content_S':content_S})
return df_content
df_content = jieba_cut(content)
df_content.head()