零基础入门NLP之新闻文本分类2

import pandas as pd
train_df = pd.read_csv('1/train_set.csv', sep='\t', nrows=100)
train_df.head()
labeltext
022967 6758 339 2021 1854 3731 4109 3792 4149 15...
1114464 486 6352 5619 2465 4802 1452 3137 5778 54...
237346 4068 5074 3747 5681 6093 1777 2226 7354 6...
327159 948 4866 2109 5520 2490 211 3956 5520 549...
433646 3055 3055 2490 4659 6065 3370 5814 2465 5...
%pylab inline
train_df['text_len'] = train_df['text'].apply(lambda x: len(x.split(' ')))
print(train_df['text_len'].describe())
Populating the interactive namespace from numpy and matplotlib
count     100.000000
mean      872.320000
std       923.138191
min        64.000000
25%       359.500000
50%       598.000000
75%      1058.000000
max      7125.000000
Name: text_len, dtype: float64
_ = plt.hist(train_df['text_len'], bins=200)
plt.xlabel('Text char count')
plt.title("Histogram of char count")
Text(0.5,1,'Histogram of char count')

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-laGZGhA6-1595436439595)(output_3_1.png)]

train_df['label'].value_counts().plot(kind='bar')
plt.title('News class count')
plt.xlabel("category")
Text(0.5,0,'category')

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WB8anlV2-1595436439597)(output_4_1.png)]

from collections import Counter
all_lines = ' '.join(list(train_df['text']))
word_count = Counter(all_lines.split(" "))
word_count = sorted(word_count.items(), key=lambda d:d[1], reverse = True)

print(len(word_count))

print(word_count[0])

print(word_count[-1])
2405
('3750', 3702)
('5034', 1)

假设字符3750,字符900和字符648是句子的标点符号,请分析赛题每篇新闻平均由多少个句子构成?

train_df['number'] = train_df['text'].apply(lambda x: x.count('3750')+x.count('900')+x.count('648')+1)
train_df.head()
labeltexttext_lennumber
022967 6758 339 2021 1854 3731 4109 3792 4149 15...1057121
1114464 486 6352 5619 2465 4802 1452 3137 5778 54...48651
237346 4068 5074 3747 5681 6093 1777 2226 7354 6...76456
327159 948 4866 2109 5520 2490 211 3956 5520 549...1570158
433646 3055 3055 2490 4659 6065 3370 5814 2465 5...30729

统计每类新闻中出现次数对多的字符

l = []
for i in range(14):
    a = train_df.loc[train_df['label']==i,'text']
    all_line = ' '.join(list(a))
    all_line = all_lines.split(" ")
    while '3750' in all_line:
         all_line.remove('3750')
    while '900' in all_line:
         all_line.remove('900')
    while '648' in all_line:
         all_line.remove('648')
    n = Counter(all_line).most_common(1)[0][0]
    l.append(n)
l
['3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370',
 '3370']
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值