import pandas as pd
train_df = pd.read_csv('1/train_set.csv', sep='\t', nrows=100)
train_df.head()
label | text | |
---|---|---|
0 | 2 | 2967 6758 339 2021 1854 3731 4109 3792 4149 15... |
1 | 11 | 4464 486 6352 5619 2465 4802 1452 3137 5778 54... |
2 | 3 | 7346 4068 5074 3747 5681 6093 1777 2226 7354 6... |
3 | 2 | 7159 948 4866 2109 5520 2490 211 3956 5520 549... |
4 | 3 | 3646 3055 3055 2490 4659 6065 3370 5814 2465 5... |
%pylab inline
train_df['text_len'] = train_df['text'].apply(lambda x: len(x.split(' ')))
print(train_df['text_len'].describe())
Populating the interactive namespace from numpy and matplotlib
count 100.000000
mean 872.320000
std 923.138191
min 64.000000
25% 359.500000
50% 598.000000
75% 1058.000000
max 7125.000000
Name: text_len, dtype: float64
_ = plt.hist(train_df['text_len'], bins=200)
plt.xlabel('Text char count')
plt.title("Histogram of char count")
Text(0.5,1,'Histogram of char count')
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-laGZGhA6-1595436439595)(output_3_1.png)]
train_df['label'].value_counts().plot(kind='bar')
plt.title('News class count')
plt.xlabel("category")
Text(0.5,0,'category')
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WB8anlV2-1595436439597)(output_4_1.png)]
from collections import Counter
all_lines = ' '.join(list(train_df['text']))
word_count = Counter(all_lines.split(" "))
word_count = sorted(word_count.items(), key=lambda d:d[1], reverse = True)
print(len(word_count))
print(word_count[0])
print(word_count[-1])
2405
('3750', 3702)
('5034', 1)
假设字符3750,字符900和字符648是句子的标点符号,请分析赛题每篇新闻平均由多少个句子构成?
train_df['number'] = train_df['text'].apply(lambda x: x.count('3750')+x.count('900')+x.count('648')+1)
train_df.head()
label | text | text_len | number | |
---|---|---|---|---|
0 | 2 | 2967 6758 339 2021 1854 3731 4109 3792 4149 15... | 1057 | 121 |
1 | 11 | 4464 486 6352 5619 2465 4802 1452 3137 5778 54... | 486 | 51 |
2 | 3 | 7346 4068 5074 3747 5681 6093 1777 2226 7354 6... | 764 | 56 |
3 | 2 | 7159 948 4866 2109 5520 2490 211 3956 5520 549... | 1570 | 158 |
4 | 3 | 3646 3055 3055 2490 4659 6065 3370 5814 2465 5... | 307 | 29 |
统计每类新闻中出现次数对多的字符
l = []
for i in range(14):
a = train_df.loc[train_df['label']==i,'text']
all_line = ' '.join(list(a))
all_line = all_lines.split(" ")
while '3750' in all_line:
all_line.remove('3750')
while '900' in all_line:
all_line.remove('900')
while '648' in all_line:
all_line.remove('648')
n = Counter(all_line).most_common(1)[0][0]
l.append(n)
l
['3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370',
'3370']