NLP系列 1. IMDB和THUCNews数据集数据集的探索

探索IMDB数据集和THUCNews数据集

IMDB的探索

由keras直接加载数据集,再将数据集中已经预处理过的代表词的数字转换回字词
代码见 https://github.com/sherpahu/NLP_practice/blob/master/Task1/imdb.ipynb

主要参考：https://tensorflow.google.cn/tutorials/keras/basic_text_classification

import keras
imdb=keras.datasets.imdb

Using TensorFlow backend.

(train_data,train_labels),(test_data,test_labels)=imdb.load_data(num_words=10000)
#num_words=10000表示只保留训练数据集中的最常见的10000个单词，舍弃低频词。防止向量的数据过大，并且保留最有用的信息

train_data[0]

[1,
 14,
 22,
 16,
 43,
 ...
 16,
 5345,
 19,
 178,
 32]

imdb里面是已经处理好的数据，每一个数字代表一个单词，

train_labels[0]

1

len(train_data),len(train_labels),len(test_data),len(test_labels)

(25000, 25000, 25000, 25000)

import pandas as pd
series=pd.Series(train_labels)
series.value_counts()

1    12500
0    12500
dtype: int64

0代表负面评价，1代表正面
由上面的train_labels的统计可以看出，正面、负面各占一半下面尝试将数字代表的评价转化为原文

# A dictionary mapping words to an integer index
word_index=imdb.get_word_index()

# The first indices are reserved
word_index = {k:(v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNK>"] = 2  # unknown
word_index["<UNUSED>"] = 3

index2word=dict([(value,key) for (key,value) in word_index.items()])
def decode_review(text):
    return ' '.join([index2word.get(i,'?') for i in text])

decode_review(train_data[0])

"<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy's that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all"

THUCNews数据集的探索

数据集来源

完整数据集： http://thuctc.thunlp.org/#中文文本分类数据集THUCNews

子数据集： 链接: https://pan.baidu.com/s/1hugrfRu 密码: qfud

子数据集来源：https://github.com/gaussic/text-classification-cnn-rnn

数据集划分如下：

• 训练集: 5000*10
• 验证集: 500*10
• 测试集: 1000*10

从原数据集生成子集的过程请参看helper下的两个脚本。其中，copy_data.sh用于从每个分类拷贝6500个文件，cnews_group.py用于将多个文件整合到一个文件中。执行该文件后，得到三个数据文件：

• cnews.train.txt: 训练集(50000条)
• cnews.val.txt: 验证集(5000条)
• cnews.test.txt: 测试集(10000条)

数据集介绍

THUCNews是根据新浪新闻RSS订阅频道2005~2011年间的历史数据筛选过滤生成，包含74万篇新闻文档（2.19 GB），均为UTF-8纯文本格式。我们在原始新浪新闻分类体系的基础上，重新整合划分出14个候选分类类别：财经、彩票、房产、股票、家居、教育、科技、社会、时尚、时政、体育、星座、游戏、娱乐。使用THUCTC工具包在此数据集上进行评测，准确率可以达到88.6%。