卷积神经网络在情感分析中取得了很好的成果，相比于之前浅层的机器学习方法如NB、SVM效果更好，特别实在数据集较大的情况下，并且CNN不用我们手动去提取特征，原浅层ML是需要进行文本特征提取、文本特征表示、归一化、最后进行文本分类，文本特征提取主要可以分为四步：（1）：对全部训练文档进行分词，由这些词作为向量的维数来表示文本;（2）：统计每一类文档中所有出现的词语及其频率，然后过滤，剔除停用词和单字词;（3）：统计每一类内出现词语的总词频，并取若干个频率更高的词汇作为这一类的特征词集;（4）：去除每一类别中都出现的词，合并所有类别的特征词集，形成总特征词集，最后得到的特征词集是我们用到的特征集合，再用该集合去筛选测试集中的特征。文本的特征表示是利用TF-IDF公式来计算词的权值，这也充分利用的是特征提取时提取的特征来计算特征权值大小的，归一化处理需要处理的数据，经过处理后限制在一定范围内，经过处理后，我们原来的文本信息已经抽象成一个向量化的样本集，然后将样本集和训练好的模板进行相似度计算，若属于该类别，则与其他类别的模板文件进行计算，直到分进相应的类别，这是浅层ML进行文本分类的方式；
CNN进行文本分类相对简单一些，我结合最近做的一些实验总结了一下：
在利用CNN进行文本分类的时候，首先要将原始文本进行预处理，主要还是分词、去除停用词等，然后对预处理后的文本进行向量化利用word2vec，我利用的时word2vec中的skip-gram模型，将搜狗数据集表示为了200维的词向量形式;转化为词向量后就可以将每一句话转化为一个矩阵的形式，这样就跟利用CNN处理图像分类很相似;
说一下实验，我的实验环境：
tensorflow1.2、gpu1050Ti、Ubuntu16.04、pycharm、python2.7
# encoding=utf-8 from __future__ import unicode_literals import tensorflow as tf import numpy as np class TextCNN(object): """ 使用CNN用于情感分析 整个CNN架构包括词嵌入层，卷积层，max-pooling层和softmax层 """ def __init__( self, sequence_length, num_classes,vocab_size,embedding_size, embedding_table, filter_sizes, num_filters, l2_reg_lambda=0.0): # 输入，输出，dropout的placeholder self.input_x = tf.placeholder(tf.int32, [None, sequence_length], name="input_x") self.input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y") self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob") # Keeping track of l2 regularization loss (optional) l2_loss = tf.constant(0.0) # 词嵌入层 with tf.device('/cpu:0'), tf.name_scope("embedding"): W = tf.Variable(embedding_table,name="W")
#embedding_table就是词向量表，W还有另一种简单的表达
#W=tf.Variable([vocab_size,embedding_size],-1.0,1.0),随机初始化这个词向量表；
self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
#这个tf.nn.embedding_lookup()的作用就是从词向量表中去找input_x所对应的词向量；
self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)
#由于CNN输入都是四维，所以在最后一维添加一个维度，与CNN的输入维度对照起来。
# 生成卷积层和max-pooling层
pooled_outputs = []
for i, filter_size in enumerate(filter_sizes):
with tf.name_scope("conv-maxpool-%s" % filter_size):
# Convolution Layer
filter_shape = [filter_size, embedding_size, 1, num_filters]
W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
conv = tf.nn.conv2d(
self.embedded_chars_expanded,
W,
strides=[1, 1, 1, 1],
padding="VALID",
name="conv")
# Apply nonlinearity
# h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
h=tf.nn.relu6(tf.nn.bias_add(conv,b),name="relu")
# Maxpooling over the outputs
# pooled = tf.nn.max_pool(
# h,
# ksize=[1, sequence_length - filter_size + 1, 1, 1],
# strides=[1, 1, 1, 1],
# padding='VALID',
# name="pool")
# pooled_outputs.append(pooled)
pooled = tf.nn.avg_pool(
h,
ksize=[1, sequence_length - filter_size + 1, 1, 1],
strides=[1, 1, 1, 1],
padding='VALID',
name="pool")
pooled_outputs.append(pooled)
# 将max-pooling层的各种特征整合在一起
num_filters_total = num_filters * len(filter_sizes)
self.h_pool = tf.concat(pooled_outputs,3)
self.h_pool_flat = tf.reshape(self.h_pool, [-1, num_filters_total])
# 添加全连接层，用于分类
with tf.name_scope("full-connection"):
W_fc1 = tf.Variable(tf.truncated_normal([num_filters_total,500], stddev=0.1))
b_fc1 = tf.Variable(tf.constant(0.1,shape=[500]))
self.h_fc1 = tf.nn.relu6(tf.matmul(self.h_pool_flat, W_fc1) + b_fc1)
# 添加dropout层用于缓和过拟化
with tf.name_scope("dropout"):
# self.h_drop = tf.nn.dropout(self.h_pool_flat, self.dropout_keep_prob)
self.h_drop = tf.nn.dropout(self.h_fc1, self.dropout_keep_prob)
# 产生最后的输出和预测
with tf.name_scope("output"):
# W = tf.get_variable(
# "W",
# shape=[num_filters_total, num_classes],
# initializer=tf.contrib.layers.xavier_initializer())
W = tf.get_variable(
"W",
shape=[500, num_classes],
initializer=tf.contrib.layers.xavier_initializer())
b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="b")
l2_loss += tf.nn.l2_loss(W)
l2_loss += tf.nn.l2_loss(b)
self.scores = tf.nn.xw_plus_b(self.h_drop, W, b, name="scores")
self.predictions = tf.argmax(self.scores, 1, name="predictions")
# 定义模型的损失函数
with tf.name_scope("loss"):
losses = tf.nn.softmax_cross_entropy_with_logits(logits=self.scores, labels=self.input_y)
self.loss = tf.reduce_mean(losses) + l2_reg_lambda * l2_loss
# 定义模型的准确率
with tf.name_scope("accuracy"):
correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1))
self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, "float"), name="accuracy")
以上时TextCNN的模型结构代码，然后开始进行train，并利用summary和checkpoints来记录模型和训练时的参数等等，利用十折交叉验证来产生准确率，最后利用tensorboard查看accuracy、loss、w、b等等变化图;训练py的代码：
#! /usr/bin/env python # encoding=utf-8 import tensorflow as tf import numpy as np import os import time import datetime import data_loader from cnn_graph import TextCNN from tensorflow.contrib import learn from sklearn import cross_validation import preprocessing # tf.global_variables # 伴随tensorflow的summary和checkout # ================================================== # Model Hyperparameters tf.flags.DEFINE_integer("embedding_dim", 200, "Dimensionality of character embedding (default: 128)") tf.flags.DEFINE_string("filter_sizes", "3,4,5", "Comma-separated filter sizes (default: '3,4,5')") tf.flags.DEFINE_integer("num_filters", 40, "Number of filters per filter size (default: 128)") tf.flags.DEFINE_float("dropout_keep_prob", 0.5, "Dropout keep probability (default: 0.5)") tf.flags.DEFINE_float("l2_reg_lambda", 3.0, "L2 regularizaion lambda (default: 0.0)") # Training parameters tf.flags.DEFINE_integer("batch_size", 50, "Batch Size (default: 64)") tf.flags.DEFINE_integer("num_epochs", 100, "Number of training epochs (default: 200)") tf.flags.DEFINE_integer("evaluate_every", 100, "Evaluate model on dev set after this many steps (default: 100)") tf.flags.DEFINE_integer("checkpoint_every", 100, "Save model after this many steps (default: 100)") # Misc Parameters tf.flags.DEFINE_boolean("allow_soft_placement", True, "Allow device soft device placement") tf.flags.DEFINE_boolean("log_device_placement", False, "Log placement of ops on devices") # w2v文件路径 tf.flags.DEFINE_string("w2v_path", "./w2v_model/retrain_vectors_100.bin", "w2v file") tf.flags.DEFINE_string("file_dir","./data_process/jd","train/test dataSet") FLAGS = tf.flags.FLAGS FLAGS._parse_flags() print("\nParameters:") for attr, value in sorted(FLAGS.__flags.items()): print("{}={}".format(attr.upper(), value)) print("") # Data Preparatopn # ================================================== # Load data print("Loading data...") files = ["reviews.neg","reviews.pos"] # 加载所有的未切分的数据 x_text, y_labels,neg_examples,pos_examples = data_loader.\ load_data_and_labels(data_dir=FLAGS.file_dir,files=files,splitable=False) # 获取消极数据的2/3,得到的评论的长度离散度更低 neg_accept_length = preprocessing.freq_factor(neg_examples, percentage=0.8, drawable=False) neg_accept_length = [item[0] for item in neg_accept_length] neg_examples = data_loader.load_data_by_length(neg_examples,neg_accept_length) # 获取积极数据的2/3,得到的评论的长度离散度更低 pos_accept_length = preprocessing.freq_factor(pos_examples, percentage=0.8, drawable=False) pos_accept_length = [item[0] for item in pos_accept_length] pos_examples = data_loader.load_data_by_length(pos_examples,pos_accept_length) x_text = neg_examples + pos_examples neg_labels = [[1,0] for _ in neg_examples] pos_labels = [[0,1] for _ in pos_examples] y_labels = np.concatenate([neg_labels,pos_labels], axis=0) print("Loading data finish") # Build vocabulary max_document_length = max([len(x.split(" ")) for x in x_text]) # 最长的句子的长度 print(max_document_length) vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length) x = np.array(list(vocab_processor.fit_transform(x_text))) # 加载提前训练的w2v数据集 word_vecs = data_loader.load_bin_vec(fname=FLAGS.w2v_path, vocab=list(vocab_processor.vocabulary_._mapping), ksize=FLAGS.embedding_dim) # 加载嵌入层的table W = data_loader.get_W(word_vecs=word_vecs, vocab_ids_map=vocab_processor.vocabulary_._mapping, k=FLAGS.embedding_dim,is_rand=False) # 随机化数据 np.random.seed(10) shuffle_indices = np.random.permutation(np.arange(len(y_labels))) x_shuffled = x[shuffle_indices] y_shuffled = y_labels[shuffle_indices] out_path = os.path.abspath(os.path.join(os.path.curdir, "runs","parameters")) parameters = "新全连接+jd数据+10\n" \ "embedding_dim: {},\n" \ "filter_sizes:{},\n" \ "num_filters:{},\n" \ "dropout_keep_prob:{},\n" \ "l2_reg_lambda:{},\n" \ "num_epochs:{},\n" \ "batch_size:{}".format(FLAGS.embedding_dim,FLAGS.filter_sizes,FLAGS.num_filters, FLAGS.dropout_keep_prob,FLAGS.l2_reg_lambda,FLAGS.num_epochs, FLAGS.batch_size) open(out_path, 'w').write(parameters) # Training # ================================================== def train(X_train, X_dev, x_test, y_train, y_dev, y_test): with tf.Graph().as_default(): session_conf = tf.ConfigProto( allow_soft_placement=FLAGS.allow_soft_placement, log_device_placement=FLAGS.log_device_placement) sess = tf.Session(config=session_conf) with sess.as_default(): cnn = TextCNN( sequence_length=max_document_length, num_classes=2, vocab_size=len(vocab_processor.vocabulary_), embedding_size=FLAGS.embedding_dim, embedding_table=W, filter_sizes=list(map(int, FLAGS.filter_sizes.split(","))), num_filters=FLAGS.num_filters, l2_reg_lambda=FLAGS.l2_reg_lambda) # Define Training procedure global_step = tf.Variable(0, name="global_step", trainable=False) optimizer = tf.train.AdamOptimizer(1e-3) grads_and_vars = optimizer.compute_gradients(cnn.loss) train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step) # Keep track of gradient values and sparsity (optional) grad_summaries = [] for g, v in grads_and_vars: if g is not None: grad_hist_summary = tf.summary.histogram("{}/grad/hist".format(v.name), g) sparsity_summary = tf.summary.scalar("{}/grad/sparsity".format(v.name), tf.nn.zero_fraction(g)) grad_summaries.append(grad_hist_summary) grad_summaries.append(sparsity_summary) grad_summaries_merged = tf.summary.merge(grad_summaries) # Output directory for models and summaries timestamp = str(int(time.time())) out_dir = os.path.abspath(os.path.join(os.path.curdir, "runs", timestamp)) print("Writing to {}\n".format(out_dir)) # Summaries for loss and accuracy loss_summary = tf.summary.scalar("loss", cnn.loss) acc_summary = tf.summary.scalar("accuracy", cnn.accuracy) # Train Summaries train_summary_op = tf.summary.merge([loss_summary, acc_summary, grad_summaries_merged]) train_summary_dir = os.path.join(out_dir, "summaries", "train") train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph) # Dev summaries dev_summary_op = tf.summary.merge([loss_summary, acc_summary]) dev_summary_dir = os.path.join(out_dir, "summaries", "dev") dev_summary_writer = tf.summary.FileWriter(dev_summary_dir, sess.graph) # Checkpoint directory. Tensorflow assumes this directory already exists so we need to create it checkpoint_dir = os.path.abspath(os.path.join(out_dir, "checkpoints")) checkpoint_prefix = os.path.join(checkpoint_dir, "model") if not os.path.exists(checkpoint_dir): os.makedirs(checkpoint_dir) saver = tf.train.Saver(tf.global_variables()) # Write vocabulary vocab_processor.save(os.path.join(out_dir, "vocab")) # Initialize all variables # sess.run(tf.initialize_all_variables()) sess.run(tf.global_variables_initializer()) def train_step(x_batch, y_batch): """ A single training step """ feed_dict = { cnn.input_x: x_batch, cnn.input_y: y_batch, cnn.dropout_keep_prob: FLAGS.dropout_keep_prob } _, step, summaries, loss, accuracy = sess.run( [train_op, global_step, train_summary_op, cnn.loss, cnn.accuracy], feed_dict) # _, step, loss, accuracy = sess.run( # [train_op, global_step, cnn.loss, cnn.accuracy], # feed_dict) time_str = datetime.datetime.now().isoformat() print("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy)) train_summary_writer.add_summary(summaries, step) def dev_step(x_batch, y_batch, writer=None): """ Evaluates model on a dev set """ feed_dict = { cnn.input_x: x_batch, cnn.input_y: y_batch, cnn.dropout_keep_prob: 1.0 } step, summaries, loss, accuracy = sess.run( [global_step, dev_summary_op, cnn.loss, cnn.accuracy], feed_dict) # step, loss, accuracy = sess.run( # [global_step, cnn.loss, cnn.accuracy], # feed_dict) time_str = datetime.datetime.now().isoformat() print("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy)) if writer: writer.add_summary(summaries, step) # Generate batches batches = data_loader.batch_iter( list(zip(X_train, y_train)), FLAGS.batch_size, FLAGS.num_epochs) # Training loop. For each batch... for batch in batches: x_batch, y_batch = zip(*batch) train_step(x_batch, y_batch) current_step = tf.train.global_step(sess, global_step) if current_step % FLAGS.evaluate_every == 0: print("\nEvaluation:") dev_step(X_dev, y_dev, writer=dev_summary_writer) # dev_step(X_dev, y_dev, writer=None) print("") if current_step % FLAGS.checkpoint_every == 0: path = saver.save(sess, checkpoint_prefix, global_step=current_step) print("Saved model checkpoint to {}\n".format(path)) # Test loop # Generate batches for one epoch batches = data_loader.batch_iter(list(x_test), FLAGS.batch_size, 1, shuffle=False) # Collect the predictions here all_predictions = [] for x_test_batch in batches: batch_predictions = sess.run(cnn.predictions, {cnn.input_x: x_test_batch, cnn.dropout_keep_prob: 1.0}) all_predictions = np.concatenate([all_predictions, batch_predictions]) correct_predictions = float(sum( all_predictions == np.argmax(y_test,axis=1))) print("Total number of test examples: {}".format(len(y_test))) print("Accuracy: {:g}".format(correct_predictions / float(len(y_test)))) # open(os.path.join(out_dir,"test"),'a').write("Accuracy: {:g}".format(correct_predictions / float(len(y_test)))) out_path = os.path.abspath(os.path.join(os.path.curdir, "runs","test")) open(out_path,'a').write("{:g},".format(correct_predictions / float(len(y_test)))) print("\n写入成功！\n") # cross-validation kf = cross_validation.KFold(len(x_shuffled), n_folds=3) for train_index, test_index in kf: X_train_total = x_shuffled[train_index] y_train_total = y_shuffled[train_index] x_test = x_shuffled[test_index] y_test = y_shuffled[test_index] # 分割训练集与验证集 X_train, X_dev, y_train, y_dev = cross_validation.train_test_split( X_train_total, y_train_total, test_size=0.2, random_state=0) print("Vocabulary Size: {:d}".format(len(vocab_processor.vocabulary_))) print("Train/Dev split: {:d}/{:d}".format(len(y_train), len(y_dev))) 训练完成后准确率83%左右，还需要在进一步进行改进来提升正确率，比如利用chunk max-pooling方法代替max-pooling，利用集成的方法，因为word embedding词忽略了当前上下文的含义，潜在认为相同词在不同文本中的含义相同，所以可以利用词义消歧来提升其正确率等等; 训练模型保存在checkpoints中，由model-4000.index,model-4000.meta,model-4000.data等; 最后tensorboard --logdir /home/yang/PycharmProjects/cnn-text-classification-master/runs/1515468832 /home/yang/PycharmProjects/cnn-text-classification-master/runs/1515468832/checkpoints/model-4300 Total number of test examples: 1333 Accuracy: 0.825956
扫一扫
598
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
