NLP(一)电影评论情感分析，并在此基础上观察单词的相似性

NLP（一）电影评论情感分析

• 电影评论数据集预处理

import sys
f = open('reviews.txt')
raw_reviews = f.readlines()
f.close()
f = open('labels.txt')
raw_labels = f.readlines()
f.close()

tokens = list(map(lambda x:set(x.split(" ")), raw_reviews)) #map对右边的参数做左边参数操作
vocab = set()
for sent in tokens:
    for word in sent:
        if len(word)>0:
            vocab.add(word)
vocab = list(vocab)

word2index = {}
for i,word in enumerate(vocab):
    word2index[word] = i

#构建输入数据
input_dataset = list()
for sent in tokens:
    sent_indices = list()
    for word in sent:
        try:
            sent_indices.append(word2index[word])
        except:
            ""
    input_dataset.append(list(set(sent_indices))) #在各个可能出现重复的地方都用了set

target_dataset = list()
for label in raw_labels:
    if label == 'positive\n':
        target_dataset.append(1)
    else:
        target_dataset.append(0)
print(target_dataset[0:10])

[1, 0, 1, 0, 1, 0, 1, 0, 1, 0]

• 训练,两层网络
• 解释一下 outer
  result = numpy.outer(a, b) 用来求外积的，非常直观，比矩阵相乘简单 ， a，b是两个数组，如果a，b是高维数组，函数会自动将其flatten成1维 ， a的长度是m，b的长度是n，外积的结果result是 m * n的数组,数组元素为 result[i,j] = a[i] * a[j]

import numpy as np
np.random.seed(1)

def sigmoid(x):
    return 1/(1 + np.exp(-x))
alpha, iteration = (0.01, 2)
hidden_size = 100
weight_0_1 = 0.2 * np.random.random((len(vocab),hidden_size)) - 0.1
weight_1_2 = 0.2 * np.random.random((hidden_size,1)) - 0.1

correct, total = (0,0)
for iter in range(iteration):
    for i in range(10000): #训练10000条
        x,y = (input_dataset[i], target_dataset[i])
        layer_1 = sigmoid(np.sum(weight_0_1[x],axis=0)) #因为输入都是0，1，所以这样比计算矩阵乘法好很多,把相关的权重加起来
        layer_2 = sigmoid(np.dot(layer_1,weight_1_2))
        #反向传播
        layer_2_delta = layer_2 - y
        layer_1_delta = layer_2_delta.dot(weight_1_2.T)
        
        weight_0_1[x] -= layer_1_delta * alpha
        weight_1_2 -= np.outer(layer_1, layer_2_delta) * alpha
        
        if(np.abs(layer_2_delta) < 0.5): #正确结果非0即1，所以可以用这个判断
            correct += 1
        total += 1
        if(i % 1000 == 0):
            print('iter:'+ str(iter) + 'index:'+ str(i) + 'correct:' + str(correct/total))

iter:0index:0correct:1.0
iter:0index:1000correct:0.46653346653346656
iter:0index:2000correct:0.5977011494252874
iter:0index:3000correct:0.6717760746417861
iter:0index:4000correct:0.7040739815046239
iter:0index:5000correct:0.7218556288742252
iter:0index:6000correct:0.7390434927512082
iter:0index:7000correct:0.7563205256391944
iter:0index:8000correct:0.7695288088988876
iter:0index:9000correct:0.7775802688590157
iter:1index:0correct:0.7872212778722127
iter:1index:1000correct:0.7942005272247977
iter:1index:2000correct:0.8015998666777768
iter:1index:3000correct:0.8098607799400046
iter:1index:4000correct:0.8162274123276909
iter:1index:5000correct:0.8219452036530898
iter:1index:6000correct:0.8267608274482845
iter:1index:7000correct:0.8320687018410682
iter:1index:8000correct:0.8372868173990334
iter:1index:9000correct:0.8411136255986527

• 随便找个样本测一下
• 下面这句话对应的token其实是一样的，只不过因为set，单词的排序乱了

sentence = raw_reviews[10]
sen = tokens[10]
print(sentence)
print(sen)

this isn  t the comedic robin williams  nor is it the quirky  insane robin williams of recent thriller fame . this is a hybrid of the classic drama without over  dramatization  mixed with robin  s new love of the thriller . but this isn  t a thriller  per se . this is more a mystery  suspense vehicle through which williams attempts to locate a sick boy and his keeper .  br    br   also starring sandra oh and rory culkin  this suspense drama plays pretty much like a news report  until william  s character gets close to achieving his goal .  br    br   i must say that i was highly entertained  though this movie fails to teach  guide  inspect  or amuse . it felt more like i was watching a guy  williams   as he was actually performing the actions  from a third person perspective . in other words  it felt real  and i was able to subscribe to the premise of the story .  br    br   all in all  it  s worth a watch  though it  s definitely not friday  saturday night fare .  br    br   it rates a  .     from . . .  br    br   the fiend  .  

{'', 'hybrid', 'of', 'without', 'must', 'person', 'highly', 'to', 'that', 'se', 'rates', 'was', 'over', 'boy', 'keeper', 'a', 's', 'comedic', 'rory', 'sick', 'mystery', 'starring', 'mixed', 'watch', 'the', 'inspect', 'achieving', 'fiend', 'as', 'also', 'he', 'real', 'vehicle', 'guy', 'plays', 'per', 'premise', 'new', 'oh', '.', 'robin', 'movie', 'br', 'is', 'with', 'insane', 'like', 'from', 'culkin', 'more', 't', 'thriller', 'third', 'words', 'locate', 'close', 'performing', 'drama', 'fails', 'entertained', 'other', 'i', 'guide', 'report', 'felt', 'perspective', 'fare', 'his', 'which', 'all', 'news', 'actions', 'teach', 'watching', 'suspense', 'goal', 'isn', 'gets', 'until', 'love', 'through', 'sandra', 'and', 'actually', 'though', 'friday', 'character', 'definitely', 'story', 'dramatization', 'worth', 'william', 'pretty', '\n', 'classic', 'say', 'quirky', 'not', 'or', 'williams', 'fame', 'attempts', 'able', 'night', 'but', 'this', 'amuse', 'in', 'saturday', 'subscribe', 'it', 'nor', 'much', 'recent'}

layer_1_value = sigmoid(np.sum(weight_0_1[input_dataset[10]],axis=0))
pre = sigmoid(np.dot(layer_1_value, weight_1_2))
print(pre)
if(pre > 0.5):
    print('预测为积极')
else:
    print('预测为消极')
print('真实值：' + raw_labels[10])

[0.92190673]
预测为积极
真实值：positive

在这个例子中，看看单词的相似性

• 因为第一层权重是 len(vocab) * hiddensize 的，所以每个词都有自己对应的hiddensize个权重值，那么每个词对positive / nagetive的学习情况都在权重里面表现了。

word = 'man'
import math
from collections import Counter
#通过看单词到隐藏层的权重的相似程度来判断
def word_similar(x):
    x_index = word2index[x]
    scores = Counter()
    for word,index in word2index.items():
        distance = (weight_0_1[index] - weight_0_1[x_index])**2
        scores[word] = -math.sqrt(np.sum(distance))
    return scores.most_common(10)
print(word_similar(word))

[('man', -0.0), ('slingblade', -0.5648583520454541), ('ashok', -0.6107447183616072), ('talen', -0.6244865719084539), ('tsurube', -0.6288293245899911), ('mulroney', -0.6339924052895604), ('conga', -0.6347788631488298), ('unprofessionally', -0.6373325281488093), ('longinotto', -0.6391352679326392), ('causality', -0.6397687287907046)]

数据集链接：
https://download.csdn.net/download/zjh12312311/12868190