一、如何代表文字数据
one-hot representation：由0和1构成，容易导致过大的维度，计算低效。并且无法识别语义内容，如近义词。
word embedding（词嵌入）:根据一个词的上下文去代表一个单词。相比上面的one-hot，有维度低、连续向量、能够识别语义信息等优点。参考知乎：有谁可以解释一下word embedding?
word2vec是由Tomas Mikolv带领团队创建的一组用来处理word embedding的模型。其中主要有两种模型：skip-gram和CBOW，前者根据中心词预测上下文，后者根据上下文预测中心词。
二、建立skip-gram模型
我们的任务是训练模型，使它能够根据中心词预测上下文。给定一句话的中心词，随机在前后挑选一个单词，然后网络告诉你这个词是我们的字母表中某一个词的可能性。
为了获取上下文单词的概率分布，通常我们使用softmax，它将任意值xi变换到一个概率分布pi。在这种情况下softmax(xi)就是一个临近单词是字母表中某一单词的可能性。但softmax意味着指数运算，这严重限制了计算速度。通常有两种方法解决这一瓶颈： hierarchical softmax和sample-based softmax。Mikolv在他的论文中指出，对于skip-gram模型的训练，negative sample的效果表现得比hierarchical softmax更好。negative sampling实际上是Noise Contrastive Estimation（NCE）的一种简化版本，但只有NCE能保证是对softmax的近似。注意，它们只能减少训练时间，softmax仍需要完整计算以获得归一化的概率分布。
三、Variable Sharing
1、Name Scope
为了便于在tensorboard中观察图和节点，我们需要使用name scope对节点进行分组。
with tf.name_scope(name_of_that_scope):
# declare op_1
# declare op_2
# ...
分组后的图如上所示。在tensorboard中，有三种线：实心灰色箭头，实心橙色箭头，虚线灰色箭头。实心灰色箭头表示数据流向，实心橙色箭头表示节点的控制，在图中optimizer可以改变nce_weight、nce_bias和embed_matrix，虚线灰色箭头表示控制依赖关系，如nce_weight必须在init后才能使用。
2、Variable Scope
建立一个两层的神经网络，输入两组不同的数据x1和x2：
x1 = tf.truncated_normal([200, 100], name='x1')
x2 = tf.truncated_normal([200, 100], name='x2')def two_hidden_layers(x):
assert x.shape.as_list() == [200, 100]
w1 = tf.Variable(tf.random_normal([100, 50]), name="h1_weights")
b1 = tf.Variable(tf.zeros([50]), name="h1_biases")
h1 = tf.matmul(x, w1) + b1
assert h1.shape.as_list() == [200, 50]
w2 = tf.Variable(tf.random_normal([50, 10]), name="h2_weights")
b2 = tf.Variable(tf.zeros([10]), name="h2_biases")
logits = tf.matmul(h1, w2) + b2
return logits
logits1 = two_hidden_layers(x1)
logits2 = two_hidden_layers(x2)使用tensorboard可视化：
with tf.variable_scope('two_layers') as scope:
logits1 = two_hidden_layers_2(x1)
scope.reuse_variables()
logits2 = two_hidden_layers_2(x2)
def fully_connected(x, output_dim, scope):
with tf.variable_scope(scope) as scope:
w = tf.get_variable("weights", [x.shape[1], output_dim], initializer=tf.random_normal_initializer())
b = tf.get_variable("biases", [output_dim], initializer=tf.constant_initializer(0.0))
return tf.matmul(x, w) + b
def two_hidden_layers(x):
h1 = fully_connected(x, 50, 'h1')
h2 = fully_connected(h1, 10, 'h2')
with tf.variable_scope('two_layers') as scope:
logits1 = two_hidden_layers(x1)
scope.reuse_variables()
logits2 = two_hidden_layers(x2)
重新可视化就会得到我们希望的结果。
四、word2vec源码
""" starter code for word2vec skip-gram model with NCE loss
CS 20: "TensorFlow for Deep Learning Research"
cs20.stanford.edu
Chip Huyen (chiphuyen@cs.stanford.edu)
Lecture 04
"""
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
import numpy as np
from tensorflow.contrib.tensorboard.plugins import projector
import tensorflow as tf
import utils
import word2vec_utils
# Model hyperparameters
VOCAB_SIZE = 50000
BATCH_SIZE = 128
EMBED_SIZE = 128 # dimension of the word embedding vectors
SKIP_WINDOW = 1 # the context window
NUM_SAMPLED = 64 # number of negative examples to sample
LEARNING_RATE = 1.0
NUM_TRAIN_STEPS = 100000
VISUAL_FLD = 'visualization'
SKIP_STEP = 5000
# Parameters for downloading data
DOWNLOAD_URL = 'http://mattmahoney.net/dc/text8.zip'
EXPECTED_BYTES = 31344016
NUM_VISUALIZE = 3000 # number of tokens to visualize
def word2vec(dataset):
""" Build the graph for word2vec model and train it """
# Step 1: get input, output from the dataset
with tf.name_scope('data'):
iterator = dataset.make_initializable_iterator()
center_words, target_words = iterator.get_next()
""" Step 2 + 3: define weights and embedding lookup.
In word2vec, it's actually the weights that we care about
"""
with tf.name_scope('embed'):
embed_matrix = tf.get_variable('embed_matrix',
shape=[VOCAB_SIZE, EMBED_SIZE],
initializer=tf.random_uniform_initializer())
embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embedding')
# Step 4: construct variables for NCE loss and define loss function
with tf.name_scope('loss'):
nce_weight = tf.get_variable('nce_weight', shape=[VOCAB_SIZE, EMBED_SIZE],
initializer=tf.truncated_normal_initializer(stddev=1.0 / (EMBED_SIZE ** 0.5)))
nce_bias = tf.get_variable('nce_bias', initializer=tf.zeros([VOCAB_SIZE]))
# define loss function to be NCE loss function
loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight,
biases=nce_bias,
labels=target_words,
inputs=embed,
num_sampled=NUM_SAMPLED,
num_classes=VOCAB_SIZE), name='loss')
# Step 5: define optimizer
with tf.name_scope('optimizer'):
optimizer = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss)
utils.safe_mkdir('checkpoints')
with tf.Session() as sess:
sess.run(iterator.initializer)
sess.run(tf.global_variables_initializer())
total_loss = 0.0 # we use this to calculate late average loss in the last SKIP_STEP steps
writer = tf.summary.FileWriter('graphs/word2vec_simple', sess.graph)
for index in range(NUM_TRAIN_STEPS):
try:
loss_batch, _ = sess.run([loss, optimizer])
total_loss += loss_batch
if (index + 1) % SKIP_STEP == 0:
print('Average loss at step {}: {:5.1f}'.format(index, total_loss / SKIP_STEP))
total_loss = 0.0
except tf.errors.OutOfRangeError:
sess.run(iterator.initializer)
writer.close()
def gen():
yield from word2vec_utils.batch_gen(DOWNLOAD_URL, EXPECTED_BYTES, VOCAB_SIZE,
BATCH_SIZE, SKIP_WINDOW, VISUAL_FLD)
def main():
dataset = tf.data.Dataset.from_generator(gen,
(tf.int32, tf.int32),
(tf.TensorShape([BATCH_SIZE]), tf.TensorShape([BATCH_SIZE, 1])))
word2vec(dataset)
if __name__ == '__main__':
main()
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
