一、前言
在NLP篇【02】白话Word2vec原理以及层softmax、负采样的实现 文章中介绍了层softmax与负采样,有兴趣可以根据介绍,自己实现一下负采样或层softmax。当然,对于只想拿来直接使用的,可以看看这篇文章,直接利用TensorFlow 的api实现负采样,也是非常的简单。
二、TensorFlow 代码
api介绍:
tf.nn.sampled_softmax_loss(weights, # Shape (num_classes, dim) - floatXX
biases, # Shape (num_classes) - floatXX
labels, # Shape (batch_size, num_true) - int64
inputs, # Shape (batch_size, dim) - floatXX
num_sampled, # - int
num_classes, # - int
num_true=1,
sampled_values=None,
remove_accidental_hits=True,
partition_strategy="mod",
name="sampled_softmax_loss")
具体代码:
import tensorflow as tf
# Network Parameters
n_hidden_1 = 256 # 1st layer number of features
n_input = 784 # MNIST data input (img shape: 28*28)
n_classes = 10 # MNIST total classes (0-9 digits)
# Dependent & Independent Variable Placeholders
x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes]) #
# Weights and Biases
weights = {
'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'out': tf.Variable(tf.random_normal([n_hidden_1, n_classes]))
}
biases = {
'b1': tf.Variable(tf.random_normal([n_hidden_1])),
'out': tf.Variable(tf.random_normal([n_classes]))
}
# Super simple model builder
def tiny_perceptron(x, weights, biases):
layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
out_layer = tf.nn.relu(layer_1)
# out_layer = tf.matmul(layer_1, weights['out']) + biases['out']
return out_layer #(bt,dim)
# Create the model
pred = tiny_perceptron(x, weights, biases)
# Set up loss function inputs and inspect their shapes
w = tf.transpose(weights['out']) #(num_class,dim)
b = biases['out']
labels = tf.reshape(tf.argmax(y, 1), [-1,1])
inputs = pred
num_sampled = 3
num_true = 1
num_classes = n_classes
print('Shapes\n------\nw:\t%s\nb:\t%s\nlabels:\t%s\ninputs:\t%s' % (w.shape, b.shape, labels.shape, inputs.shape))
# Shapes
# ------
# w: (10, 256) # Requires (num_classes, dim) - CORRECT
# b: (10,) # Requires (num_classes) - CORRECT
# labels: (?, 1) # Requires (batch_size, num_true) - CORRECT
# inputs: (?, 256) # Requires (batch_size, dim) - CORRECT
loss_function = tf.reduce_mean(tf.nn.sampled_softmax_loss(
weights=w, #(num_class,dim)
biases=b,
labels=labels,
inputs=inputs, #(bt,dim)
num_sampled=num_sampled,
num_true=num_true,
num_classes=num_classes)
)
注意:这里的labels如果是one-hot类型编码,需要labels=tf.reshape(tf.argmax(labels_one_hot, 1), [-1,1])
示例:
import tensorflow as tf
a = tf.nn.sampled_softmax_loss(weights=tf.constant([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]),
# [num_classes, dim] = [10, 2]
biases=tf.constant([0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),
# [num_classes] = [10]
labels=tf.constant([[2], [3], [5]]),
# [batch_size, num_true] = [3, 1]
inputs=tf.constant([[0.2, 0.1], [0.4, 0.1], [0.22, 0.12]]),
# [batch_size, dim] = [3, 2]
num_sampled=3,
num_classes=10,
num_true = 1,
seed = 2020,
name = "sampled_softmax_loss"
)
结果:
with tf.compat.v1.Session() as sess:
print(sess.run(a))
# [0.37478584 0.2859666 0.0703702 ]
在源码的介绍中,train的方式和eval的方式是分开的:
if mode == "train":
loss = tf.nn.sampled_softmax_loss(
weights=weights,
biases=biases,
labels=labels,
inputs=inputs,
...)
elif mode == "eval":
logits = tf.matmul(inputs, tf.transpose(weights))
logits = tf.nn.bias_add(logits, biases)
labels_one_hot = tf.one_hot(labels, n_classes)
loss = tf.nn.softmax_cross_entropy_with_logits(
labels=labels_one_hot,
logits=logits)
train步骤中除了选择一批label作为负样本外,实现方式与eval中描述的一致。