cross entropy loss_tf.nn.sampled_softmax_loss 示例

语法结构：

tf.nn.sampled_softmax_loss(
    weights, biases, labels, inputs, num_sampled, num_classes, num_true=1,
    sampled_values=None, remove_accidental_hits=True, seed=None,
    name='sampled_softmax_loss'
)

weightsATensorof shape[num_classes,dim], or a list ofTensorobjects whose concatenation along dimension 0 has shape [num_classes, dim]. The (possibly-sharded) class embeddings.

biasesATensorof shape[num_classes]. The class biases.

labelsATensorof typeint64and shape[batch_size,num_true]. The target classes. Note that this format differs from thelabelsargument ofnn.softmax_cross_entropy_with_logits.

inputsATensorof shape[batch_size,dim]. The forward activations of the input network.

num_sampledAnint. The number of classes to randomly sample per batch.

num_classesAnint. The number of possible classes.

num_trueAnint. The number of target classes per training example.

sampled_valuesa tuple of (sampled_candidates,true_expected_count,sampled_expected_count) returned by a*_candidate_samplerfunction. (if None, we default tolog_uniform_candidate_sampler)

remove_accidental_hitsAbool. whether to remove "accidental hits" where a sampled class equals one of the target classes. Default is True.

seedrandom seed for candidate sampling. Default to None, which doesn't set the op-level random seed for candidate sampling.

nameA name for the operation (optional).

示例：

import tensorflow as tf

a = tf.nn.sampled_softmax_loss(weights=tf.constant([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]),
                               # [num_classes, dim] = [10, 2]
                               biases=tf.constant([0., 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),
                               # [num_classes] = [10]
                               labels=tf.constant([[2], [3], [5]]),
                               # [batch_size, num_true] = [3, 1]
                               inputs=tf.constant([[0.2, 0.1], [0.4, 0.1], [0.22, 0.12]]),
                               # [batch_size, dim] = [3, 2]
                               
                               num_sampled=3,
                               num_classes=10,
                               num_true = 1,
                               seed = 2020,
                               name = "sampled_softmax_loss"
                          )

结果：

with tf.compat.v1.Session() as sess:
    print(sess.run(a))

# [0.37478584 0.2859666  0.0703702 ]

在源码的介绍中，train的方式和eval的方式是分开的：

if mode == "train":
    loss = tf.nn.sampled_softmax_loss(
        weights=weights,
        biases=biases,
        labels=labels,
        inputs=inputs,
        ...)

elif mode == "eval":
    logits = tf.matmul(inputs, tf.transpose(weights))
    logits = tf.nn.bias_add(logits, biases)
    labels_one_hot = tf.one_hot(labels, n_classes)
    loss = tf.nn.softmax_cross_entropy_with_logits(
        labels=labels_one_hot,
        logits=logits)

train步骤中除了选择一批label作为负样本外，实现方式与eval中描述的一致。

参考：

https://tensorflow.google.cn/api_docs/python/tf/nn/sampled_softmax_loss?hl=zh-cn

https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/ops/nn_impl.py#L2066-L2154