一、 embedding + dropout.
embedded = tf.nn.embedding_lookup(self.var, x)
if self.keep_prob < 1.:
shape = embedded.get_shape().as_list()
# Use same dropout masks at each timestep with specifying noise_shape.
# This slightly improves performance.
# Please see https://arxiv.org/abs/1512.05287 for the theoretical
# explanation.
embedded = tf.nn.dropout(embedded, 'self.keep_prob, noise_shape=(shape[0], 1, shape[2]))
return embedded
该代码, 在lookup后的embedded进行了drop_out ,并且使用了noise_shape, 据说是有提升的。
- noise_shape
By default, each element is kept or dropped independently. If `noise_shape`
is specified, it must be [broadcastable](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
to the shape of `x`, and only dimensions with `noise_shape[i] == shape(x)[i]`
will make independent decisions. For example, if `shape(x) = [k, l, m, n]`
and `noise_shape = [k, 1, 1, n]`, each batch and channel component will be
kept independently and each row and column will be kept or not kept together.