variables

a tensorflow variable is the best way to represent shared, persistent state manipulated by your program


variables are manipulated via the tf.Variable class. a tf.Variable represents a tensor whose value can be changed by running ops on it. unlike tf.Tensor objects, a tf.Variable exists outside the context of a single session.run call


internally, a tf.Variable stores a persistent tensor. specific ops allow you to read and modify the values of this tensor. these modifications are visible across multiple tf.Session, so multiple workers can see the same values for a tf.Variable


the best way to create a variable is to call the tf.get_variable function. this function requires you to specify the Variable's name. this name will be used by other replicas to access the same variable, as well as to name this variable's value when checkpointing and exporting models. tf.get_variable also allows you to reuse a previously created variable of the same name, making it easy to define models which reuse layers


note that when the initializer is a tf.Tensor you should not specify the variable's shape, as the shape of the initializer tensor will be used


because disconnected parts of a tensorflow program might want to create variables, it is sometimes useful to have a single way to access all of them. for this reason tensorflow provides collections, which are named lists of tensors or other objects, such as tf.Variable instances


by default every tf.Variable gets placed in the following two collections:
(1)tf.GraphKeys.GLOBAL_VARIABLES -- variables that can be shared across multiple devices
(2)tf.GraphKeys.TRAINABLE_VARIABLES -- variables for which tensorflow will calculate gradients


if you don't want a variable to be trainable, add it to the tf.GraphKeys.LOCAL_VARIABLES collection instead
###
my_local = tf.get_variable("my_local", shape=(),
collections=[tf.GraphKeys.LOCAL_VARIABLES])


alternatively, you can specify trainable=False as an argument to tf.get_variable
###
my_non_trainable = tf.get_variable("my_non_trainable",
shape=(),
trainable=False)


you can also use your own collections. any string is a valid collection name, and there is no need to explicitly create a collection. to add a variable (or any other object) to a collection after creating the variable, call tf.add_to_collection. for example, the following code adds an existing variable named my_local to a collection named my_collection_name
###
tf.add_to_collection("my_collection_name", my_local)


and to retrieve a list of all the variables (or other objects) you've placed in a collection you can use
###
tf.get_collection("my_collection_name")


just like any other tensorflow operation, you can place variables on particular devices


it is particularly important for variables to be in the correct device in distributed settings. accidentally putting variables on workers instead of parameter servers, for example, can severely slow down training. for this reason we provide tf.train.replica_device_setter, which can automatically place variables in parameter servers


before you can use a variable, it must be initialized. most high-level frameworks such as tf.contrib.slim, tf.estimator.Estimator and Keras automatically initialize variables for you before training a model


explicit initialization is otherwise useful because it allows you not to rerun potentially expensive initializes when reloading a model from a checkpoint as well as allowing determinism when randomly-initialized variables are shared in a distributed setting


to initalize all trainable variables in one go, before training starts, call tf.global_variables_initializer(). this function returns a single operation responsible for initializing all variables in the tf.GraphKeys.GLOBAL_VARIABLES collection. running this operation initializes all variables


if you do need to initialize variables yourself, you can run the variable's initializer operation
###
sess.run(my_variable.initializer)


note that by default tf.global_variables_initializer does not specify the order in which variables are initialized. therefore, if the initial value of a variable depends on another variable's value, it's likely that you'll get an error. any time you use the value of a variable in a context in which not all variables are initialized(say, if you use a variable's value while initializing another variable), it is best to use variable.initialized_value() instead of variable
###
v = tf.get_variable("v", shape=(), initializer=tf.zeros_initializer())
w = tf.get_variable("w", initializer=v.initialized_value() + 1)


to assign a value to a variable, use the methods assign, assign_add, and friends in the tf.Variable class


because variables are mutable it's sometimes useful to know what version of a variable's value is being used at point in time. to force a re-read of the value of a variable after something has happened, you can use tf.Variable.read_value


tensorflow supports two ways of sharing variables:
(1) explicitly passing tf.Variable objects around
(2) implicitly wrapping tf.Variable objects within tf.variable_scope objects


while code which explicitly passes variables around is very clear, it is sometimes convenient to write tensorflow functions that implicitly use variables in their implementations. most of the function layers from tf.layer use this approach, as well as all tf.metrics, and a few other library utilities


variable scopes allow you to control variable reuse when calling functions which implicitly create and use variables. they also allow you to name your variables in a hierarchical and understandable way


def conv_relu(input, kernel_shape, bias_shape):
weights = tf.get_variable("weights", kernel_shape, 
initializer=tf.random_normal_initializer())
biases = tf.get_variable("biases", bias_shape,
initializer=tf.zeros_initializer())
conv = tf.nn.conv2d(input, weights,
strides=[1, 1, 1, 1], padding="SAME")
return tf.nn.relu(conv + biases)


input1 = tf.random_normal([1, 10, 10, 32])
input2 = tf.random_normal([1, 20, 20, 3])
x = conv_relu(input1, kernel_shape=[5, 5, 32, 32], bias_shape=[32])
x = conv_relu(input2, kernel_shape=[3, 3, 3, 1], bias_shape=[1])