TensorFlow流程控制函数：tf.group tf.control_dependencies

group(
    *inputs,
    **kwargs
)

定义在：tensorflow/python/ops/control_flow_ops.py。

请参阅指南：控制流程>控制流程操作

1. 创建一个操作，该操作可以对 TensorFlow 的多个操作进行分组。

当这个操作完成后，所有 input 中的所有 ops 都已完成。这个操作没有输出。

另请参见 tuple 和 control_dependencies 获得更多信息。

参数：

input：需要进行分组的零个或多个张量。

kwargs：构造 NodeDef 时要传递的可选参数。 

name：此操作的名称（可选）。

返回值：

该函数返回执行其所有输入的操作。

可能引发的异常：

• ValueError：如果提供了一个未知的关键字参数。

import tensorflow as tf
tag =tf.reduce_any([True])
b = tf.Variable(tf.random_normal([5,3],stddev=0.35),name="weights")
C = b
#init = tf.initialize_all_variables()
sess = tf.Session()
#sess.run(init)
# print(sess.run(b.initialized_value()))
with tf.control_dependencies([tag]):
    do_updates = tf.group(
        tf.reduce_sum(input_tensor=b),
        b.assign(b+10),
        C)
init = tf.initialize_all_variables()
sess.run(init)

print(sess.run(C))
print(sess.run([do_updates]))
print(sess.run(tag))
print(sess.run(b))
print(sess.run(C))

 

输出

 

[[ 0.42285722  0.46158823 -0.26109645]
 [-0.64646924  0.05288799 -0.2152937 ]
 [-0.0093157   0.52074057 -0.47927105]
 [ 0.5385346   0.05081145  0.04212087]
 [ 0.40977108  0.15353793  0.05755007]]
[None]
True
[[10.422857  10.461588   9.738904 ]
 [ 9.353531  10.052888   9.784706 ]
 [ 9.9906845 10.5207405  9.520729 ]
 [10.538534  10.050812  10.042121 ]
 [10.409771  10.153538  10.05755  ]]
[[10.422857  10.461588   9.738904 ]
 [ 9.353531  10.052888   9.784706 ]
 [ 9.9906845 10.5207405  9.520729 ]
 [10.538534  10.050812  10.042121 ]

 [10.409771  10.153538  10.05755  ]]

import tensorflow as tf
tag =tf.reduce_any([True])
b = tf.Variable(tf.random_normal([5,3],stddev=0.35),name="weights")
C = b
#init = tf.initialize_all_variables()
sess = tf.Session()
#sess.run(init)
# print(sess.run(b.initialized_value()))
with tf.control_dependencies([tag]):
    do_updates = tf.group(
        tf.reduce_sum(input_tensor=b),
        b.assign(b+10),
        C)
init = tf.initialize_all_variables()
sess.run(init)

print(sess.run(C))
# print(sess.run([do_updates]))
print(sess.run(tag))
print(sess.run(b))
print(sess.run(C))

 

输出

 

[[ 0.5533742   0.19960515  0.44480166]
 [-0.23665977 -0.29964206 -0.17953995]
 [-0.40328383  0.05090697 -0.67651725]
 [-0.02874281 -0.15773363 -0.12723291]
 [-0.10975635 -0.04863227  0.1665113 ]]
True
[[ 0.5533742   0.19960515  0.44480166]
 [-0.23665977 -0.29964206 -0.17953995]
 [-0.40328383  0.05090697 -0.67651725]
 [-0.02874281 -0.15773363 -0.12723291]
 [-0.10975635 -0.04863227  0.1665113 ]]
[[ 0.5533742   0.19960515  0.44480166]
 [-0.23665977 -0.29964206 -0.17953995]
 [-0.40328383  0.05090697 -0.67651725]
 [-0.02874281 -0.15773363 -0.12723291]
 [-0.10975635 -0.04863227  0.1665113 ]]

 

 

1. difference between tf.group and  tf.control_dependencies

 

If you look at the graphdef, the c=tf.group(a, b) produces the same graph as

with tf.control_dependencies([a, b]):
    c = tf.no_op() 

There's no specific order in which ops will run, TensorFlow tries to execute operations as soon as it can (i.e. in parallel).

2. difference between tf.group and  tf.control_dependencies

 

As you can see from Yaroslav's answer, the main difference is that tf.control_depenencies adds no ops (nodes) to the computation graph, while tf.group creates and returns a new op (node).

In addition, if inputs belong to multiple devices, tf.group will insert an intermediate layer between the node it returns and the inputs. That layer will contain one node per device, so that the dependencies are organized by device. This reduces the cross-device data flow.

So if your dependencies are on multiple devices, tf.group adds some optimization.

On the other hand, tf.control_dependencies has a nice behavior with nesting. The inner context will add dependencies to the union of all the ops in the outer contexts.

 

2. tf.control_dependencies()

tf.control_dependencies()设计是用来控制计算流图的，给图中的某些计算指定顺序。比如：我们想要获取参数更新后的值，那么我们可以这么组织我们的代码。

opt = tf.train.Optimizer().minize(loss)

with tf.control_dependencies([opt]):
  updated_weight = tf.identity(weight)

with tf.Session() as sess:
  tf.global_variables_initializer().run()
  sess.run(updated_weight, feed_dict={...}) # 这样每次得到的都是更新后的weight

关于tf.control_dependencies的具体用法，請移步官网https://www.tensorflow.org/api_docs/python/tf/Graph#control_dependencies,总结一句话就是，在执行某些op,tensor之前，某些op,tensor得首先被运行。

下面说明两种 control_dependencies 不 work 的情况

下面有两种情况，control_dependencies不work，其实并不是它真的不work，而是我们的使用方法有问题。

第一种情况:

import tensorflow as tf
w = tf.Variable(1.0)
ema = tf.train.ExponentialMovingAverage(0.9)
update = tf.assign_add(w, 1.0)

ema_op = ema.apply([update])
with tf.control_dependencies([ema_op]):
    ema_val = ema.average(update)

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    for i in range(3):
        print(sess.run([ema_val]))

 

也许你会觉得，在我们 sess.run([ema_val])， ema_op 都会被先执行，然后再计算ema_val，实际情况并不是这样，为什么？
有兴趣的可以看一下源码，就会发现 ema.average(update) 不是一个 op，它只是从ema对象的一个字典中取出键对应的 tensor 而已，然后赋值给ema_val。这个 tensor是由一个在 tf.control_dependencies([ema_op]) 外部的一个 op 计算得来的，所以 control_dependencies会失效。解决方法也很简单，看代码：

import tensorflow as tf
w = tf.Variable(1.0)
ema = tf.train.ExponentialMovingAverage(0.9)
update = tf.assign_add(w, 1.0)

ema_op = ema.apply([update])
with tf.control_dependencies([ema_op]):
    ema_val = tf.identity(ema.average(update)) #一个identity搞定

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    for i in range(3):
        print(sess.run([ema_val]))

 

第二种情况： 这个情况一般不会碰到，这是我在测试 control_dependencies 发现的

import tensorflow as tf
w = tf.Variable(1.0)
ema = tf.train.ExponentialMovingAverage(0.9)
update = tf.assign_add(w, 1.0)

ema_op = ema.apply([update])
with tf.control_dependencies([ema_op]):
    w1 = tf.Variable(2.0)
    ema_val = ema.average(update)

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    for i in range(3):
        print(sess.run([ema_val, w1]))

这种情况下，control_dependencies也不 work。读取 w1 的值并不会触发 ema_op， 原因请看代码：

#这段代码出现在Variable类定义文件中第287行，
# 在创建Varible时，tensorflow是移除了dependencies了的
#所以会出现 control 不住的情况
with ops.control_dependencies(None):