转载请注明作者和出处: http://blog.csdn.net/john_bh/
文章目录
1. TensorFlow 使用GPU
1.1 指定GPU
如果安装的是GPU版本,在运行的过程中TensorFlow能够自动检测。如果检测到GPU,TensorFlow会尽可能的利用找到的第一个GPU来执行操作。
如果机器上有超过一个可用的GPU,除了第一个之外的其他的GPU默认是不参与计算的。为了让TensorFlow使用这些GPU,必须将OP明确指派给他们执行。with…device语句能够用来指派特定的CPU或者GPU执行操作:
import tensorflow as tf
import numpy as np
with tf.Session() as sess:
with tf.device('/cpu:0'):
a = tf.placeholder(tf.int32)
b = tf.placeholder(tf.int32)
add = tf.add(a, b)
sum = sess.run(add, feed_dict={a: 3, b: 4})
print(sum)
cpu:0 机器的第一个cpu。
gpu:0 机器的第一个gpu,如果有的话
gpu:1 机器的第二个gpu,依次类推
另外指定GPU也可以这样设置:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" # 指定gpu编号,从0开始
使用 tf.ConfigProto
来构建一个config,在config中指定相关的GPU,并且在session中传入参数config=“自己创建的config”
来指定gpu操作:
import tensorflow as tf
import numpy as np
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0" # 指定gpu编号,从0开始
# log_device_placement=True: 是否打印设备分配日志
# allow_soft_placement=True: 如果指定的设备不存在,允许TF自动分配设备
config = tf.ConfigProto(log_device_placement=True, allow_soft_placement=True)
with tf.Session(config=config) as sess:
a = tf.placeholder(tf.int32)
b = tf.placeholder(tf.int32)
add = tf.add(a, b)
sum = sess.run(add, feed_dict={a: 3, b: 4})
print(sum)
1.2 设置GPU使用资源
tf.ConfigProto
函数生成的config之后,还可以设置其属性来分配GPU的运算资源,缓解TensorFlow一上来就占满整个GPU导致的问题。在创建sess的时候传入附加的参数,更好的控制GPU的使用方法是,如下代码就是按需分配:
gpu_options = tf.GPUOptions(allow_growth=True)# allow_grouth=True:按需分配
config = tf.ConfigProto(gpu_options = gpu_options,allow_soft_placement=True,allow_grouth=True)
sess = tf.Session(config=config)
gpu分配固定大小的计算资源:
gpu_options= tf.GPUOptions(allow_growth=True, per_process_gpu_memory_fraction=0.9) #占用90%显存
config = tf.ConfigProto(gpu_options = gpu_options,allow_soft_placement=True,allow_grouth=True)
sess = tf.Session(config=config)
1.3 MNIST 例子
以MNIST为例,实现单个GPU训练模型,具体代码如下:
import tensorflow as tf
import tensorflow.contrib.slim as slim
import numpy as np
import os
import time
os.environ["CUDA_VISIBLE_DEVICES"] = "0" # 指定gpu编号,从0开始
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
# mnist=input_data.read_data_sets("/tmp/mnist/",one_hot=True)
MNIST_data=r"/tmp/MNIST_data"
mnist = input_data.read_data_sets(MNIST_data, False, one_hot=True)
# print(mnist)
#Parameters
num_steps=200
max_epoch=15
learning_rate=0.001
batch_size=1024
display_step=10
mode_dir="./models/MNIST_test"
pretrained_model=None
# pretrained_model="./models/MNIST_test/model.ckpt-10"
# pretrained_model="./models/MNIST_test"
log_dir = './tensorboard'
#Network Parameters
num_input=784 # MNIST data input (img shape: 28*28)
num_classes=10 # MNIST total classes (0-9 digits)
dropout=0.75 # Dropout, probability to keep units
def conv_net(x,is_training):
batch_norm_params={"is_training":is_training,"decay":0.9,"updates_collections":None}
with slim.arg_scope([slim.conv2d,slim.fully_connected],activation_fn=tf.nn.relu,weights_initializer=tf.truncated_normal_initializer(mean=0.0,stddev=0.01),weights_regularizer=slim.l2_regularizer(0.0005),normalizer_fn=slim.batch_norm,normalizer_params=batch_norm_params):
with tf.variable_scope("ConvNet",reuse=tf.AUTO_REUSE):
x = tf.reshape(x, [-1, 28, 28, 1])
net = slim.conv2d(x, 6, [5, 5], scope="conv_1")
net = slim.max_pool2d(net, [2, 2], scope="pool_1")
net = slim.conv2d(net, 12, [5, 5], scope="conv_2")
net = slim.max_pool2d(net, [2, 2], scope="pool_2")
net = slim.flatten(net, scope="flatten")
net = slim.fully_connected(net, 100, scope="fc")
net = slim.dropout(net, is_training=is_training)
net = slim.fully_connected(net, num_classes, scope="prob", activation_fn=None, normalizer_fn=None)
return net
def train_single():
# 0. 准备数据
with tf.Graph().as_default():
epoch_size=int(mnist.train.num_examples / batch_size)
# 1.使用占位符填充数据
X=tf.placeholder(tf.float32,[None,num_input])
Y=tf.placeholder(tf.float32,[None,num_classes])
# 2.定义并使用网络模型
logits=conv_net(X,True)
# 3.计算loss,使用优化器优化
loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=Y,logits=logits))
opt=tf.train.AdamOptimizer(learning_rate)
train_op=opt.minimize(loss)
# 4.测试,计算 accruacy
logits_test=conv_net(X,False)
correct_prediction=tf.equal(tf.arg_max(logits_test,1),tf.argmax(Y,1)) #计算预测值和真实值
accuracy=tf.reduce_mean(tf.cast(correct_prediction,tf.float32)) #计算准确率
# 保存TensorBoard
train_loss=tf.Variable(tf.constant(0.0),dtype=tf.float32,name='train_loss')
test_acc=tf.Variable(tf.constant(0.0),dtype=tf.float32,name='test_acc')
tf.summary.scalar("train_loss",train_loss)
tf.summary.scalar('test_acc',test_acc)
# 5. 保存模型
if not os.path.exists(mode_dir):
os.mkdir(mode_dir)
global_step=tf.Variable(0,name='global_step',trainable=False) #计数器变量,设置trainable=False,不需要被训练
save_params=tf.trainable_variables()
saver=tf.train.Saver(save_params,max_to_keep=None)
# 6.开启Session
# 6.1 配置tensorflow
gpu_options=tf.GPUOptions(per_process_gpu_memory_fraction=1.0) #占满资源
config=tf.ConfigProto(gpu_options = gpu_options,allow_soft_placement=False, log_device_placement=False)
sess=tf.Session(config=config) #创建会话
## 6.2 初始化操作
sess.run(tf.global_variables_initializer())
sess.run(tf.local_variables_initializer())
with sess.as_default():
epoch_start = 0
# 检查是否存在预训练模型
if pretrained_model:
if (not os.path.isdir(pretrained_model)): #从指定的模型中恢复
print('Restoring pretrained model: {}'.format(pretrained_model))
saver.restore(sess,pretrained_model)
else: #给定一个文件夹,从最新的模型中恢复
print('Model directory: {}'.format(pretrained_model))
ckpt=tf.train.get_checkpoint_state(pretrained_model)
# 恢复模型
model_path=ckpt.model_checkpoint_path
# print('Checkpoint file: {}'.format(model_path))
assert(ckpt and model_path)
epoch_start = int(model_path[model_path.find('model.ckpt-') + 11:]) + 1 #更新编号
saver.restore(sess,model_path)
# 开始训练
print('############## Running train.##############')
merged=tf.summary.merge_all()
train_write=tf.summary.FileWriter(log_dir,sess.graph)
start=time.time()
for epoch in range(epoch_start,max_epoch):
loss_value=0
for step in range(epoch_size):
batch_x,batch_y=mnist.train.next_batch(batch_size)
_,loss_value,acc = sess.run([train_op,loss,accuracy],feed_dict={X:batch_x,Y:batch_y})
#每隔 display_step 步 打印一次损失值和验证集的准确率
if (step+1) % display_step == 0 or (step+1)==epoch_size:
print("Epoch:[{}] [{} / {} ] loos:{}, acc:{}".format(epoch,step+1,epoch_size,str(loss_value),str(acc)))
# 保存模型
# print("############## save ckpt file ##############")
checkpoint_path=os.path.join(mode_dir,'model.ckpt')
metagraph_path=os.path.join(mode_dir,'model.meta')
saver.save(sess,checkpoint_path,global_step=epoch,write_meta_graph=False) #write_meta_graph=False 不用每次都保存meta_graph
if not os.path.exists(metagraph_path):
saver.export_meta_graph(metagraph_path)
# 设置验证集
print("############## test this epoch ##############")
acc=np.mean([sess.run(accuracy,feed_dict={X:mnist.test.images[i:i+batch_size],Y:mnist.test.labels[i:i+batch_size]}) for i in range (0,len(mnist.test.images),batch_size)])
print("Epoch-{}, TestData Accuracy: {} \n".format(epoch,str(acc)))
# tensorboard
summary,_,_=sess.run([merged,train_loss.assign(loss_value),test_acc.assign(acc)])
train_write.add_summary(summary,epoch)
print("Cost Time: {}".format(time.time()-start))
print("Done")
if __name__=="__main__":
train_single()
输出结果:
############## Running train.##############
Epoch:[0] [10 / 53 ] loos:1.791963, acc:0.10449219
Epoch:[0] [20 / 53 ] loos:1.3828717, acc:0.115234375
Epoch:[0] [30 / 53 ] loos:1.012688, acc:0.123046875
Epoch:[0] [40 / 53 ] loos:0.74021804, acc:0.11328125
Epoch:[0] [50 / 53 ] loos:0.5501305, acc:0.109375
Epoch:[0] [53 / 53 ] loos:0.50863796, acc:0.13476562
############## test this epoch ##############
Epoch-0, TestData Accuracy: 0.15481704
Epoch:[1] [10 / 53 ] loos:0.40868193, acc:0.8339844
Epoch:[1] [20 / 53 ] loos:0.33527768, acc:0.9550781
Epoch:[1] [30 / 53 ] loos:0.26853693, acc:0.97558594
Epoch:[1] [40 / 53 ] loos:0.24702275, acc:0.97265625
Epoch:[1] [50 / 53 ] loos:0.21551938, acc:0.9794922
Epoch:[1] [53 / 53 ] loos:0.19377959, acc:0.9892578
############## test this epoch ##############
Epoch-1, TestData Accuracy: 0.9835579
Epoch:[2] [10 / 53 ] loos:0.18332487, acc:0.9814453
Epoch:[2] [20 / 53 ] loos:0.16378397, acc:0.98535156
Epoch:[2] [30 / 53 ] loos:0.16006559, acc:0.98828125
Epoch:[2] [40 / 53 ] loos:0.1388208, acc:0.9824219
Epoch:[2] [50 / 53 ] loos:0.14397822, acc:0.98046875
Epoch:[2] [53 / 53 ] loos:0.14302078, acc:0.984375
############## test this epoch ##############
Epoch-2, TestData Accuracy: 0.98285234
Epoch:[3] [10 / 53 ] loos:0.12545954, acc:0.98535156
Epoch:[3] [20 / 53 ] loos:0.12351326, acc:0.9785156
Epoch:[3] [30 / 53 ] loos:0.12902205, acc:0.98535156
Epoch:[3] [40 / 53 ] loos:0.10811641, acc:0.9902344
Epoch:[3] [50 / 53 ] loos:0.10564952, acc:0.9824219
Epoch:[3] [53 / 53 ] loos:0.10889536, acc:0.98339844
############## test this epoch ##############
Epoch-3, TestData Accuracy: 0.9875916
Epoch:[4] [10 / 53 ] loos:0.0850852, acc:0.99316406
Epoch:[4] [20 / 53 ] loos:0.10087522, acc:0.98339844
Epoch:[4] [30 / 53 ] loos:0.083879426, acc:0.99121094
Epoch:[4] [40 / 53 ] loos:0.076213315, acc:0.99609375
Epoch:[4] [50 / 53 ] loos:0.09560504, acc:0.984375
Epoch:[4] [53 / 53 ] loos:0.08533226, acc:0.9863281
############## test this epoch ##############
Epoch-4, TestData Accuracy: 0.9868104
Cost Time: 7.931570291519165
Done
2. TensorFlow实现多GPU并行
2.1 数据并行
数据并行:模型只有一个,每个GPU运行不同的batch,并行计算梯度
。根据参数更新方式又分为 同步数据并行 和 异步数据并行。其中数据并行比较常用。
训练步骤:
- 在GPU1和GPU2上分别定义模型网络结构;
- 对于单个GPU,分别从数据管道中读取不同的数据块,然后进行向前传播,计算出损失,在计算当前变量的梯度;
- 把所有GPU输出的梯度数据转移到CPU上,先进行梯度求平均操作,然后进行模型变量的更新;
- 重复 1~3 步骤,知道模型变量收敛位置。
数据并行主要提高SGD的效率,例如,加入每次SGD的mini-batch 大小是1000个,那么如果切分10份,每份100个,然后将模型复制10份,就可以在10个模型上同时计算。
-
同步数据并行
:把数据分给不同的卡,等所有的GPU都计算完梯度后进行平均,最后再更新梯度。收敛速度快,最终训练效果优于异步数据并行方式,但是速度略慢,因为速度取决于性能最差的那个GPU。 如下图所示:优点:
- 每个训练批次考虑了所有工作节点的训练情况,损失下降比较稳定;
缺点:
- 在最慢的工作节点上存在性能瓶颈,尤其在不同的设备中,工作节点性能常常不同,这个劣势非常明显。
命名空间和变量空间:
变量的定义是在tf.variable_scope()
下,求解梯度过程是在tf.name_scope()
下。tf.variable_scope()
下相同的scope_name
可以让变量有相同的命名,包括tf.get_variable()
得到的变量,还有tf.Variable()
的变量,不加tf.get_variable_scope().reuse_variables()
的话就不能重用。 所以不同GPU用了不同的tf.name_scope()
,全部GPU在同一个tf.variable_scope()
下。
import tensorflow as tf
import tensorflow.contrib.slim as slim
import numpy as np
import time
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" # 指定gpu编号,从0开始
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
# mnist=input_data.read_data_sets("/tmp/mnist/",one_hot=True)
MNIST_data=r"/tmp/MNIST_data"
mnist = input_data.read_data_sets(MNIST_data, False, one_hot=True)
def get_avilable_gpus():
"""
code from http://stackoverflow.com/questions/38559755/how-to-get-current-available-gpus-in-tensorflow
"""
from tensorflow.python.client import device_lib as _device_lib
local_device_protos = _device_lib.list_local_devices()
return [x.name for x in local_device_protos if x.device_type=="GPU"]
num_gpus=len(get_avilable_gpus())
print("Available GPU Number: "+str(num_gpus))
#Parameters
num_gpus=2
num_steps=200
max_epoch=5
learning_rate=0.001
batch_size=1024
display_step=10
log_dir = './tensorboard'
mode_dir="./models/MNIST_test"
pretrained_model=None
# pretrained_model="./models/MNIST_test/model.ckpt-14"
# pretrained_model="./models/MNIST_test"
#Network Parameters
num_input=784 # MNIST data input (img shape: 28*28)
num_classes=10 # MNIST total classes (0-9 digits)
dropout=0.75 # Dropout, probability to keep units
def conv_net(x,is_training):
batch_norm_params={"is_training":is_training,"decay":0.9,"updates_collections":None}
with slim.arg_scope([slim.conv2d,slim.fully_connected],activation_fn=tf.nn.relu,weights_initializer=tf.truncated_normal_initializer(mean=0.0,stddev=0.01),weights_regularizer=slim.l2_regularizer(0.0005),normalizer_fn=slim.batch_norm,normalizer_params=batch_norm_params):
with tf.variable_scope("ConvNet",reuse=tf.AUTO_REUSE):
x = tf.reshape(x, [-1, 28, 28, 1])
net = slim.conv2d(x, 6, [5, 5], scope="conv_1")
net = slim.max_pool2d(net, [2, 2], scope="pool_1")
net = slim.conv2d(net, 12, [5, 5], scope="conv_2")
net = slim.max_pool2d(net, [2, 2], scope="pool_2")
net = slim.flatten(net, scope="flatten")
net = slim.fully_connected(net, 100, scope="fc")
net = slim.dropout(net, is_training=is_training)
net = slim.fully_connected(net, num_classes, scope="prob", activation_fn=None, normalizer_fn=None)
return net
def average_gradients(tower_grads):
"""
计算平均梯度函数
:param tower_grads:
:return:
"""
average_grads=[]
for grad_and_vars in zip(*tower_grads):
grads=[]
for g,_ in grad_and_vars:
expend_g=tf.expand_dims(g,0)
grads.append(expend_g)
grad= tf.concat(grads,0)
grad=tf.reduce_mean(grad,0)
v=grad_and_vars[0][1]
grad_and_vars=(grad,v)
average_grads.append(grad_and_vars)
return average_grads
PS_OPS=['Variable','VariableV2','AutoReloadVariable']
def assign_to_device(device,ps_device='/cpu:0'):
def _assign(op):
node_def=op if isinstance(op,tf.NodeDef) else op.node_def
if node_def.op in PS_OPS:
return '/'+ps_device
else:
return device
return _assign
def train_multi_GPU():
with tf.Graph().as_default(),tf.device("/cpu:0"):
epoch_size = int(mnist.train.num_examples / batch_size)
global_step=tf.train.get_or_create_global_step()
tower_grads=[] #定义全局的梯度的list
# 1.使用占位符填充数据
X=tf.placeholder(tf.float32,[None,num_input])
Y=tf.placeholder(tf.float32,[None,num_classes])
opt=tf.train.AdamOptimizer(learning_rate)
# 保存TensorBoard
# train_loss = tf.Variable(tf.constant(0.0), dtype=tf.float32, name='train_loss')
# test_acc = tf.Variable(tf.constant(0.0), dtype=tf.float32, name='test_acc')
# tf.summary.scalar("train_loss", train_loss)
# tf.summary.scalar('test_acc', test_acc)
# 变量不要使用tf.Variable()得到,这个需要配合tf.get_variable_scope().reuse_variables()才能使得变量在多GPU之间重用
with tf.variable_scope(tf.get_variable_scope()):
for i in range(num_gpus):
with tf.device(assign_to_device('/gpu:{}'.format(i),ps_device='/cpu:0')):
_x = X[i*batch_size:(i+1)*batch_size]
_y = Y[i*batch_size:(i+1)*batch_size]
# 2.定义并使用网络模型
logits=conv_net(_x,True)
# 3.计算loss,使用优化器优化
loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=_y,logits=logits))
# tf.summary.scalar('loss_{}'.format(i), loss) # tensorboard显示各GPU的loss
tf.get_variable_scope().reuse_variables() # 重用变量
grads=opt.compute_gradients(loss) # # 计算梯度
tower_grads.append(grads)
if i==0:
logits_test = conv_net(_x, False)
correct_prediction = tf.equal(tf.argmax(logits_test, 1), tf.argmax(_y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
grads=average_gradients(tower_grads) # 计算平均梯度
train_op=opt.apply_gradients(grads)
# 5. 保存模型
if not os.path.exists(mode_dir):
os.mkdir(mode_dir)
global_step = tf.Variable(0, name='global_step', trainable=False) # 计数器变量,设置trainable=False,不需要被训练
save_params = tf.trainable_variables()
saver = tf.train.Saver(save_params, max_to_keep=None)
# 6.开启Session
# 6.1 配置tensorflow
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=1.0) # 占满资源
config = tf.ConfigProto(gpu_options=gpu_options, allow_soft_placement=False, log_device_placement=False)
sess = tf.Session(config=config) # 创建会话
## 6.2 初始化操作
sess.run(tf.global_variables_initializer())
sess.run(tf.local_variables_initializer())
with sess.as_default():
epoch_start=0
# 检查是否存在预训练模型
if pretrained_model:
if (not os.path.isdir(pretrained_model)): # 从指定的模型中恢复
print('Restoring pretrained model: {}'.format(pretrained_model))
saver.restore(sess, pretrained_model)
else: # 给定一个文件夹,从最新的模型中恢复
print('Model directory: {}'.format(pretrained_model))
ckpt = tf.train.get_checkpoint_state(pretrained_model)
# 恢复模型
model_path = ckpt.model_checkpoint_path
# print('Checkpoint file: {}'.format(model_path))
assert (ckpt and model_path)
epoch_start = int(model_path[model_path.find('model.ckpt-') + 11:]) + 1 # 更新编号
saver.restore(sess, model_path)
# 开始训练
print('############## Running train.##############')
merged = tf.summary.merge_all()
train_write = tf.summary.FileWriter(log_dir, sess.graph)
start = time.time()
for epoch in range(epoch_start,max_epoch):
loss_value = 0
for step in range(epoch_size):
batch_x,batch_y=mnist.train.next_batch(batch_size*num_gpus)
_,loss_value,acc=sess.run([train_op,loss,accuracy],feed_dict={X:batch_x,Y:batch_y})
# 每隔 display_step 步 打印一次损失值和验证集的准确率
if (step+1)% display_step==0 or (step+1) == epoch_size:
print("Epoch:[{}] [{} / {} ] loos:{}, acc:{}".format(epoch, step + 1, epoch_size, str(loss_value),str(acc)))
# loss_value,acc=sess.run([loss,accuracy],feed_dict={X:batch_x,Y:batch_y})
# print("Step: "+ str(step) + ": "+str(loss_value)+" " + str(acc)+", %i Examples/sec" % int(len(batch_x)/te))
# 保存模型
# print("############## save ckpt file ##############")
checkpoint_path = os.path.join(mode_dir, 'model.ckpt')
metagraph_path = os.path.join(mode_dir, 'model.meta')
saver.save(sess, checkpoint_path, global_step=epoch,write_meta_graph=False) # write_meta_graph=False 不用每次都保存meta_graph
if not os.path.exists(metagraph_path):
saver.export_meta_graph(metagraph_path)
# 设置验证集
print("############## test this epoch ##############")
acc = np.mean([sess.run(accuracy, feed_dict={X: mnist.test.images[i:i + batch_size],
Y: mnist.test.labels[i:i + batch_size]}) for i in
range(0, len(mnist.test.images), batch_size)])
print("Epoch-{}, TestData Accuracy: {} \n".format(epoch, str(acc)))
# # tensorboard
# summary, _, _ = sess.run([merged, train_loss.assign(loss_value), test_acc.assign(acc)])
# summary, _ = sess.run([merged, test_acc.assign(acc)])
# train_write.add_summary(summary, epoch)
print("Cost Time: {}".format(time.time() - start))
print("Done")
# print("Testing Accuracy:",np.mean([sess.run(accuracy,feed_dict={X:mnist.test.images[i:i+batch_size],Y:mnist.test.labels[i:i+batch_size]})for i in range(0,len(mnist.test.images),batch_size)]))
if __name__=="__main__":
train_multi_GPU()
Available GPU Number: 2
############## Running train.##############
2020-07-24 11:19:10.914410: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
Epoch:[0] [10 / 53 ] loos:1.7844108, acc:0.13085938
Epoch:[0] [20 / 53 ] loos:1.3553904, acc:0.11621094
Epoch:[0] [30 / 53 ] loos:1.0151663, acc:0.12207031
Epoch:[0] [40 / 53 ] loos:0.7468412, acc:0.9609375
Epoch:[0] [50 / 53 ] loos:0.5506376, acc:0.9667969
Epoch:[0] [53 / 53 ] loos:0.5286949, acc:0.9658203
############## test this epoch ##############
Epoch-0, TestData Accuracy: 0.9709004
Epoch:[1] [10 / 53 ] loos:0.40451363, acc:0.9794922
Epoch:[1] [20 / 53 ] loos:0.3416087, acc:0.96777344
Epoch:[1] [30 / 53 ] loos:0.2718013, acc:0.9785156
Epoch:[1] [40 / 53 ] loos:0.22224262, acc:0.9746094
Epoch:[1] [50 / 53 ] loos:0.20745455, acc:0.9873047
Epoch:[1] [53 / 53 ] loos:0.20323172, acc:0.98828125
############## test this epoch ##############
Epoch-1, TestData Accuracy: 0.98297197
Epoch:[2] [10 / 53 ] loos:0.17522728, acc:0.984375
Epoch:[2] [20 / 53 ] loos:0.16538306, acc:0.98046875
Epoch:[2] [30 / 53 ] loos:0.14809372, acc:0.9873047
Epoch:[2] [40 / 53 ] loos:0.14841683, acc:0.98828125
Epoch:[2] [50 / 53 ] loos:0.12726341, acc:0.984375
Epoch:[2] [53 / 53 ] loos:0.124505565, acc:0.9863281
############## test this epoch ##############
Epoch-2, TestData Accuracy: 0.98696786
Epoch:[3] [10 / 53 ] loos:0.119042374, acc:0.98828125
Epoch:[3] [20 / 53 ] loos:0.110665604, acc:0.9873047
Epoch:[3] [30 / 53 ] loos:0.10843476, acc:0.9892578
Epoch:[3] [40 / 53 ] loos:0.10401645, acc:0.9873047
Epoch:[3] [50 / 53 ] loos:0.07692264, acc:0.9902344
Epoch:[3] [53 / 53 ] loos:0.097311236, acc:0.9902344
############## test this epoch ##############
Epoch-3, TestData Accuracy: 0.9888533
Epoch:[4] [10 / 53 ] loos:0.09335728, acc:0.9863281
Epoch:[4] [20 / 53 ] loos:0.09267537, acc:0.99609375
Epoch:[4] [30 / 53 ] loos:0.0856861, acc:0.99316406
Epoch:[4] [40 / 53 ] loos:0.07281289, acc:0.9921875
Epoch:[4] [50 / 53 ] loos:0.08372472, acc:0.9873047
Epoch:[4] [53 / 53 ] loos:0.08419474, acc:0.9873047
############## test this epoch ##############
Epoch-4, TestData Accuracy: 0.98871773
Cost Time: 11.453099250793457
Done
- 异步数据并行:每个GPU计算完梯度后就各自更新,速度快,但是最终模型的性能略低于同步数据并行方式。
异步数据更新并行也称异步训练,异步更新,它是在每个工作节点的任务独立计算局部梯度,并异步地更新到模型的参数中,不需要执行协调和等待操作。如下图所示:优点:
- 性能不存在瓶颈; 缺点:
- 每个节点计算的梯度值发送回参数服务器会有参数更新冲突,一定程度上影响算法的收敛速度,在损失下降中抖动较大。
同步更新和异步更新小结: 同步更新和异步更新的实现区别主要在于更新参数服务器的策略。
在数据量小,各个节点计算能力比较均衡情况下,推荐使用同步更新;
在数据量很大,各个机器的计算性能参差不齐的情况下,推荐使用异步更新。
2.2 模型并行
模型并行:将不同的模型计算部分部署在不同的GPU上面同时执行。
模型并行是对模型进行切分,让模型的不同部分执行在不同的设备上,这样一个批次样本可以在不同的设备上同时执行。为了充分利用同一台设备的计算能力,TensorFlow会尽量让相邻的计算在同一台设备上完成来节省网络开销。如下图所示,一个LSTM模型,展示一个批次的样本在设备1,设备2,设备3上同时训练,分别执行模型的不同部分,分别训练出P1、P2、P3三个不同的参数。
3.存在的问题
3.1 Memory-Usage占满而GPU-Util为0%的问题
在GPU上使用开源代码训练模型时往往会遇到 明明Memory-Usage占满而GPU-Util为0%的问题,用nvidia-smi检查GPU运行情况,发现Memory-Usage占满而GPU-Util为0%。
使用 watch -n 0.1 nvidia-smi
实时监控gpu使用情况,发现大部分时间4个gpu的GPU-Util为0%,突然有一个时间4个gpu的GPU-Util为35%了。这说明多gpu训练部分的程序是成功运行的,只是大部分时间在cpu上运行,极少部分时间在gpu上运行。查看代码,做了数据增强操作,而这部分操作是在CPU上进行的,占用大部分时间,把数据增强部分代码去掉后,GPU-Util不再为0%了。
当发现GPU使用效率不高时:
- 要用watch nvidia-smi命令查看GPU运行状况;
- 检查模型训练代码,看是否存在数据增强操作或者其它需要在CPU上处理的操作,然后优化该部分代码,使得充分使用GPU资源。
参考文章: