Tensorflow--分布式训练

实验任务:

集群上多节点多GPU分布式训练

CUDA_VISIBLE_DEVICES=” python distributed.py –job_name=ps –task_index=0

CUDA_VISIBLE_DEVICES=’0’ python distributed.py –job_name=worker –task_index=0

CUDA_VISIBLE_DEVICES=’1’ python distributed.py –job_name=worker –task_index=1

CUDA_VISIBLE_DEVICES=’0’ python distributed.py –job_name=worker –task_index=2

CUDA_VISIBLE_DEVICES=’1’ python distributed.py –job_name=worker –task_index=3

实验环境:

TeslaK20c集群,使用了3个节点，其中1个节点使用1个cpu作为参数服务器,2个节点分别使用2个gpu作为工作服务器，分布式训练方式可以选择同步和异步两种。

# encoding:utf-8
import math
import tempfile
import time
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

flags = tf.app.flags

flags.DEFINE_string('data_dir', '/home/zhangzhaoyu/incubator-mxnet-master/example/image-classification/data', 'Directory  for storing mnist data')
flags.DEFINE_integer('hidden_units', 100, 'Number of units in the hidden layer of the NN')
flags.DEFINE_integer('train_steps', 100000, 'Number of training steps to perform')
flags.DEFINE_integer('batch_size', 100, 'Training batch size ')
flags.DEFINE_float('learning_rate', 0.01, 'Learning rate')

flags.DEFINE_string('ps_hosts', '172.16.1.182:2222', 'Comma-separated list of hostname:port pairs')

flags.DEFINE_string('worker_hosts', '172.16.1.183:2223,172.16.1.183:2224,172.16.1.187:2225,172.16.1.187:2226',
                    'Comma-separated list of hostname:p