实验任务:
集群上多节点多GPU分布式训练
CUDA_VISIBLE_DEVICES=” python distributed.py –job_name=ps –task_index=0
CUDA_VISIBLE_DEVICES=’0’ python distributed.py –job_name=worker –task_index=0
CUDA_VISIBLE_DEVICES=’1’ python distributed.py –job_name=worker –task_index=1
CUDA_VISIBLE_DEVICES=’0’ python distributed.py –job_name=worker –task_index=2
CUDA_VISIBLE_DEVICES=’1’ python distributed.py –job_name=worker –task_index=3
实验环境:
TeslaK20c集群,使用了3个节点,其中1个节点使用1个cpu作为参数服务器,2个节点分别使用2个gpu作为工作服务器,分布式训练方式可以选择同步和异步两种。
# encoding:utf-8
import math
import tempfile
import time
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
flags = tf.app.flags
flags.DEFINE_string('data_dir', '/home/zhangzhaoyu/incubator-mxnet-master/example/image-classification/data', 'Directory for storing mnist data')
flags.DEFINE_integer('hidden_units', 100, 'Number of units in the hidden layer of the NN')
flags.DEFINE_integer('train_steps', 100000, 'Number of training steps to perform')
flags.DEFINE_integer('batch_size', 100, 'Training batch size ')
flags.DEFINE_float('learning_rate', 0.01, 'Learning rate')
flags.DEFINE_string('ps_hosts', '172.16.1.182:2222', 'Comma-separated list of hostname:port pairs')
flags.DEFINE_string('worker_hosts', '172.16.1.183:2223,172.16.1.183:2224,172.16.1.187:2225,172.16.1.187:2226',
'Comma-separated list of hostname:p