python mpi4py 读取json_用mpi4py实现分布式学习

mpi_learn

Distributed learning with mpi4py

Dependencies: OpenMPI and mpi4py (v. >= 2.0.0), keras (v. >= 1.2.0)

Test with the MNIST dataset:

git clone https://github.com/duanders/mpi_learn.git

cd mpi_learn

python BuildModel.py mnist

python models/get_mnist.py

mpirun -np 3 ./MPIDriver.py mnist_arch.json train_mnist.list test_mnist.list --loss categorical_crossentropy --epochs 3

Using MPIDriver.py to train your model

MPIDriver.py will load a keras model of your choice and train it on the input data you provide. The script has three required arguments:

Path to JSON file specifying the Keras model (your model can be converted to JSON using the model's to_json() method)

File containing a list of training data. This should be a simple text file with one input data file per line. By default the script expects data stored in HDF5 format; see below for instructions for handling arbitrary input data.

File containing a list of validation data. This should be a simple text file with one input data file per line.

See MPIDriver.py for supported optional arguments. Run the script via mpirun or mpiexec. It should automatically detect available NVIDIA GPUs