SGD平行算法 - Downpour SGD （单机python多线程版）

SGD 被广泛运用到机器学习(machine learning)中最优化等问题中，学术界一直热衷于提升SGD在优化问题中的收敛速率，并行计算也是热点研究的方向(包括Hogwild! [1], Delay-tolerant Algorithm for SGD [2], Elastic Average SGD [3])，本篇实现了现在比较火的Downpour SGD [4]的样例代码 (选择这个的原因引用量最大)。

理论思想

核心思想，将数据随机划分成数个子数据集sub-data, 将模型变量划分数个机器/进程/线程，基于子集合数据更新各个机器/进程/线程内的变量n次，然后到master 节点更新模型变量，各个机器/进程/线程训练独立互不干扰, 而且各个机器/进程/线程内的模型变量在训练中也互不干扰。引用原文原话 [4]:

We divide the training data into a number of subsets and run a copy of the model on
each of these subsets. The models communicate updates through a centralized parameter server,
which keeps the current state of all parameters for the model, sharded across many machines (e.g.,
if we have 10 parameter server shards, each shard is responsible for storing and applying updates
to 1/10th of the model parameters) (Figure 2). This approach is asynchronous in two distinct aspects:
the model replicas run independently of each other, and the parameter server shards also run
independently of one another

代码部分

# -*- encoding: utf-8 -*-
import re
import sys
import numpy as np
import copy
import time
import threading

def timeConsumption(func):