Python2.7.x multiprocessing多进程/多线程,批量转换文件, 从gbk到utf-8编码.

本文旨在通过一个批量转换文件编码(从gbk到utf-8编码)的案例, 来实战multiprocessing模块的多进程和多线程, 通过分析和测试, 来论证如何通过该模块提升性能.

在较早的文章中, 我曾用threadpool模块, 实现多线程版本的批量转换. 但在测试后发现, 这个模块较老, 且通过增加线程池数量的大小, 性能并没有实质的提高. 在一些调查之后. 发现了python支持的多进程和多线程模块中, 如何选用进程池或者线程池, 是有一定的场景前提的.

简要的说, 多进程和进程池大多用于有Cpu密集运算需求的应用中, 而多线程和线程池多用于I/O密集型运算的需求.

IO密集型: 网络请求, 网络传输, 网络通信, 以及tcp/udp类型的服务器. 大文件的存储读写. 外部存储(缓存)等读写
CPU密集型: 文件编码转换, 数据压缩解压缩, 数据加密和解密等等.

由此看出, 编码转换应该采用多进程/进程池来提升性能.

使用multiprocessing模块需要了解几个概念

1. import模块

   import multiprocessing  ##多进程模块
   import multiprocessing.dummy  ##多线程模块
   

2. 如何获取线程名或者进程名

       processName = multiprocessing.current_process().name
       threadName = multiprocessing.dummy.current_process().name
   

3. 进程池或者线程池, 如何传参数给 worker 进程或者线程.
   单参数

   # 定义worker
   def handler(arg):
   	print arg
   	pass
       
   arg = "入参"
   pool.map(handler,  [arg])
   

   多参数, 传入一个数组, 数组元素为元组对象.

   # 定义worker
   def convert(filename, seq, out_enc="UTF-8"):
   	print filename
   	print seq
   	
   def handler_args(args):
   	(path, seq) = args
   		return convert(path, seq)
   
   func_var = []
   func_var.append((path, seq))	    
   pool.map(handler_args, func_var)
   

4. 测试性能
   210个文件检测并转换,
   单进程模式需要2.28秒,
   多进程模式, 进程池设置为4, 只需要1.24秒. 由于测试样本规模不大. 所以4个进程就可以达到最大性能. 更多的进程反而会由于进程之间的上下文切换, 拖慢性能.
   因为转换得worker,在0.1秒内完成转换, 速度太快. 可以模拟worker用更多的时间完成, 加入sleep(1), 强制使得每个worker大约1秒-2秒完成.
   210个文件,
   单进程需要240秒+,
   多进程模式, 100个进程池大小, 仅需要2.4秒+, 性能提升了一百倍.

   由此看出, 当一个worker所需时间较多时(CPU密集型), 进程数量对性能提升的很显著, 如果一个worker完成在一秒内, 那么进程数量就无需太多, 设置2-4即可.

了解完以上几个要点, 下面就是完整的源码

#!/usr/bin/python
# -*-coding:utf-8-*-

import os
import sys
import codecs
import multiprocessing.dummy
from multiprocessing import Pool as ProcessPool
from multiprocessing.dummy import Pool as ThreadPool
from time import sleep, time
import chardet
from random import randrange
import itertools
import logging

def handler(sec, seq):
    threadName = multiprocessing.dummy.current_process().name
    s = '%s.%s, now I will sleep %s seconds' % (seq, threadName, sec)
    sleep(sec)
    print s

    return threadName


def handler_args(args):
    """Convert `f([1,2])` to `f(1,2)` call."""
    # return handler(*args)
    (path, seq) = args
    return convert(path, seq)


def get_pool(b_dummy=True, num=4):
    """
    if b_dummy is True then get ThreadPool, or get process pool
    :param b_dummy: dummy thread Pool or Process pool
    :param num: thread or process num
    :return: pool object
    """
    if b_dummy:
        pool = ThreadPool(num)
    else:
        pool = ProcessPool(num)

    return pool

def convert(filename, seq, out_enc="UTF-8"):
    start_time = time()
    # processName = multiprocessing.current_process().name
    # threadName = multiprocessing.dummy.current_process().name
    # threadName = threading.currentThread().getName()
    str = ""
    source_encoding = ""
    try:

        fpath, fname = os.path.split(filename)
        # codecs.open()这个方法打开的文件读取返回的将是unicode。
        content = codecs.open(filename, 'r').read()
        # chardet.detect 获取编码格式
        source_encoding = chardet.detect(content)['encoding']
        if None != source_encoding:  # 空的文件,返回None
            s = ""
            if source_encoding != 'utf-8':
                content = content.decode(source_encoding).encode(out_enc)
                codecs.open(filename, 'w').write(content)
                s = "Convert to utf-8 done! %s" % (fname)
            else:
                s = "Doesn't do anything. %s" % (fname)

            str = "%s=%s \n\t%s" % (source_encoding, filename, s)

        else:
            str = "None. %s " % (filename)

    except IOError as err:
        str = "I/O error:{0}".format(err)
    # sleep(1)
    logging.info("seq=%s, consume=%s seconds \n\t%s", seq, time() - start_time, str)
    return seq


def explore(dir, source_encoding_list):
    start_time = time()
    seq = 0
    for root, dirs, files in os.walk(dir, topdown=True):
        for file in files:
            # os.path.splitext(path)  分割路径，返回路径名和文件扩展名的元组
            if os.path.splitext(file)[1] in source_encoding_list:
                path = os.path.join(root, file)
                convert(path, seq)
                seq = seq + 1
    logging.info('time consume %s seconds', time() - start_time)


def exploreThread(dir, source_encoding_list):
    start_time = time()
    global pool
    pool = get_pool(b_dummy=False, num=4)
    func_var = []
    seq = 0
    for root, dirs, files in os.walk(dir, topdown=True):
        for file in files:
            # os.path.splitext(path)  分割路径，返回路径名和文件扩展名的元组
            if os.path.splitext(file)[1] in source_encoding_list:
                path = os.path.join(root, file)
                # convert(path)
                # func_var.append(randrange(3, 10))
                func_var.append((path, seq))
                seq = seq + 1

    # results = pool.map(handler_args, itertools.izip(func_var1, func_var2))
    results = pool.map(handler_args, func_var)
    pool.close()
    pool.join()
    logging.info('time consume %s seconds', time() - start_time)


def main():
    # srcPath是待转换的GBK文件的所在目录的路径, 可根据需要自行修改.
    srcPath = "/Users/[username]/Desktop/gbk/"
    # explore(srcPath, ['.xml', '.java', '.js', '.txt', '.css', '.jsp', '.html', '.htm', '.tpl'])
    exploreThread(srcPath, ['.xml', '.java', '.js', '.txt', '.css', '.jsp', '.html', '.htm', '.tpl'])

logging.basicConfig(level=logging.INFO,
                    format='%(levelname)s %(asctime)s %(processName)s %(threadName)s %(message)s',
                    datefmt='%Y-%m-%d %I:%M:%S')

if __name__ == "__main__":
    main()