python一次读取多个文件_python – 使用多处理读取多个文件

我需要读取一些非常大的文本文件(100 Mb),使用正则表达式处理每一行并将数据存储到结构中.我的结构继承自defaultdict,它有一个read(self)方法,读取self.file_name文件.

看看这个非常简单(但不是真实的)的例子,我没有使用正则表达式,但我正在拆分行：

import multiprocessing

from collections import defaultdict

def SingleContainer():

return list()

class Container(defaultdict):

"""

this class store odd line in self["odd"] and even line in self["even"].

It is stupid, but it's only an example. In the real case the class

has additional methods that do computation on readen data.

"""

def __init__(self,file_name):

if type(file_name) != str:

raise AttributeError, "%s is not a string" % file_name

defaultdict.__init__(self,SingleContainer)

self.file_name = file_name

self.readen_lines = 0

def read(self):

f = open(self.file_name)

print "start reading file %s" % self.file_name

for line in f:

self.readen_lines += 1

values = line.split()

key = {0: "even", 1: "odd"}[self.readen_lines %2]

self[key].append(values)

print "readen %d lines from file %s" % (self.readen_lines, self.file_name)

def do(file_name):

container = Container(file_name)

container.read()

return container.items()

if __name__ == "__main__":

file_names = ["r1_200909.log", "r1_200910.log"]

pool = multiprocessing.Pool(len(file_names))

result = pool.map(do,file_names)

pool.close()

pool.join()

print "Finish"

最后,我需要将每个结果加入一个Container中.重要的是保持线的顺序.返回值时,我的方法太慢了.好的解决方案

我在Linux上使用python 2.6