pythondict转jsonu,如何以多线程方式将Python Dict转换为JSON

I have a number of large files with many thousands of lines in python dict format. I'm converting them with json.dumps to json strings.

import json

import ast

mydict = open('input', 'r')

output = open('output.json', "a")

for line in mydict:

line = ast.literal_eval(line)

line = json.dumps(line)

output.write(line)

output.write("\n")

This works flawlessly, however, it does so in a single threaded fashion. Is there an easy way to utilize the remaining cores in my system to speed things up?

Edit:

Based on the suggestions I've started here with the multiprocessing library:

import os

import json

import ast

from multiprocessing import Process, Pool

mydict = open('twosec.in', 'r')

def info(title):

print title

print 'module name:', __name__

print 'parent process: ', os.getppid()

print 'process id:', os.getpid()

def converter(name):

info('converter function')

output = open('twosec.out', "a")

for line in mydict:

line = ast.literal_eval(line)

line = json.dumps(line)

output.write(line)

output.write("\n")

if __name__ == '__main__':

info('main line')

p = Process(target=converter, args=(mydict))

p.start()

p.join()

I don't quite understand where Pool comes into play, can you explain more?

解决方案

Wrap the code above in a function that takes as its single argument a filename and that writes the json to an output file.

Then create a Pool object from the multiprocessing module, and use Pool.map() to apply your function in parallel to the list of all files. This will automagically use all cores on your CPU, and because it uses multiple processes instead of threads, you won't run into the global interpreter lock.

Edit: Change the main portion of your program like so;

if __name__ == '__main__':

files = ['first.in', 'second.in', 'third.in'] # et cetera

info('main line')

p = Pool()

p.map(convertor, files)

p.close()

Of course you should also change convertor() to derive the output name from the input name!

Below is a complete example of a program to convert DICOM files into PNG format, using the ImageMagick program

"Convert DICOM files to PNG format, remove blank areas."

import os

import sys # voor argv.

import subprocess

from multiprocessing import Pool, Lock

def checkfor(args):

try:

subprocess.check_output(args, stderr=subprocess.STDOUT)

except CalledProcessError:

print "Required program '{}' not found! exiting.".format(progname)

sys.exit(1)

def processfile(fname):

size = '1574x2048'

args = ['convert', fname, '-units', 'PixelsPerInch', '-density', '300',

'-crop', size+'+232+0', '-page', size+'+0+0', fname+'.png']

rv = subprocess.call(args)

globallock.acquire()

if rv != 0:

print "Error '{}' when processing file '{}'.".format(rv, fname)

else:

print "File '{}' processed.".format(fname)

globallock.release()

## This is the main program ##

if __name__ == '__main__':

if len(sys.argv) == 1:

path, binary = os.path.split(sys.argv[0])

print "Usage: {} [file ...]".format(binary)

sys.exit(0)

checkfor('convert')

globallock = Lock()

p = Pool()

p.map(processfile, sys.argv[1:])

p.close()