与a previous question相关,我正在尝试对大量大型CSV文件进行替换。在
列顺序(和内容)在文件之间会发生变化,但对于每个文件,我需要大约10个列,并且可以通过列标题名称进行标识。我也有1-2本字典,每个专栏我想要。所以对于我想要的列,我只想使用正确的字典,并希望按顺序实现它们。在
我试图解决这个问题的一个例子:# -*- coding: utf-8 -*-
import re
# imaginary csv file. pretend that we do not know the column order.
Header = [u'col1', u'col2']
Line1 = [u'A',u'X']
Line2 = [u'B',u'Y']
fileLines = [Line1,Line2]
# dicts to translate lines
D1a = {u'A':u'a'}
D1b = {u'B':u'b'}
D2 = {u'X':u'x',u'Y':u'y'}
# dict to correspond header names with the correct dictionary.
# i would like the dictionaries to be read sequentially in col1.
refD = {u'col1':[D1a,D1b],u'col2':[D2]}
# clunky replace function
def freplace(str, dict):
rc = re.compile('|'.join(re.escape(k) for k in dict))
def trans(m):
return dict[m.group(0)]
return rc.sub(trans, str)
# get correspondence between dictionary and column
C = []
for i in range(len(Header)):
if Header[i] in refD:
C.append([refD[Header[i]],i])
# loop through lines and make replacements
for line in fileLines:
for i in range(len(line)):
for j in range(len(C)):
if C[j][1] == i:
for dict in C[j][0]:
line[i] = freplace(line[i], dict)
我的问题是这段代码很慢,我不知道如何加快速度。我是个初学者,我的猜测是我的freplace函数很大程度上是在减慢速度,因为它必须为每行中的每一列编译。我想从该函数中去掉rc = re.compile('|'.join(re.escape(k) for k in dict))行,但不知道如何做到这一点,而且仍然保留了我其余代码的功能。在