背景
处于业务上的原因,我需要先用pandas读取一下文件首行,再用pyarrow读取剩余的数据。但是在程序运行完后发现少了700多行数据。
原代码
import zipFIle
import pandas as pd
from pyarrow import feather as ft, csv as psv
with zipfile.ZipFile(file_path) as zip_file:
with zip_file.open(zip_file.namelist()[0], pwd=b"abc") as f:
fcolumns = f.readline().decode("utf-8").split(",")
first_line = f.readline().decode("utf-8").split(",")
f.seek(0)
columns_df = pd.read_csv(f, nrows=0, delimiter=",", index_col=False)
data = psv.read_csv(f)
猜测
经过反复测试,发现是因为columns_df = pd.read_csv(f, nrows=0, delimiter=",", index_col=False)
这行代码出了问题,虽然我指定了nrows=0,但是实际上文件操作符往后跳跃了700多行。通俗来说就是,虽然我只想用pandas读第一行,但它可能读了700多行,只给我返回一行。
解决
在代码最后加上 f.seek(0)
使文件操作符返回文件开头即可
最终代码
import zipFIle
import pandas as pd
from pyarrow import feather as ft, csv as psv
with zipfile.ZipFile(file_path) as zip_file:
with zip_file.open(zip_file.namelist()[0], pwd=b"abc") as f:
fcolumns = f.readline().decode("utf-8").split(",")
first_line = f.readline().decode("utf-8").split(",")
f.seek(0)
columns_df = pd.read_csv(f, nrows=0, delimiter=",", index_col=False)
data = psv.read_csv(f)
# 加这一行代码即可
f.seek(0)