pythonpandas进度_python – pandas.DataFrame.to_sql的进度条

不幸的是,DataFrame.to_sql不提供chunk-by-chunk回调,tqdm需要它来更新其状态.但是,您可以按块处理数据帧块：

import sqlite3

import pandas as pd

from tqdm import tqdm

DB_FILENAME='/tmp/test.sqlite'

def chunker(seq, size):

# from https://stackoverflow.com/a/434328

return (seq[pos:pos + size] for pos in xrange(0, len(seq), size))

def insert_with_progress(df, dbfile):

con = sqlite3.connect(dbfile)

chunksize = int(len(df) / 10) # 10%

with tqdm(total=len(df)) as pbar:

for i, cdf in enumerate(chunker(df, chunksize)):

replace = "replace" if i == 0 else "append"

cdf.to_sql(con=con, name="MLS", if_exists=replace, index=False)

pbar.update(chunksize)

df = pd.DataFrame({'a': range(0,100000)})

insert_with_progress(df, DB_FILENAME)

注意我在这里生成DataFrame inline是为了拥有一个没有依赖性的完整可行的例子.

结果令人惊叹：