示例：用python_dataset导入900w条数据到oracle

最新推荐文章于 2024-08-15 08:00:00 发布

myt2000

最新推荐文章于 2024-08-15 08:00:00 发布

阅读量636

点赞数

分类专栏：示例

本文链接：https://blog.csdn.net/myt2000/article/details/106186189

版权

示例专栏收录该内容

6 篇文章 1 订阅

订阅专栏

示例：python用dataset导入900w条数据到oracle

dataset可以批量导入数据库，以dict结构就可以导入数据库

import dataset
import logging
import time
def handle_txt():
    ids = []
    uids= []
    with open('./xxx.txt',mode='r',encoding="utf8") as f:
        rows = f.readlines()
        for row in rows:
            text = row.split(',')
            ids.append(text[0])
            uids.append(text[1])

    logging.StreamHandler(stream=None)
    db = dataset.connect('oracle://"用户名":“密码”@“数据库连接名”')
    table = db['tablename']


    import_data = []
    data = zip(ids,uids)
    data = list(data)
    for i in data :
        start_time = time.time()
        import_data.append(dict(id=i[0], uid=i[1]))
        if len(import_data) == 20000:
            table.insert_many(import_data)
            import_data = []
        end_time = time.time()
        cost_time = end_time - start_time
        logging.info("%s" % cost_time)

    import_data = []
    start_num = len(data) % 20000
    print(start_num)
    data_save = data[-start_num:]
    for i in data_save:
        import_data.append(dict(id=i[0], uid=i[1]))
        table.insert_many(import_data)
    db.commit()


if __name__ == '__main__':
    data= handle_txt()