python读取千万行的mysql

import os
import json
import pymysql
import logging
import logging.handlers
from database.database import *

if __name__ == '__main__':

    conn = pymysql.connect(host=db_host, user=db_user, passwd=db_passwd, db=db_name, port=db_port, charset='utf8')
    cursor = pymysql.cursors.SSCursor(conn)
    cursor.execute("SELECT user_id, share_uid FROM anchor")

    anchor_dict = {}
    num = 0
    while True:
        num = num + 1
        row = cursor.fetchone()
        print(num, row)

        try:
            anchor_id = row[0]
            word = row[1]
            count = row[2]
            weight = row[3]
            if (weight > 0) and (num > 0):
                if anchor_id in anchor_dict:
                    list1 = [word, count]
                    anchor_dict[anchor_id].append(list1)
                else:
                    anchor_dict[anchor_id] = []
                if (row == None) or (row == "None") or (num > 11253900):
                    break
        except:
            if (row == None) or (row == "None") or (num > 11253900):
                break
            pass

    # 固化人物信息
    with open("anchor_new.json", "w", encoding="utf-8") as f:
        json.dump(anchor_dict, f)

    os._exit(0)
    cursor.close()
    conn.close()

 

  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值