使用 AsyncIOMotorClient 异步库,进行读写分离。
读:从数据库不停读数据存到队列,
写: 从本地队列不停批量向数据库写数据。
场景: 数据大的时候,你可以快速将A的数据导入到B中。
可以提高数据插入效率,不必等待写完才读取下一批数据
import asyncio
from custom_conf import MONGODB_URL
from motor.motor_asyncio import AsyncIOMotorClient
from pymongo import UpdateOne
class CombineShop:
def __init__(self, db: str, collection_read: str, collection_write: str, batch: int = 200):
"""
初始化
:param db: mongo database
:param collection_read: mongo read collection
:param collection_write: mongo write collection
:param batch: bulk write
"""
self.batch = batch
self.db = db
self.collection_read = collection_read
self.collection_write = collection_write
self.url = MONGODB_URL.format(self.db)
self.queue = asyncio.Queue() # 共享队列
self.finished =