Python 操作 MongoDB：增、删、改、查

擒贼先擒王

已于 2023-07-11 10:22:23 修改

阅读量3k

点赞数 2

分类专栏： MongoDB 文章标签： mongodb python 数据库

于 2018-06-25 11:01:19 首次发布

原文链接：https://www.cnblogs.com/melonjiang/p/6536876.html

版权

MongoDB 专栏收录该内容

2 篇文章 1 订阅

订阅专栏

MongoDB 教程、高级教程：https://www.runoob.com/mongodb/mongodb-tutorial.html

官网：https://www.mongodb.com/

Pymongo 多线程安全。多进程不保证安全

Pymongo "多线程" 是安全的。"多进程"不保证是安全的。如果想要多进程同时写入同一个 collection中，需要在每个进程中创建自己的MongoClient对象。

官方文档：https://pymongo.readthedocs.io/en/stable/faq.html?highlight=thread#is-pymongo-thread-safe

pymongo.MongoClient 类初始化

：https://www.mongodb.com/docs/manual/reference/connection-string/

参数说明

host 参数可以是完整的 mongodb URI <http://dochub.mongodb.org/core/connections>
密码中如果包含保留字符: ':', '/', '+' and '@' 必须按照RFC2396编码:
   from urllib.parse import quote_plus

   uri = "mongodb://%s:%s@%s" % (
       quote_plus(user), quote_plus(password), host)
   client = MongoClient(uri)

参数：( 更详细说明，看源码中注释 )
- `host` (可选，默认：127.0.0.1): hostname、IP地址、mongodb URI
- `port` (可选，默认：27017): 端口号。
- `document_class` (optional): 用于此客户机上查询返回的文档的默认类
- `tz_aware` (optional): if ``True``,
   :class:`~datetime.datetime` instances returned as values
   in a document by this :class:`MongoClient` will be timezone
   aware (otherwise they will be naive)

- `type_registry` (optional): instance of
   :class:`~bson.codec_options.TypeRegistry` to enable encoding
   and decoding of custom types.
- `datetime_conversion`:
指定如何在BSON中解码UTC日期时间。有效的选项包括
'datetime_ms'作为日期时间返回，
'datetime'作为日期时间返回。默认为'datetime'。
'datetime_auto'用于在基础日期时间超出范围时返回DatetimeMS对象，
'datetime_clamp'用于钳住最小和最大可能的日期时间。
See :ref:`handling-out-of-range-datetimes` for details.

**其他可选参数通过 **kwargs 传递
- `directConnection` (optional):
- `maxPoolSize` (optional):
- `minPoolSize` (optional):
- `maxIdleTimeMS` (optional):
- `maxConnecting` (optional):
- `timeoutMS`:
- `socketTimeoutMS`:
- `connectTimeoutMS`:
- `server_selector`:
- `serverSelectionTimeoutMS`:
- `waitQueueTimeoutMS`:
- `heartbeatFrequencyMS`:
- `appname`:
- `driver`:
- `event_listeners`:
- `retryWrites`:
- `retryReads`:
- `compressors`:
- `zlibCompressionLevel`:
- `uuidRepresentation`:
- `unicode_decode_error_handler`:
- `srvServiceName`:
- `w`:
- `wTimeoutMS`:
- `journal`:
- `fsync`:
- `replicaSet`:
- `readPreference`:
- `readPreferenceTags`:
- `maxStalenessSeconds`:
- `username`: A string.
- `password`: A string.
- `authSource`: 在哪个database上进行认证. 默认在URI中指定,

URI中没有指定时默认是admin.
- `authMechanism`:
- `authMechanismProperties`:
- `tls`:
- `tlsInsecure`:
- `tlsAllowInvalidCertificates`:
- `tlsAllowInvalidHostnames`:
- `tlsCAFile`:
- `tlsCertificateKeyFile`:
- `tlsCRLFile`:
- `tlsCertificateKeyFilePassword`:
- `tlsDisableOCSPEndpointCheck`:
- `ssl`:
- `readConcernLevel`:
- `auto_encryption_opts`:
- `server_api`:

PyMongo 连接有用户名和密码的数据库

注意：该方法在 PyMongo 4.x 中被删除！！！

import pymongo

client = pymongo.MongoClient(host='host', port=27017)
# info_data 需要用户名和密码进行身份认证的数据库
db = client.info_data
# username：用户名；password：密码
db.authenticate('username', 'password')
print(db.list_collection_names())

方法 1：通过 ip、port

import pymongo

client = pymongo.MongoClient(
# 主机
host=host,
# 端口
port=port,
# 用户名
username=username,
# 密码
password=password,
# 需要用户名和密码进行身份认证的数据库
authSource='info_data'
)
db = client.info_data
print(db.list_collection_names())

方法 2：通过 mongo uri

import pymongo

client = pymongo.MongoClient('mongodb://username:password@host:port/?authSource=info_data')
db = client.info_data
print(db.list_collection_names())

示例 1：简单封装

# -*- coding: utf-8 -*-

import json
import pymongo


class MongoDBOperate(object):

    def __init__(self, host=None, port=27017, username=None, password=None):
        super(MongoDBOperate, self).__init__()
        mongodb_config = dict(
            host=host,
            port=port,
            username=username,
            password=password,
        )
        self.__client = pymongo.MongoClient(**mongodb_config)
        pass

    def __del__(self):
        """
            析构函数，释放资源或者关闭连接
        :return:
        """
        # self.__client.close()
        pass

    def insert_one_or_many(self, db_name=None, tb_name=None, data=None):
        """
            插入一条
        :param db_name:   数据库名
        :param tb_name:   集合名（表名）
        :param data:      插入的数据（文档）: dict 类型 或者 list 类型
        """
        if db_name and tb_name and data:
            mongodb_tb = self.__client.get_database(db_name).get_collection(tb_name)
            if isinstance(data, list):
                mongodb_tb.insert_many(data)
            else:
                _id = data.get('_id', None)
                if _id:
                    filter_condition = {'_id': _id}
                    # 替换文档（upsert=True, 存在则覆盖，不存在则插入）
                    mongodb_tb.replace_one(filter_condition, data, upsert=True)
                else:
                    mongodb_tb.insert_one(data)
        else:
            raise Exception('db_name or tb_name is None')

    def find_one_or_many(self, db_name=None, tb_name=None, query_condition=None, field_list: list = None,
                         query_one=False):
        """
        :param db_name: 数据库名
        :param tb_name: 数据表名
        :param query_condition: 查询条件
        :param field_list: 要查询的字段列表
        :param query_one: 是否只查询一条数据
        :return:
        """
        if db_name and tb_name:
            mongodb_tb = self.__client.get_database(db_name).get_collection(tb_name)
            if query_condition:
                func_find = mongodb_tb.find_one if query_one else mongodb_tb.find
                query_field = {field: 1 for field in field_list} if field_list else None
                result_list = func_find(query_condition, query_field) if field_list else func_find(query_condition)
            else:
                # 没有查询条件，默认返回所有
                query_field = {field: 1 for field in field_list} if field_list else None
                result_list = mongodb_tb.find({}, query_field) if field_list else mongodb_tb.find()
            for result in result_list:
                yield result
        else:
            raise Exception('db_name or tb_name is None')

    def update_one_or_many(self, db_name=None, tb_name=None, update_condition=None, update_one=True, update_id=None, ):
        """
            Update a single document matching the filter ( update_condition )
        :param db_name:
        :param tb_name:
        :param update_condition:  filter 条件
        :param update_one:
        :param update_id: 根据 _id 进行更新
        :return:
        """
        if db_name and tb_name and update_condition:
            mongodb_tb = self.__client.get_database(db_name).get_collection(tb_name)
            if update_id:
                result = mongodb_tb.update_one({'_id': update_id}, update_condition)
                return result
            func_update = mongodb_tb.update_one if update_one else mongodb_tb.update_many
            result = func_update(update_condition)
        else:
            raise Exception('db_name or tb_name or update_condition is None')
        return result

    def remove_one_or_many(self, db_name=None, tb_name=None, remove_condition=None, remove_one=True):
        if db_name and tb_name and remove_condition:
            mongodb_tb = self.__client.get_database(db_name).get_collection(tb_name)
            # remove_condition = { "name": "test_test" }
            func_delete = mongodb_tb.delete_one if remove_one else mongodb_tb.delete_many
            func_delete(filter=remove_condition)
        else:
            raise Exception('db_name or tb_name or remove_condition is None')


if __name__ == '__main__':
    mongo_db = MongoDBOperate(
        host='127.0.0.1',
        username='admin_username',
        password='xxxxxxxx'
    )
    mongo_db.insert_one_or_many('test_db', 'test_test', {'test_data': 'test_data'})
    pass

示例 2：( feapder 项目 MongoDB )

feapder 项目的 db 操作：https://github.com/Boris-code/feapder/tree/master/feapder/db

import re
import pymongo
from pymongo import MongoClient
from pymongo.database import Database
from pymongo.collection import Collection
from typing import List, Dict, Optional
from pymongo.errors import DuplicateKeyError, BulkWriteError
from loguru import logger


class MongoDB(object):

    def __init__(self, host='127.0.0.1', port=27017, db='admin', username=None, password=None, **kwargs):
        if host.startswith('mongodb://'):
            self.client = MongoClient(host=host)
        else:
            mongo_db_config = dict(
                host=host, port=port,
                username=username, password=password
            )
            self.client = MongoClient(host=host)
            kwargs.update(mongo_db_config)
            self.client = MongoClient(**kwargs)
        self.db = self.get_database(db)
        # 缓存索引信息
        self.__index__cached = {}
        pass

    def __del__(self):
        pass

    def get_database(self, database, **kwargs) -> Database:
        """
        获取数据库对象
        @param database: 数据库名
        @return:
        """
        return self.client.get_database(database, **kwargs)

    def get_collection(self, coll_name, **kwargs) -> Collection:
        """
        根据集合名获取集合对象
        @param coll_name: 集合名
        @return:
        """
        return self.db.get_collection(coll_name, **kwargs)

    def find(
            self, coll_name: str, condition: Optional[Dict] = None, limit: int = 0, **kwargs
    ) -> List[Dict]:
        """
        @summary:
        无数据： 返回[]
        有数据： [{'_id': 'xx', ...}, ...]
        ---------
        @param coll_name: 集合名(表名)
        @param condition: 查询条件
        @param limit: 结果数量
        @param kwargs:
            更多参数 https://docs.mongodb.com/manual/reference/command/find/#command-fields

        ---------
        @result:
        """
        condition = {} if condition is None else condition
        command = {"find": coll_name, "filter": condition, "limit": limit}
        command.update(kwargs)
        result = self.run_command(command)
        cursor = result["cursor"]
        cursor_id = cursor["id"]
        dataset = cursor["firstBatch"]
        while True:
            if cursor_id == 0:
                break
            result = self.run_command(
                {
                    "getMore": cursor_id,
                    "collection": coll_name,
                    "batchSize": kwargs.get("batchSize", 100),
                }
            )
            cursor = result["cursor"]
            cursor_id = cursor["id"]
            dataset.extend(cursor["nextBatch"])
        return dataset

    def add(
            self,
            coll_name,
            data: Dict,
            replace=False,
            update_columns=(),
            update_columns_value=(),
            insert_ignore=False,
    ):
        """
        添加单条数据
        Args:
            coll_name: 集合名
            data: 单条数据
            replace: 唯一索引冲突时直接覆盖旧数据，默认为False
            update_columns: 更新指定的列（如果数据唯一索引冲突，则更新指定字段，如 update_columns = ["name", "title"]
            update_columns_value: 指定更新的字段对应的值, 不指定则用数据本身的值更新
            insert_ignore: 索引冲突是否忽略 默认False

        Returns: 插入成功的行数

        """
        affect_count = 1
        collection = self.get_collection(coll_name)
        try:
            collection.insert_one(data)
        except DuplicateKeyError as e:
            # 存在则更新
            if update_columns:
                if not isinstance(update_columns, (tuple, list)):
                    update_columns = [update_columns]

                condition = self.__get_update_condition(
                    coll_name, data, e.details.get("errmsg")
                )

                # 更新指定的列
                if update_columns_value:
                    # 使用指定的值更新
                    doc = {
                        key: value
                        for key, value in zip(update_columns, update_columns_value)
                    }
                else:
                    # 使用数据本身的值更新
                    doc = {key: data[key] for key in update_columns}

                collection.update_one(condition, {"$set": doc})

            # 覆盖更新
            elif replace:
                condition = self.__get_update_condition(
                    coll_name, data, e.details.get("errmsg")
                )
                # 替换已存在的数据
                collection.replace_one(condition, data)

            elif not insert_ignore:
                raise e

        return affect_count

    def add_batch(
            self,
            coll_name: str,
            datas: List[Dict],
            replace=False,
            update_columns=(),
            update_columns_value=(),
            condition_fields: dict = None,
    ):
        """
        批量添加数据
        Args:
            coll_name: 集合名
            datas: 数据 [{'_id': 'xx'}, ... ]
            replace:  唯一索引冲突时直接覆盖旧数据，默认为False
            update_columns: 更新指定的列（如果数据的唯一索引存在，则更新指定字段，如 update_columns = ["name", "title"]
            update_columns_value: 指定更新的字段对应的值, 不指定则用数据本身的值更新
            condition_fields: 用于条件查找的字段，不指定则用索引冲突中的字段查找

        Returns: 添加行数，不包含更新

        """
        add_count = 0

        if not datas:
            return add_count

        collection = self.get_collection(coll_name)
        if not isinstance(update_columns, (tuple, list)):
            update_columns = [update_columns]

        try:
            add_count = len(datas)
            collection.insert_many(datas, ordered=False)
        except BulkWriteError as e:
            write_errors = e.details.get("writeErrors")
            for error in write_errors:
                if error.get("code") == 11000:
                    # 数据重复
                    # 获取重复的数据
                    data = error.get("op")

                    def get_condition():
                        # 获取更新条件
                        if condition_fields:
                            condition = {
                                condition_field: data[condition_field]
                                for condition_field in condition_fields
                            }
                        else:
                            # 根据重复的值获取更新条件
                            condition = self.__get_update_condition(
                                coll_name, data, error.get("errmsg")
                            )

                        return condition

                    if update_columns:
                        # 更新指定的列
                        if update_columns_value:
                            # 使用指定的值更新
                            doc = {
                                key: value
                                for key, value in zip(
                                    update_columns, update_columns_value
                                )
                            }
                        else:
                            # 使用数据本身的值更新
                            doc = {key: data.get(key) for key in update_columns}

                        collection.update_one(get_condition(), {"$set": doc})
                        add_count -= 1

                    elif replace:
                        # 覆盖更新
                        collection.replace_one(get_condition(), data)
                        add_count -= 1

                    else:
                        # log.error(error)
                        add_count -= 1

        return add_count

    def count(self, coll_name, condition: Optional[Dict], limit=0, **kwargs):
        """
        计数
        @param coll_name: 集合名
        @param condition: 查询条件
        @param limit: 限制数量
        @param kwargs:
        ----
        command = {
          count: <collection or view>,
          query: <document>,
          limit: <integer>,
          skip: <integer>,
          hint: <hint>,
          readConcern: <document>,
          collation: <document>,
          comment: <any>
        }
        https://docs.mongodb.com/manual/reference/command/count/#mongodb-dbcommand-dbcmd.count
        @return: 数据数量
        """
        command = {"count": coll_name, "query": condition, "limit": limit, **kwargs}
        result = self.run_command(command)
        return result["n"]

    def update(self, coll_name, data: Dict, condition: Dict, upsert: bool = False):
        """
        更新
        Args:
            coll_name: 集合名
            data: 单条数据 {"xxx":"xxx"}
            condition: 更新条件 {"_id": "xxxx"}
            upsert: 数据不存在则插入,默认为 False

        Returns: True / False
        """
        try:
            collection = self.get_collection(coll_name)
            collection.update_one(condition, {"$set": data}, upsert=upsert)
        except Exception as e:
            logger.error(
                """
                error:{}
                condition: {}
            """.format(
                    e, condition
                )
            )
            return False
        else:
            return True

    def delete(self, coll_name, condition: Dict) -> bool:
        """
        删除
        Args:
            coll_name: 集合名
            condition: 查找条件
        Returns: True / False

        """
        try:
            collection = self.get_collection(coll_name)
            collection.delete_one(condition)
        except Exception as e:
            logger.error(
                """
                error:{}
                condition: {}
            """.format(
                    e, condition
                )
            )
            return False
        else:
            return True

    def run_command(self, command: Dict):
        """
        运行指令
        参考文档 https://www.geek-book.com/src/docs/mongodb/mongodb/docs.mongodb.com/manual/reference/command/index.html
        @param command:
        @return:
        """
        return self.db.command(command)

    def create_index(self, coll_name, keys, unique=True):
        collection = self.get_collection(coll_name)
        _keys = [(key, pymongo.ASCENDING) for key in keys]
        collection.create_index(_keys, unique=unique)

    def get_index(self, coll_name):
        return self.get_collection(coll_name).index_information()

    def drop_collection(self, coll_name):
        return self.db.drop_collection(coll_name)

    def get_index_key(self, coll_name, index_name):
        """
        获取参与索引的key
        Args:
            index_name: 索引名

        Returns:

        """
        cache_key = f"{coll_name}:{index_name}"

        if cache_key in self.__index__cached:
            return self.__index__cached.get(cache_key)

        index = self.get_index(coll_name)
        index_detail = index.get(index_name)
        if not index_detail:
            errmsg = f"not found index {index_name} in collection {coll_name}"
            raise Exception(errmsg)

        index_keys = [val[0] for val in index_detail.get("key")]
        self.__index__cached[cache_key] = index_keys
        return index_keys

    def __get_update_condition(
            self, coll_name: str, data: dict, duplicate_errmsg: str
    ) -> dict:
        """
        根据索引冲突的报错信息 获取更新条件
        Args:
            duplicate_errmsg: E11000 duplicate key error collection: feapder.test index: a_1_b_1 dup key: { : 1, : "你好" }
            data: {"a": 1, "b": "你好", "c": "嘻嘻"}

        Returns: {"a": 1, "b": "你好"}

        """
        index_name = re.search(r"index: (\w+)", duplicate_errmsg).group(1)
        index_keys = self.get_index_key(coll_name, index_name)

        condition = {key: data.get(key) for key in index_keys}
        return condition


if __name__ == '__main__':
    mongo_url = 'mongodb://admin:admin_password@10.10.10.10:27017'
    mongo_db = MongoDB(host=mongo_url, db='test_db')
    mongo_db.add('test_tb', {'test_data': 'test_data'})
    pass

Python 连接 MongoDB 集群

Python 连接 MongoDB 的驱动为 pymongo, 基于 pymongo 有封装的 mongoengine，以及基于mongoengine 的 flask_mongoengie。

方法 1：使用 pymongo 连接 MongoDB 集群

示例代码：

from pymongo import MongoClient

# 创建 MongoDB 连接 URI
uri = "mongodb://username:password@hostname1:port1,hostname2:port2,hostname3:port3/?replicaSet=replica_set_name"

# 连接到 MongoDB
client = MongoClient(uri)

# 获取数据库和集合
db = client.your_database
collection = db.your_collection

# 执行操作（示例）
result = collection.find_one()
print(result)

说明：

username：MongoDB 6 的用户名
password：MongoDB 6 的密码
hostname1:port1,hostname2:port2,hostname3:port3：MongoDB 分片集群的主机名和端口号列表，以逗号分隔
replica_set_name：MongoDB 分片集群的副本集名称
your_database：要操作的数据库名称
your_collection：要操作的集合名称

示例代码 2：

建立了一个MongoDB集群，集群名称 replSet=rs0，包含以下 3 个节点：

172.16.250.233:27017 # SECONDARY
172.16.250.234:27017 # PRIMARY
172.16.250.237:27017 # SECONDARY

集群的 test 用户的密码为 “123456” ，只能访问 test 数据库。另外还有一个 MongoDB 数据库单独部署在 172.16.250.238，其 admin 用户的密码为 “123456” ，只能访问 admin 数据库。MongoDB 集群没有提供类似MySQL集群和Redis集群中的虚拟IP。在使用的时候需要客户端连接时指定MongoDB 集群的所有节点。

# coding=utf-8

from pymongo import MongoClient

# ################################ single mongo ##########################################
# ret : ['admin', 'config', 'local']
c = MongoClient(host="172.16.250.238", port=27017)
# c = MongoClient(host="mongodb://172.16.250.238:27017")
# c = MongoClient(host="mongodb://admin:123456@172.16.250.238:27017")
# c = MongoClient(host="mongodb://admin:123456@172.16.250.238:27017/admin")

# ################################ mongo cluster ##########################################
# ret : ['test']
mongodb_cluster_ip = '172.16.250.233:27017,172.16.250.234:27017,172.16.250.237:27017'
c = MongoClient(f'mongodb://test:123456@{mongodb_cluster_ip}/test')
# c = MongoClient(f'mongodb://test:123456@{mongodb_cluster_ip}/test?replicaSet=rs0')
# c = MongoClient(f'mongodb://test:123456@{mongodb_cluster_ip}/test?readPreference=secondary')

print(c.list_database_names())

方法 2：使用 mongoengine 连接 MongoDB 集群

示例代码：

# coding=utf-8

from mongoengine import connect
from mongoengine import Document
from mongoengine import StringField

# single mongo
connect('admin', host='mongodb://admin:123456@172.16.250.238:27017')


# mongo cluster
cluster_ip = '172.16.250.233:27017,172.16.250.234:27017,172.16.250.237:27017'
# connect('test', host=f'mongodb://test:123456@{cluster_ip}/test?readPreference=secondary')


class User(Document):  # 类名需要与数据库中的集合名称一致
    name = StringField(required=True, max_length=200)


users = User.objects.all()  # 返回所有的文档对象列表

for u in users:
    print("name:", u.name)
pass

方法 3：使用 flask_mongoengine 连接 MongoDB 集群

示例代码：

# coding=utf-8
from flask_mongoengine import MongoEngine
from flask import Flask

app = Flask(__name__)

# app.config['MONGODB_SETTINGS'] = {
#         'host': 'mongodb://admin:123456@172.16.250.238:27017/admin',
#         }

# app.config.from_pyfile('mongo.cfg')

# app.config['MONGODB_SETTINGS'] = {
#        'db': 'admin',
#        'username': 'admin',
#        'password': '123456',
#        'host': '172.16.250.238',
#        'port': 27017
#        }

app.config['MONGODB_SETTINGS'] = [
    {
        'db': 'test',
        'username': 'test',
        'password': '123456',
        'host': '172.16.250.233',
        'port': 27017
    },
    {
        'db': 'test',
        'username': 'test',
        'password': '123456',
        'host': '172.16.250.234',
        'port': 27017
    }, {
        'db': 'test',
        'username': 'test',
        'password': '123456',
        'host': '172.16.250.237',
        'port': 27017
    }
]

db = MongoEngine(app)


class User(db.Document):
    name = db.StringField(required=True, max_length=50)


users = User.objects.all()  # 返回所有的文档对象列表

for u in users:
    print("name:", u.name)

配置文件 mongo.cfg 格式如下：

MONGODB_SETTINGS = {
    'host': 'mongodb://admin:123456@172.16.250.238:27017/admin',
}