compare mongo 3.0.0 rc7 & PostgreSQL 9.3.5 insert use python

测试机用的CentOS 6.6 x64 96G内存, 8核, 数据文件放在 OCZ SSD

下载mongo 3.0

# tar -zxvf mongodb-linux-x86_64-3.0.0-rc7.gz
# cd mongodb-linux-x86_64-3.0.0-rc7
# cd bin/

创建数据文件目录

# mkdir -p /data01/mongodb

启动数据库, 参数全部在命令行中

# ./mongod --port 5281 --bind_ip 0.0.0.0 --maxConns 10000 --logpath /tmp/mongo.log --logappend --logRotate reopen --pidfilepath /tmp/mongo.pid --noauth --dbpath /data01/mongodb --fork



2015-02-04T23:23:42.438+0800 I CONTROL  ***** SERVER RESTARTED *****
2015-02-04T23:23:42.470+0800 I CONTROL  [initandlisten] MongoDB starting : pid=627122 port=5281 dbpath=/data01/mongodb 64-bit host=localhost.localdomain
2015-02-04T23:23:42.470+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-02-04T23:23:42.470+0800 I CONTROL  [initandlisten] 
2015-02-04T23:23:42.471+0800 I CONTROL  [initandlisten] db version v3.0.0-rc7
2015-02-04T23:23:42.471+0800 I CONTROL  [initandlisten] git version: e4c60053b2967e16f765fa25d16aa6d629faa196
2015-02-04T23:23:42.471+0800 I CONTROL  [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-02-04T23:23:42.471+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2015-02-04T23:23:42.471+0800 I CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", maxIncomingConnections: 10000, port: 5281 }, processManagement: { fork: true, pidFilePath: "/tmp/mongo.pid" }, security: { authorization: "disabled" }, storage: { dbPath: "/data01/mongodb" }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/tmp/mongo.log" } }
2015-02-04T23:23:42.483+0800 I JOURNAL  [initandlisten] journal dir=/data01/mongodb/journal
2015-02-04T23:23:42.483+0800 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2015-02-04T23:23:42.586+0800 I JOURNAL  [durability] Durability thread started
2015-02-04T23:23:42.587+0800 I JOURNAL  [journal writer] Journal writer thread started
2015-02-04T23:23:42.594+0800 I NETWORK  [initandlisten] waiting for connections on port 5281

下载,安装 pymongo驱动

https://pypi.python.org/pypi/pymongo/

# wget https://pypi.python.org/packages/source/p/pymongo/pymongo-2.8.tar.gz#md5=23100361c9af1904eb2d7722f2658114
# tar -zxvf pymongo-2.8.tar.gz
# cd pymongo-2.8
# python setup.py install

退出pymongo的源码目录

测试

# cd
[root@localhost ~]# python
Python 3.4.2 (default, Jan 27 2015, 00:11:11) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymongo
# 因为我们这里使用的是单节点mongo, 所以使用MongoClient链接即可, 如果用到了master-slave或replica, 最好使用对应的链接接口.
>>> c=pymongo.MongoClient(host='127.0.0.1',port=5281,max_pool_size=1000)
# 创建或连接到一个数据库
>>> db = c.test_database
# 创建或使用一个已有的collection
>>> collection = db.test_collection
# 生成条dict记录
>>> row={'id':1, 'username': 'digoal.zhou', 'age':32, 'email':'digoal@126.com', 'qq':'276732431'}
>>> row
{'age': 32, 'qq': '276732431', 'email': 'digoal@126.com', 'id': 1, 'username': 'digoal.zhou'}
>>> type(row)
<class 'dict'>
# 插入到collection
>>> collection.insert(row)
ObjectId('54d22c8de138238e62bcd73f')
>>> collection
Collection(Database(MongoClient('127.0.0.1', 5281), 'test_database'), 'test_collection')
# 注意再次插入为什么失败呢?
# 原因是insert后row被变更了, 自动添加一个_id, 即mongodb 中唯一标识一个document的值, 必须唯一.
>>> r1=collection.insert(row)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/site-packages/pymongo-2.8-py3.4-linux-x86_64.egg/pymongo/collection.py", line 410, in insert
    _check_write_command_response(results)
  File "/usr/local/lib/python3.4/site-packages/pymongo-2.8-py3.4-linux-x86_64.egg/pymongo/helpers.py", line 202, in _check_write_command_response
    raise DuplicateKeyError(error.get("errmsg"), 11000, error)
pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: test_database.test_collection.$_id_ dup key: { : ObjectId('54d22c8de138238e62bcd73f') }
>>> print(row)
{'age': 32, 'qq': '276732431', 'email': 'digoal@126.com', '_id': ObjectId('54d22d37e138238e62bcd740'), 'id': 1, 'username': 'digoal.zhou'}

测试mongodb插入性能

# vi test.py
import pymongo
import time

# 使用unix socket连接
c=pymongo.MongoClient('/tmp/mongodb-5281.sock')
# c=pymongo.MongoClient(host='127.0.0.1',port=5281,max_pool_size=1000)
db = c.test_database
db.drop_collection('test_collection')
collection = db.test_collection
print(time.time())
print(collection.count())
for i in range(0,1000000):
  collection.insert({'id':1, 'username': 'digoal.zhou', 'age':32, 'email':'digoal@126.com', 'qq':'276732431'})
print(time.time())
print(collection.count())

测试结果 : 

100万条记录, 单线程插入耗时378秒.

# python ./test.py
1423061694.137046
0
1423062072.2314758
1000000

对比测试同一台主机的postgresql插入性能 : 

$ vi test.py
import postgresql
import time

# 使用unix socket连接
conn = { "user": "postgres", 
         "database": "postgres",
         "unix": "/data01/pgdata/pg_root/.s.PGSQL.1921"
}

db = postgresql.open(**conn)
db.execute("drop table if exists tt")
db.execute("create table tt(id int, username name, age int2, email text, qq text)")
ins = db.prepare("insert into tt values($1,$2,$3,$4,$5)")
print(time.time())
print(db.query("select count(1) as a from tt"))
for i in range(0,1000000):
  ins(1,'digoal.zhou',32,'digoal@126.com','276732431')
print(time.time())
print(db.query("select count(1) as a from tt"))

测试结果 :

100万条记录, 单线程插入耗时187秒.

$ python test.py
1423062311.1081557
[(0,)]
1423062498.4938233
[(1000000,)]

测试结果仅供参考.

[参考]

1.  http://api.mongodb.org/python/current/api/pymongo/index.html

2.  http://blog.163.com/digoal@126/blog/static/16387704020151210840303/