Mongo实战之数据空洞的最佳实践

问题背景： 某天,开发部的同事跑过来反映: mongodb数据文件太大，快把磁盘撑爆了！其中某个db占用最大(运营环境这个db的数据量其实很小)

分析: 开发环境有大量测试的增/删/改操作,而由于MongoDB顺序写的原因，在我们删除部分无用数据后，它的storageSize并不会变小，这就造成了大量的数据空洞。

解决办法

1. 使用MongoDB自带的compact命令：

  db.collectionName.runCommand(“compact”)
 这种方式是collection级别的压缩，只能去除collection内的碎片。但是MongoDB的数据分配是DB级别的，效果有限。
 这个压缩是线上压缩，磁盘IO会比较高，从而影响线上服务。

2.采用复制集的方式滚动瘦身(offline)

1.检查服务器各节点是否正常运行 (ps -ef |grep mongod)
2.登入要处理的主节点,做降权处理rs.stepDown(),并通过命令 rs.status()来查看是否降权
3.切换成功之后，停掉该节点,删除数据文件,比如： rm -fr /mongodb/data/*
4.重新启动该节点，执行重启命令，比如：如:/mongodb/bin/mongod --config /mongodb/mongodb.conf
5.通过日志查看进程以及数据同步的进度
6.数据同步完成后,在修改后的主节点上执行命令 rs.stepDown(),做降权处理。

通过这种方式,可以做到收缩率是100%,数据完全无碎片.当然也会带来运维成本的增加，并且在Replic-Set集群只有2个副本的情况下，
还会存在一段时间内的单点风险(在下面的实验中就发生了这样的情况)。通过Offline的数据收缩后,收缩前后效果非常明显.

系统环境(实验环境,为了简单只做了1个副本,请勿在生产环境操作！！！)

主库: 192.168.2.130:27017
从库: 192.168.2.138:27017

1.安装mongo

2.配置参数(两台上配置)

[root@localhost bin]# vi mongodb.conf

dbpath=/u01/mongodb/data
port=27017
oplogSize = 2048
logpath=/u01/mongodb/logs/mongodb.log
logappend = true
fork = true
nojournal = true
bind_ip=0.0.0.0
shardsvr=true
replSet=relp1

3.启动mongo

[root@localhost bin]# ./mongod --config ./mongodb.conf

4.初始化副本集

[root@localhost bin]# ./mongo  192.168.2.130:27017

> cfg = {"_id":"repl1","members":[ {"_id":0, "host":"192.168.2.130:27017"}, {"_id":1, "host":"192.168.2.138:27017"}]}; 
{
        "_id" : "repl1",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.2.130:27017"
                },
                {
                        "_id" : 1,
                        "host" : "192.168.2.138:27017"
                }
        ]
}
>  
> rs.initiate(cfg)rs.initiate(cfg)
{ "ok" : 1 }

5.插入数据

#准备工作 安装pip工具 pymongo
[root@localhost mongodb]# tar zxvf setuptools-0.6c11.tar.gz
[root@localhost mongodb]# cd setuptools-0.6c11
[root@localhost mongodb]# python setup.py install

[root@localhost mongodb]# wget "https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb" --no-check-certificate
[root@localhost mongodb]# tar zxvf [root@localhost mongodb]# tar pip-1.5.4.tar.gz
[root@localhost mongodb]# cd pip-1.5.4
[root@localhost pip-1.5.4]# python setup.py install


[root@localhost mongodb]# pip install pymongo

#插入脚本
vi insert.py
#!/usr/bin/python
import random
from pymongo import MongoClient

client = MongoClient('192.168.2.137', 27017)

test = client.test
students = test.students
students_count = students.count()
print "student count is ", students_count

for i in xrange(0,5000000):
    classid = random.randint(1,4)
    age = random.randint(10, 30)
    student = {"classid":classid, "age":age,
"name":"fujun"}
    students.insert_one(student)
    print i

students_count = students.count()
print "student count is ", students_count

#执行插入,可以开多个窗口执行
[root@localhost mongodb]#python insert.py

#插入完毕 2kw条记录
repl1:PRIMARY> db.students.find().count() 
21511778

#查看复制集状态
repl1:PRIMARY> rs.status()rs.status()
{
        "set" : "repl1",
        "date" : ISODate("2018-03-17T02:42:01.126Z"),
        "myState" : 1,
        "term" : NumberLong(3),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.2.130:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 60089,
                        "optime" : {
                                "ts" : Timestamp(1521231587, 19),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2018-03-16T20:19:47Z"),
                        "electionTime" : Timestamp(1521213635, 1),
                        "electionDate" : ISODate("2018-03-16T15:20:35Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "192.168.2.138:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 40892,
                        "optime" : {
                                "ts" : Timestamp(1521231587, 19),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2018-03-16T20:19:47Z"),
                        "lastHeartbeat" : ISODate("2018-03-17T02:42:01Z"),
                        "lastHeartbeatRecv" : ISODate("2018-03-17T02:42:00.376Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "192.168.2.130:27017",
                        "configVersion" : 1
                }
        ],
        "ok" : 1
}

#查看storageSize和fileSize:
repl1:PRIMARY> db.stats() 
{
        "db" : "test",
        "collections" : 1,
        "objects" : 21511778,
        "avgObjSize" : 60,
        "dataSize" : 1290706680,
        "storageSize" : 410132480,
        "numExtents" : 0,
        "indexes" : 1,
        "indexSize" : 217350144,
        "ok" : 1
}

#storageSize = dataSize+size(删除的文档)（文档删除后，storageSize并不会变小）。

6.制造空洞数据

#删除3/4数据
repl1:PRIMARY> db.students.remove({"classid":{"$lt":4}}) 

WriteResult({ "nRemoved" : 9050980 })

#还剩下500w
repl1:PRIMARY> db.students.find().count()db.students.find().count()
5376340


#storageSize没有变下。
repl1:PRIMARY>db.stats()
{
        "db" : "test",
        "collections" : 1,
        "objects" : 5376340,
        "avgObjSize" : 60,
        "dataSize" : 322580400,
        "storageSize" : 425127936,
        "numExtents" : 0,
        "indexes" : 1,
        "indexSize" : 219590656,
        "ok" : 1
}

7.primary降权、停止

repl1:PRIMARY> rs.setrs.sers.stepDown()rs.stepDown()
2018-03-17T13:46:49.588+0800 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '192.168.2.130:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:135:1
DB.prototype.adminCommand@src/mongo/shell/db.js:153:16
rs.stepDown@src/mongo/shell/utils.js:1202:12
@(shell):1:1

2018-03-17T13:46:49.600+0800 I NETWORK  [thread1] trying reconnect to 192.168.2.130:27017 (192.168.2.130) failed
2018-03-17T13:46:49.618+0800 I NETWORK  [thread1] reconnect 192.168.2.130:27017 (192.168.2.130) ok
repl1:SECONDARY> 

#停止192.168.2.130
[root@localhost mongodb]# mongod --shutdown --config=/u01/mongodb/mongodb.conf
killing process with pid: 32322

8.删除192.168.2.130 上的数据文件

[root@localhost data]# rm -rf /u01/mongodb/data/*

9.重启192.168.2.130

[root@localhost data]# mongod  --config=/u01/mongodb/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 48079
child process started successfully, parent exiting

10.查看日志,跟踪数据同步过程

[root@localhost mongodb]# cd /u01/mongodb/logs/
[root@localhost logs]# tail -50f  mongodb.log

2018-03-17T13:53:31.077+0800 I REPL     [replExecDBWorker-0] Starting replication applier threads
2018-03-17T13:53:31.077+0800 I REPL     [ReplicationExecutor] 
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] ** WARNING: This replica set is running without journaling enabled but the 
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          writeConcernMajorityJournalDefault option to the replica set config 
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          is set to true. The writeConcernMajorityJournalDefault 
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          option to the replica set config must be set to false 
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          or w:majority write concerns will never complete.
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] 
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "repl1", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "192.168.2.130:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "192.168.2.138:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aab9f92276e84b589038188') } }
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] This node is 192.168.2.130:27017 in the config
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] transition to STARTUP2
2018-03-17T13:53:31.079+0800 I REPL     [rsSync] ******
2018-03-17T13:53:31.079+0800 I REPL     [rsSync] creating replication oplog of size: 2048MB...
2018-03-17T13:53:31.298+0800 I REPL     [ReplicationExecutor] Member 192.168.2.138:27017 is now in state SECONDARY
2018-03-17T13:53:31.915+0800 I STORAGE  [rsSync] Starting WiredTigerRecordStoreThread local.oplog.rs
2018-03-17T13:53:31.915+0800 I STORAGE  [rsSync] The size storer reports that the oplog contains 0 records totaling to 0 bytes
2018-03-17T13:53:31.916+0800 I STORAGE  [rsSync] Scanning the oplog to determine where to place markers for truncation
2018-03-17T13:53:34.635+0800 I REPL     [rsSync] ******
2018-03-17T13:53:34.635+0800 I REPL     [rsSync] initial sync pending
2018-03-17T13:53:37.219+0800 I REPL     [ReplicationExecutor] syncing from: 192.168.2.138:27017
2018-03-17T13:53:39.164+0800 I REPL     [rsSync] initial sync drop all databases
2018-03-17T13:53:39.165+0800 I STORAGE  [rsSync] dropAllDatabasesExceptLocal 1
2018-03-17T13:53:39.165+0800 I REPL     [rsSync] initial sync clone all databases
2018-03-17T13:53:39.281+0800 I REPL     [rsSync] fetching and creating collections for test
2018-03-17T13:53:47.061+0800 I REPL     [rsSync] initial sync cloning db: test
2018-03-17T14:19:07.020+0800 I STORAGE  [rsSync] clone test.students 70015
2018-03-17T14:19:07.021+0800 I STORAGE  [rsSync] 70105 objects cloned so far from collection test.students
2018-03-17T14:20:34.083+0800 I STORAGE  [rsSync] 139955 objects cloned so far from collection test.students
2018-03-17T14:20:34.084+0800 I STORAGE  [rsSync] clone test.students 140031
2018-03-17T14:21:38.094+0800 I STORAGE  [rsSync] 489459 objects cloned so far from collection test.students
2018-03-17T14:21:38.094+0800 I STORAGE  [rsSync] clone test.students 489471
2018-03-17T14:22:51.519+0800 I STORAGE  [rsSync] 769113 objects cloned so far from collection test.students
2018-03-17T14:22:51.519+0800 I STORAGE  [rsSync] clone test.students 769151
2018-03-17T14:24:07.410+0800 I STORAGE  [rsSync] 978790 objects cloned so far from collection test.students
2018-03-17T14:24:07.410+0800 I STORAGE  [rsSync] clone test.students 978815
2018-03-17T14:25:10.198+0800 I STORAGE  [rsSync] 1468121 objects cloned so far from collection test.students
2018-03-17T14:25:10.198+0800 I STORAGE  [rsSync] clone test.students 1468159
2018-03-17T14:25:44.018+0800 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:48042 #7 (4 connections now open)
2018-03-17T14:26:01.100+0800 I NETWORK  [conn5] end connection 127.0.0.1:48032 (3 connections now open)
2018-03-17T14:26:01.100+0800 I NETWORK  [conn7] end connection 127.0.0.1:48042 (3 connections now open)
2018-03-17T14:26:16.318+0800 I STORAGE  [rsSync] 1887602 objects cloned so far from collection test.students
2018-03-17T14:26:16.318+0800 I STORAGE  [rsSync] clone test.students 1887615
2018-03-17T14:27:19.201+0800 I STORAGE  [rsSync] clone test.students 2307071
2018-03-17T14:27:19.201+0800 I STORAGE  [rsSync] 2307083 objects cloned so far from collection test.students
2018-03-17T14:28:22.898+0800 I STORAGE  [rsSync] clone test.students 3145855
2018-03-17T14:28:22.899+0800 I STORAGE  [rsSync] 3145918 objects cloned so far from collection test.students
2018-03-17T14:29:25.991+0800 I STORAGE  [rsSync] 3565272 objects cloned so far from collection test.students
2018-03-17T14:29:25.991+0800 I STORAGE  [rsSync] clone test.students 3565311
2018-03-17T14:30:28.780+0800 I STORAGE  [rsSync] 3984753 objects cloned so far from collection test.students
2018-03-17T14:30:28.780+0800 I STORAGE  [rsSync] clone test.students 3984767
2018-03-17T14:31:28.001+0800 I STORAGE  [rsSync] clone test.students 4667519
2018-03-17T14:31:29.457+0800 I STORAGE  [rsSync] 4683761 objects cloned so far from collection test.students
2018-03-17T14:32:34.884+0800 I STORAGE  [rsSync] clone test.students 5103231
2018-03-17T14:32:34.884+0800 I STORAGE  [rsSync] 5103242 objects cloned so far from collection test.students
2018-03-17T14:33:46.017+0800 I STORAGE  [rsSync] 5452746 objects cloned so far from collection test.students
2018-03-17T14:33:46.017+0800 I STORAGE  [rsSync] clone test.students 5452799
2018-03-17T14:34:47.770+0800 I STORAGE  [rsSync] clone test.students 5872127
2018-03-17T14:34:47.770+0800 I STORAGE  [rsSync] 5872227 objects cloned so far from collection test.students
2018-03-17T14:35:52.885+0800 I STORAGE  [rsSync] 6291581 objects cloned so far from collection test.students
2018-03-17T14:35:52.885+0800 I STORAGE  [rsSync] clone test.students 6291583
2018-03-17T14:36:57.050+0800 I STORAGE  [rsSync] 6571235 objects cloned so far from collection test.students
2018-03-17T14:36:57.050+0800 I STORAGE  [rsSync] clone test.students 6571263
2018-03-17T14:37:59.514+0800 I STORAGE  [rsSync] 8248905 objects cloned so far from collection test.students
2018-03-17T14:37:59.515+0800 I STORAGE  [rsSync] clone test.students 8248959
2018-03-17T14:38:37.698+0800 I INDEX    [rsSync] build index on: test.students properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "test.students" }
2018-03-17T14:38:37.698+0800 I INDEX    [rsSync]         building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2018-03-17T14:38:40.000+0800 I -        [rsSync]   Index Build: 1511900/12074974 12%
2018-03-17T14:38:43.000+0800 I -        [rsSync]   Index Build: 3571300/12074974 29%
2018-03-17T14:38:46.000+0800 I -        [rsSync]   Index Build: 5640600/12074974 46%
2018-03-17T14:38:49.001+0800 I -        [rsSync]   Index Build: 7599000/12074974 62%
2018-03-17T14:38:52.001+0800 I -        [rsSync]   Index Build: 9471000/12074974 78%
2018-03-17T14:38:55.001+0800 I -        [rsSync]   Index Build: 11341800/12074974 93%
2018-03-17T14:39:27.000+0800 I -        [rsSync]   Index: (2/3) BTree Bottom Up Progress: 8961500/12074974 74%
2018-03-17T14:39:30.222+0800 I INDEX    [rsSync]         done building bottom layer, going to commit
2018-03-17T14:39:31.465+0800 I NETWORK  [initandlisten] connection accepted from 192.168.2.138:50802 #8 (3 connections now open)
2018-03-17T14:39:31.960+0800 I COMMAND  [conn1] command local.replset.election command: replSetRequestVotes { replSetRequestVotes: 1, setName: "repl1", dryRun: false, term: 212, candidateIndex: 1, configVersion: 1, lastCommittedOp: { ts: Timestamp 1521264812000|7795, t: 193 } } keyUpdates:0 writeConflicts:0 numYields:0 reslen:63 locks:{ Global: { acquireCount: { r: 4, w: 2 } }, Database: { acquireCount: { r: 1, W: 2 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_command 3219ms
2018-03-17T14:39:31.983+0800 I INDEX    [rsSync] build index done.  scanned 12074974 total records. 54 secs
2018-03-17T14:39:31.984+0800 I REPL     [rsSync] initial sync data copy, starting syncup
2018-03-17T14:39:31.984+0800 I REPL     [rsSync] oplog sync 1 of 3
2018-03-17T14:39:32.797+0800 I REPL     [ReplicationExecutor] syncing from: 192.168.2.138:27017
2018-03-17T14:39:33.749+0800 I REPL     [ReplicationExecutor] Member 192.168.2.138:27017 is now in state PRIMARY
2018-03-17T14:46:16.481+0800 I REPL [rsSync] oplog sync 2 of 3
2018-03-17T14:46:16.708+0800 I REPL [rsSync] initial sync building indexes
2018-03-17T14:46:16.708+0800 I REPL [rsSync] initial sync cloning indexes for : test
2018-03-17T14:46:17.236+0800 I STORAGE  [rsSync] copying indexes for: { name: "students", options: {} }
2018-03-17T14:46:19.079+0800 I REPL [rsSync] oplog sync 3 of 3
2018-03-17T14:46:19.083+0800 I REPL [rsSync] initial sync finishing up
2018-03-17T14:46:19.483+0800 I REPL [rsSync] initial sync done
2018-03-17T14:46:19.491+0800 I REPL [rsSync] initial sync succeeded after 1 attempt(s).
2018-03-17T14:46:19.491+0800 I REPL [ReplicationExecutor] transition to RECOVERING
2018-03-17T14:46:19.495+0800 I REPL [ReplicationExecutor] transition to SECONDARY

由日志中可以看到,同步花了很长时间,原因:

1.192.168.2.130降级前,复制集的同步是没有完成的,这导致了降级后，192.168.2.138并没有升级为primary,这样就造成了单点故障！
2.192.168.2.130删除数据重启后,192.168.2.138继续做delete同步操作,130的状态为startup2
3.192.168.2.138删除数据完毕后(由于没有同步完全,130上的数据文件被删除了,导致138上最终的数据不是最新的！),
  然后才成为primary

repl1:PRIMARY> db.students.find().count()
12074974

通过查询结果,可以看到数据不是最新的,新数据应该至于500w条，也证明了mongo复制集是异步的，降级的时候应该观察副本数据库的状态！！！

后面又做了一次删除,并跟踪了副本的同步情况,这次切换时间就变得很短了,并且文件大小下降的十分可观！

11. 再次查看storageSize,下降十分明显

repl1:PRIMARY> db.stats() 
{
        "db" : "test",
        "collections" : 2,
        "objects" : 5376341,
        "avgObjSize" : 60.000029946017186,
        "dataSize" : 322580621,
        "storageSize" : 102256640,
        "numExtents" : 0,
        "indexes" : 2,
        "indexSize" : 54321152,
        "ok" : 1
}

12. 192.168.2.138降权

repl1:PRIMARY>rs.stepDown()
2018-03-17T02:51:29.554-0400 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:135:1
DB.prototype.adminCommand@src/mongo/shell/db.js:153:16
rs.stepDown@src/mongo/shell/utils.js:1202:12
@(shell):1:1

2018-03-17T02:51:29.652-0400 I NETWORK  [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2018-03-17T02:51:29.791-0400 I NETWORK  [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) ok

13. 重复130的操作收缩138的空间

转载于:https://www.cnblogs.com/chinesern/p/8602239.html