一次服务器异常事件导致mongodb服务挂掉无法重启,报错信息:
2017-10-10T14:35:16.078+0800 I NETWORK [initandlisten] connection accepted from 192.168.1.52:55362 #4 (2 connections now open)
2017-10-10T14:35:16.111+0800 I QUERY [conn4] query chelun.cw_user_position query: { $query: { uid: "22014082" }, $comment: "This is a query comment", $orderby: {} } planSummary: IXSCAN { uid: 1.0 } ntoreturn:999999 ntoskip:0 nscanned:1 nscannedObjects:1 keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:468 locks:{} 31ms
2017-10-10T14:35:16.166+0800 F - [conn4] Invalid access at address: 0x7fe6e3645038
2017-10-10T14:35:16.195+0800 F - [conn4] Got signal: 7 (Bus error).
0xf662c9 0xf65942 0xf65c9e 0x7ff2f79787e0 0x7ff2f64636d5 0x7ff2f6ca5886 0xd2981c 0xd4902c 0xd50ad6 0xd49976 0x9046cc 0xc3f001 0xc3df09 0xa32f4c 0xa33705 0xbd54a4 0xbd5854 0xbd5e8d 0x9a7e29 0x9a919c 0x9a96db 0x9ac11d 0x9d0a04...skipping...
查询了资料,这种情况是mongodb异常关闭导致出错,尝试使用repair修复没有成功,找寻各种启动修复方法也均没有成功。好在服务不是核心业务,利用从库的备份(或者定期的备份数据)来恢复到一个新的库;
1、新建mongodb实例
假定服务端口定位27027,配置文件如下:
bind_ip=192.168.xx.xx
#指定服务器监听的端口,默认是27017
port=27027
#以守护进程的方式运行MongoDB
fork=true
objcheck=true
#2G
#oplogSize=2000
#pidfile
pidfilepath=/home/LBSUser/pidfile/mongo_27027.pid
#指定数据目录,默认是/data/db/。每个mongod进程都需要独立的目录,
#启动mongod时就会在数据目录中创建mongod.lock文件,防止其他mongod进程使用该数据目录。
dbpath=/data/LBSdb
#指定日志输出路径,如果不指定则会在终端输出。每次启动都会覆盖原来的日志,如果不想覆盖就要用--logappend选项
logpath=/data/log/mongodb_27027.log
logappend=true
master=true
#auth=true
profile=2
slowms=1
#only:默认为空,用于从选项,指定一个数据库进行复制
#only=test
#.禁止HTTP状态接口
nohttpinterface=true
#.禁止REST接口-在生产环境下建议不要启用MongoDB的REST接口
rest=false
#keyFile=/home/LBSUser/etc/keyfile
指定配置文件启动mongodb服务:
mongod -f /home/LBSUser/LBSmongo/mongodb.conf
(这里从库或者集群搭建略)
搭建好后,准备数据
二、导出数据
从原实例的从库节点中导出最新的数据备份(原端口是27017)
mongodump -h 192.168.xx.xx --port 27017 --oplog -o /data/backup/tmp27107/
2017-10-10T15:11:59.215+0800 [#######################.] xx.cw_user_position 19578438/20169757 (97.1%) 2017-10-10T15:12:02.215+0800 [#######################.] xx.cw_user_position 19834340/20169757 (98.3%) 2017-10-10T15:12:05.215+0800 [#######################.] xx.cw_user_position 20113312/20169757 (99.7%) 2017-10-10T15:12:05.645+0800 writing xx.cw_user_position metadata to /data/backup/tmp27107/xx/cw_user_position.metadata.json 2017-10-10T15:12:05.653+0800 done dumping xx.cw_user_position 2017-10-10T15:12:05.654+0800 writing captured oplog to /data/backup/tmp27107/oplog.bson |
看到这个后确认导出成功。
三、使用mongorestore导入数据
mongorestore导入数据可以指定库导入,我这个直接全量导入,不指定库:
mongorestore -h 192.168.xx.xx:27027 -d chelun --dir /data/backup/tmp27107/
注意这里是直接导入到我们新的机器上的新实例27027了。
2017-10-10T15:40:28.345+0800 [#######################.] xx.cw_user_position 8.0 GB/8.2 GB (97.6%) 2017-10-10T15:40:31.345+0800 [#######################.] xx.cw_user_position 8.0 GB/8.2 GB (97.9%) 2017-10-10T15:40:34.345+0800 [#######################.] xx.cw_user_position 8.0 GB/8.2 GB (98.2%) 2017-10-10T15:40:37.345+0800 [#######################.] xx.cw_user_position 8.1 GB/8.2 GB (98.5%) 2017-10-10T15:40:40.345+0800 [#######################.] xx.cw_user_position 8.1 GB/8.2 GB (98.8%) 2017-10-10T15:40:43.345+0800 [#######################.] xx.cw_user_position 8.1 GB/8.2 GB (99.1%) 2017-10-10T15:40:46.345+0800 [#######################.] xx.cw_user_position 8.1 GB/8.2 GB (99.3%) 2017-10-10T15:40:49.345+0800 [#######################.] xx.cw_user_position 8.2 GB/8.2 GB (99.6%) 2017-10-10T15:40:52.345+0800 [#######################.] xx.cw_user_position 8.2 GB/8.2 GB (99.9%) 2017-10-10T15:40:53.621+0800 restoring indexes for collection xx.cw_user_position from metadata 2017-10-10T15:44:46.613+0800 finished restoring xx.cw_user_position 2017-10-10T15:44:46.613+0800 restoring users from /data/backup/tmp27107/admin/system.users.bson 2017-10-10T15:44:47.294+0800 done |
根据导入的提示,确认导入都成功了;
四、登陆检查
做基础检查和抽样检查
[root@s0055-gz data]# mongo 192.168.xx.xx:27027 MongoDB shell version: 3.0.1 connecting to: 192.168.xx.xx:27027/test Server has startup warnings: 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2017-10-10T15:12:13.752+0800 I CONTROL [initandlisten] > db.system.users.find() > use admin switched to db admin > db.system.users.find() > db.system.users.find() { "_id" : "admin.xx_user", "user" : "xx_user", "db" : "admin", "credentials" : { "SCRAM-SHA-1" : { "iterationCount" : 10000, "salt" : "IJUswtPOEvJ1qffHjACQIQ==", "storedKey" : "FoVPjbzw74bcs+8WCShuvluA7ssrU=", "serverKey" : "bQd4TlULaxd/6BMLQHoQtvInNJOw=" } }, "roles" : [ { "role" : "root", "db" : "admin" } ] } > show dbs; admin 0.078GB xx 13.947GB local 2.077GB xx2 0.078GB |