mongodb分片集群突然停电造成一些错误，分片无法启动

今天突然停电使mongodb分片集群造成这种错误，暂时不知道怎么解决，如果 有人知道请回复我

，现在把记录下来，等后期处理。

Fri Aug  8 10:49:52.165 [initandlisten] connection accepted from 172.16.0.115:59542 #2 (2 connections now open)
Fri Aug  8 10:49:52.954 [initandlisten] connection accepted from 172.16.0.103:33501 #3 (2 connections now open)
Fri Aug  8 10:49:52.954 [initandlisten] connection accepted from 172.16.0.103:33502 #4 (3 connections now open)
Fri Aug  8 10:49:52.955 [conn3] end connection 172.16.0.103:33501 (2 connections now open)
Fri Aug  8 10:49:55.104 [rsBackgroundSync] replSet syncing to: 172.16.0.115:11731
Fri Aug  8 10:49:55.108 [rsSync] replSet still syncing, not yet to minValid optime 53e421d7:3a
Fri Aug  8 10:49:55.154 [rsSync] getFile(): n=808581868
Fri Aug  8 10:49:55.154 [rsSync] Assertion: 10295:getFile(): bad file number value (corrupt db?): run repair
0xde0151 0xda188b 0x8d5465 0xb89952 0x818116 0x818244 0x81a4d4 0xa62bd5 0xa62c1c 0xab959c 0xa6c93c 0xc222ab 0xc28184 0xc28728 0xc2b45d 0xc2b73c 0xe289d9 0x3dffe0683d 0x3dfeed503d
 /home/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
 /home/mongodb/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
 /home/mongodb/bin/mongod(_ZN5mongo8Database7getFileEiib+0x395) [0x8d5465]
 /home/mongodb/bin/mongod(_ZNK5mongo7DiskLoc3recEv+0x42) [0xb89952]
 /home/mongodb/bin/mongod(_ZNK5mongo16NamespaceDetails11inCapExtentERKNS_7DiskLocE+0x26) [0x818116]
 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0xc4) [0x818244]
 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x2e4) [0x81a4d4]
 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x25) [0xa62bd5]
 /home/mongodb/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKci+0x3c) [0xa62c1c]
 /home/mongodb/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x1ec) [0xab959c]
 /home/mongodb/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x4cc) [0xa6c93c]
 /home/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0xc222ab]
 /home/mongodb/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x584) [0xc28184]
 /home/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0xc28728]
 /home/mongodb/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2d) [0xc2b45d]
 /home/mongodb/bin/mongod(_ZN5mongo15startSyncThreadEv+0x6c) [0xc2b73c]
 /home/mongodb/bin/mongod [0xe289d9]
 /lib64/libpthread.so.0 [0x3dffe0683d]
 /lib64/libc.so.6(clone+0x6d) [0x3dfeed503d]
Fri Aug  8 10:49:55.166 [rsSync] replSet syncThread: 10295 getFile(): bad file number value (corrupt db?): run repair
Fri Aug  8 10:49:56.105 [rsSyncNotifier] replset setting oplog notifier to 172.16.0.115:11731
Fri Aug  8 10:50:02.191 [conn2] end connection 172.16.0.115:59542 (1 connection now open)
Fri Aug  8 10:50:02.191 [initandlisten] connection accepted from 172.16.0.115:59549 #5 (2 connections now open)
Fri Aug  8 10:50:06.228 [repl writer worker 1] Assertion: 10334:BSONObj size: 1342177280 (0x00000050) is invalid. Size must be between 0 and 16793600(16MB) First element: .: ?type=113
0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x8016d0 0x80180f 0x80180f 0x80190c 0x814173 0x8105f1 0x813291 0xb60b84 0xb6245e 0xb626be 0xa7c88b 0xa73af7
 /home/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
 /home/mongodb/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
 /home/mongodb/bin/mongod [0xda1dcc]
 /home/mongodb/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ec92f]
 /home/mongodb/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x98) [0xa2aad8]
 /home/mongodb/bin/mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x3c) [0xa2bfcc]
 /home/mongodb/bin/mongod(_ZNK5mongo5KeyV19woCompareERKS0_RKNS_8OrderingE+0x230) [0xa2c370]
 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findERKNS_12IndexDetailsERKNS_5KeyV1ERKNS_7DiskLocERKNS_8OrderingERib+0x120) [0x7fbcf0]
 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingERiRbS8_i+0x60) [0x8016d0]
 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingERiRbS8_i+0x19f) [0x80180f]
 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_5KeyV1ERKNS_8OrderingERiRbS8_i+0x19f) [0x80180f]
 /home/mongodb/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE6locateERKNS_12IndexDetailsERKNS_7DiskLocERKNS_7BSONObjERKNS_8OrderingERiRbS8_i+0x8c) [0x80190c]
 /home/mongodb/bin/mongod(_ZN5mongo15BtreeCursorImplINS_12BtreeData_V1EE7_locateERKNS_7BSONObjERKNS_7DiskLocE+0x93) [0x814173]
 /home/mongodb/bin/mongod(_ZN5mongo11BtreeCursor33initWithoutIndependentFieldRangesEv+0x1f1) [0x8105f1]
 /home/mongodb/bin/mongod(_ZN5mongo11BtreeCursor4makeEPNS_16NamespaceDetailsERKNS_12IndexDetailsERKNS_7BSONObjES8_bi+0xa1) [0x813291]
 /home/mongodb/bin/mongod(_ZNK5mongo15CursorGenerator14shortcutCursorEv+0x6c4) [0xb60b84]
 /home/mongodb/bin/mongod(_ZN5mongo15CursorGenerator8generateEv+0x3e) [0xb6245e]
 /home/mongodb/bin/mongod(_ZN5mongo25NamespaceDetailsTransient9getCursorERKNS_10StringDataERKNS_7BSONObjES6_RKNS_24QueryPlanSelectionPolicyERKN5boost10shared_ptrIKNS_11ParsedQueryEEEbPNS_16QueryPlanSummaryE+0x3e) [0xb626be]
 /home/mongodb/bin/mongod(_ZN5mongo13deleteObjectsEPKcNS_7BSONObjEbbbPNS_11RemoveSaverE+0x14b) [0xa7c88b]
 /home/mongodb/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xc57) [0xa73af7]
Fri Aug  8 10:50:06.241 [repl writer worker 1] ERROR: writer worker caught exception: BSONObj size: 1342177280 (0x00000050) is invalid. Size must be between 0 and 16793600(16MB) First element: .: ?type=113 on: { ts: Timestamp 1407459783000|288, h: -3441636868876691989, v: 2, op: "d", ns: "mongotest1000.S270", fromMigrate: true, o: { _id: ObjectId('53dc305d31c6e0ce936d37ac') } }
Fri Aug  8 10:50:06.241 [repl writer worker 1]   Fatal Assertion 16360
0xde0151 0xd9ff43 0xc28aac 0xdada91 0xe289d9 0x3dffe0683d 0x3dfeed503d
 /home/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
 /home/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd9ff43]
 /home/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28aac]
 /home/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]
 /home/mongodb/bin/mongod [0xe289d9]
 /lib64/libpthread.so.0 [0x3dffe0683d]
 /lib64/libc.so.6(clone+0x6d) [0x3dfeed503d]
Fri Aug  8 10:50:06.245 [repl writer worker 1]

***aborting after fassert() failure

Fri Aug  8 10:50:06.245 Got signal: 6 (Aborted).

Fri Aug  8 10:50:06.250 Backtrace:
0xde0151 0x6d0529 0x3dfee30330 0x3dfee302c5 0x3dfee31d70 0xd9ff7e 0xc28aac 0xdada91 0xe289d9 0x3dffe0683d 0x3dfeed503d
 /home/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
 /home/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0529]
 /lib64/libc.so.6 [0x3dfee30330]
 /lib64/libc.so.6(gsignal+0x35) [0x3dfee302c5]
 /lib64/libc.so.6(abort+0x110) [0x3dfee31d70]
 /home/mongodb/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd9ff7e]
 /home/mongodb/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28aac]
 /home/mongodb/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]
 /home/mongodb/bin/mongod [0xe289d9]
 /lib64/libpthread.so.0 [0x3dffe0683d]
 /lib64/libc.so.6(clone+0x6d) [0x3dfeed503d]

本文出自 “飛鴻沓膤” 博客，请务必保留此出处http://jxzhfei.blog.51cto.com/1382161/1537319