MongoDB Sharding配置

基本配置为：10.25.34.238/239/240为shard1，名称rs0；10.25.34.239/240/241为shard2，名称rs1。241为config节点，238为route节点。config节点需要1-3个，route节点可以任意多个。这里只配置了一个config节点和一个route节点。

1.配置Replica Set

a.每台机器执行同步时间

ntpdate time.nist.gov
clock -w

几台主机时间相差太大，replica set会有问题

b.配置主机名，并修改所有主机的hosts文件

10.25.34.238 mongodb238
10.25.34.239 mongodb239
10.25.34.240 mongodb240
10.25.34.241 mongodb241
默认的时候，执行rs.initiate()，会把本机的主机名加入到集群中，所以需要其他节点的也能够解析当前主机名（也可以通过其他方式配置replica set，不加入主机名）

c.在238 239 240上启动rs0

./startrs0.sh
numactl --interleave=all /var/mongodb/bin/mongod --config /var/mongodb/conf/rs0.conf

rs0.conf内容为：

dbpath=/var/mongodb/datars0
logpath=/var/mongodb/log/rs0.log
fork = true
logappend = true                                                           
port = 27017
noauth = true
journal = true
rest = true
directoryperdb = true
replSet = rs0
oplogSize = 2048

d.在241,240,239上启动rs1

./startrs1.sh
numactl --interleave=all /var/mongodb/bin/mongod --config /var/mongodb/conf/rs1.conf

rs1.conf的内容为：

dbpath=/var/mongodb/datars1
logpath=/var/mongodb/log/rs1.log
fork = true
logappend = true                                                           
port = 37017
noauth = true
journal = true
rest = true
directoryperdb = true
replSet = rs1
oplogSize = 2048

e.初始化replica set

参考：http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
在238上：./mongo

rs.initiate();
rs.add("10.25.34.239:27017");
rs.add({_id:2, host:'10.25.34.240:27017', arbiterOnly: true});

PRIMARY> rs.status();
{
        "set" : "rs0",
        "date" : ISODate("2012-08-10T06:07:43Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongodb238:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "optime" : {
                                "t" : 1344578845000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-08-10T06:07:25Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "10.25.34.239:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 37,
                        "optime" : {
                                "t" : 1344578845000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-08-10T06:07:25Z"),
                        "lastHeartbeat" : ISODate("2012-08-10T06:07:42Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 2,
                        "name" : "10.25.34.240:27017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 18,
                        "optime" : {
                                "t" : 0,
                                "i" : 0
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2012-08-10T06:07:43Z"),
                        "pingMs" : 0
                }
        ],
        "ok" : 1
}

在241上：./mongo -port 37017

rs.initiate();
rs.add("10.25.34.240:37017");
rs.add({_id:2, host:'10.25.34.239:37017', arbiterOnly: true});
PRIMARY> rs.status();
{
        "set" : "rs1",
        "date" : ISODate("2012-08-10T06:09:06Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongodb241:37017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "optime" : {
                                "t" : 1344578934000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-08-10T06:08:54Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "10.25.34.240:37017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 30,
                        "optime" : {
                                "t" : 1344578934000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-08-10T06:08:54Z"),
                        "lastHeartbeat" : ISODate("2012-08-10T06:09:04Z"),
                        "pingMs" : 75558508
                },
                {
                        "_id" : 2,
                        "name" : "10.25.34.239:37017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 12,
                        "optime" : {
                                "t" : 0,
                                "i" : 0
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2012-08-10T06:09:04Z"),
                        "pingMs" : 0
                }
        ],
        "ok" : 1
}

2. 配置sharding

a.在241启动config

./startconfig.sh
numactl --interleave=all /var/mongodb/bin/mongod --config /var/mongodb/conf/configsvr.conf

configsvr.conf内容：

dbpath=/var/mongodb/dataconfig
logpath=/var/mongodb/log/configsvr.log
port = 20000
configsvr = true
fork = true
logappend = true                                                           
noauth = true
journal = true
rest = true

b.在238上启动mongos

./startroute.sh
numactl --interleave=all /var/mongodb/bin/mongos --config /var/mongodb/conf/route.conf
route.conf内容：

logpath=/var/mongodb/log/route.log
port = 30000
fork = true
logappend = true
chunkSize = 64                                                        
configdb = 10.25.34.241:20000

c.连接mongos节点

[root@mongodb238 bin]# ./mongo -port 30000
MongoDB shell version: 2.0.6
connecting to: 127.0.0.1:30000/test
mongos>

增加rs0和rs1到sharding中

mongos> use admin
switched to db admin
mongos> db.runCommand({addshard:"rs0/mongodb238:27017",allowlocal:true});
{ "shardAdded" : "rs0", "ok" : 1 }
mongos> db.runCommand({addshard:"rs1/mongodb241:37017",allowlocal:true});
{ "shardAdded" : "rs1", "ok" : 1 }

这时候可以用
db.runCommand( { listshards : 1 } )
看看是否列出了刚才加入的shard

mongos> db.runCommand( { listshards : 1 } )
{
        "shards" : [
                {
                        "_id" : "rs0",
                        "host" : "rs0/10.25.34.239:27017,mongodb238:27017"
                },
                {
                        "_id" : "rs1",
                        "host" : "rs1/10.25.34.240:37017,mongodb241:37017"
                }
        ],
        "ok" : 1
}

d.qiao数据库启用Sharding

mongos> db.adminCommand({"enableSharding":"qiao"});
{ "ok" : 1 }

e.posts collection上启用sharding

在posts上创复合索引（"FORUM" 和"ID" 为sharding key，选择sharding key的技巧可以参考官方文档）

db.posts.ensureIndex({ "FORUM" : 1, "ID" : 1 },{ "name" : "forum_id", "background" : true });
posts上启用sharding
> db.adminCommand({"shardCollection":"qiao.posts","key":{"FORUM":1,"ID":1}});
{ "collectionsharded" : "qiao.posts", "ok" : 1 }

如果出现下面的错误
command failed: {
        "ok" : 0,
        "errmsg" : "exception: invalid parameter: expected an object ()"
}
因为数据库中的某些记录不包含片键就会导致上面的错误

f.在reply collection上启用sharding

在reply上创复合索引(FORUMID" 和 "SUBJECTID"为sharding key)

db.reply.ensureIndex({ "FORUMID" : 1, "SUBJECTID" : 1 },{ "name" : "forumId_subjectId", "background" : true });

reply上启用sharding

db.adminCommand({"shardCollection":"qiao.reply","key":{"FORUMID":1,"s":1}});

g.查看当前sharding状态

db.printShardingStatus();

--- Sharding Status ---
  sharding version: { "_id" : 1, "version" : 3 }
  shards:
        {  "_id" : "rs0",  "host" : "rs0/10.25.34.239:27017,mongodb238:27017" }
        {  "_id" : "rs1",  "host" : "rs1/10.25.34.240:37017,mongodb241:37017" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "qiao",  "partitioned" : true,  "primary" : "rs0" }
                qiao.posts chunks:
                                rs1     9
                                rs0     29
                        too many chunks to print, use verbose if you want to force print
                qiao.reply chunks:
                                rs0     62
                        too many chunks to print, use verbose if you want to force print

等过一段时间（可长可短，可能几个小时后）数据在rs1和rs0上平均
                        

3.通过monogs插入同样的一倍数据，插入完毕后

mongos> db.printShardingStatus();
--- Sharding Status ---
  sharding version: { "_id" : 1, "version" : 3 }
  shards:
        {  "_id" : "rs0",  "host" : "rs0/10.25.34.239:27017,mongodb238:27017" }
        {  "_id" : "rs1",  "host" : "rs1/10.25.34.240:37017,mongodb241:37017" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "qiao",  "partitioned" : true,  "primary" : "rs0" }
                qiao.posts chunks:
                                rs1     25
                                rs0     24
                        too many chunks to print, use verbose if you want to force print
                qiao.reply chunks:
                                rs1     35
                                rs0     34
                        too many chunks to print, use verbose if you want to force print
        {  "_id" : "test",  "partitioned" : false,  "primary" : "rs1" }

查看balancer当前状态                        
sh.isBalancerRunning()    
可以在route.log上查看当前balancer的状态，可以看到如下的日志
Fri Aug 10 04:01:49 [Balancer] moving chunk ns: qiao.posts moving ( ns:qiao.posts at: rs0:rs0/10.25.34.239:27017,mongodb238:27017 lastmod: 11|1 min: { FORUM: "bs1227017682805", ID: "bs1339138469125184920653" } max: { FORUM: "bs1227017682805", ID: "bs1340936914029191164137" }) rs0:rs0/10.25.34.239:27017,mongodb238:27017 -> rs1:rs1/10.25.34.240:37017,mongodb241:37017

停止balancer
sh.setBalancerState(true)
启用balancer
sh.setBalancerState(false)

 

4.删除分片

http://docs.mongodb.org/manual/administration/sharding/#removing-a-shard-from-a-cluster

a.执行remove命令，并等待完成，此时rs1上的chunks开始转移到rs0上

db.runCommand( { removeshard: "rs1" } )
{
        "msg" : "draining ongoing",
        "state" : "ongoing",
        "remaining" : {
                "chunks" : NumberLong(5),
                "dbs" : NumberLong(1)
        },
        "ok" : 1
}

b.等所有的chunk转移到rs0上后，将rs1上的primary数据库移动上rs0

mongos> db.runCommand( { movePrimary: "test", to: "rs0" })
{ "primary " : "rs0:rs0/10.25.34.241:27017,mongodb240:27017", "ok" : 1 }

c.最后执行再次执行removeshard，此时删除shard完毕

mongos> db.runCommand( { removeshard: "rs1" } )
{
        "msg" : "removeshard completed successfully",
        "state" : "completed",
        "shard" : "rs1",
        "ok" : 1

}

5.小结

sharding比Replica Set复杂，除非数据量特别巨大，单机无法容纳，否则没有必要用sharding。总的几次实践下来，sharding环境下，balancer不是很稳定，而且需要考虑的问题也非常多。