文章目录
一、我们为什么需要分片?
在Mongodb里面存在另一种集群,就是分片技术,可以满足MongoDB数据量大量增长的需求。
当MongoDB存储海量的数据时,一台机器可能不足以存储数据,也可能不足以提供可接受的读写吞吐量。这时,我们就可以通过在多台机器上分割数据,使得数据库系统能存储和处理更多的数据。
- 复制所有的写入操作到主节点
- 延迟的敏感数据会在主节点查询
- 单个副本集限制在12个节点
- 当请求量巨大时会出现内存不足。
- 本地磁盘不足
- 垂直扩展价格昂贵
二、分片架构
上图中主要有如下所述三个主要组件:
- Shard: 用于存储实际的数据块,实际生产环境中一个shard server角色可由几台机器组个一个relica set承担,防止主机单点故障
- Config Server: mongod实例,存储了整个 ClusterMetadata,其中包括 chunk信息。从3.4版本开始,config Server必须部署为副本集架构(CSRS)
- Query Routers: 前端路由,客户端由此接入,且让整个集群看上去像单一数据库,前端应用可以透明使用。
三、测试规划
角色 | IP:PORT |
---|---|
| 172.16.113.129:27017 |
分片(shard) | 172.16.113.129:27018 |
172.16.113.129:27019 | |
配置(config) | 172.16.113.137:27017 |
路由(router) | 172.16.113.137:27018 |
备注:机器有限,shard、config、router我们都使用单节点,不用副本集。但是生产禁用
四、搭建
4.1 shard
4.1.1 配置文件
[root@node01 mongo]# cat mongod.conf
dbpath=/usr/bin/mongo/data
logpath=/usr/bin/mongo/logs/mongodb.log
pidfilepath=/usr/bin/mongo/mongod.pid
directoryperdb=true
logappend=true
bind_ip=127.0.0.1,172.16.113.129
#keyFile=/usr/bin/mongo/mongodb-keyfile
port=27017
fork=true
#auth=true
#replSet=rs0
oplogSize=128
[root@node01 mongo]# cat mongod2.conf
dbpath=/usr/bin/mongo/data2
logpath=/usr/bin/mongo/logs2/mongodb.log
pidfilepath=/usr/bin/mongo/data2/mongod.pid
directoryperdb=true
logappend=true
bind_ip=127.0.0.1,172.16.113.129
#keyFile=/usr/bin/mongo/mongodb-keyfile
port=27018
fork=true
#auth=true
#replSet=rs0
oplogSize=128
[root@node01 mongo]# cat mongod3.conf
dbpath=/usr/bin/mongo/data3
logpath=/usr/bin/mongo/logs3/mongodb.log
pidfilepath=/usr/bin/mongo/data3/mongod.pid
directoryperdb=true
logappend=true
bind_ip=127.0.0.1,172.16.113.129
#keyFile=/usr/bin/mongo/mongodb-keyfile
port=27019
fork=true
#auth=true
#replSet=rs0
oplogSize=128
[root@node01 mongo]#
4.1.2 启动
mongod -f /usr/bin/mongo/mongod.conf
mongod -f /usr/bin/mongo/mongod2.conf
mongod -f /usr/bin/mongo/mongod3.conf
4.2 config
4.2.1 配置文件
dbpath=/usr/bin/mongo/config
logpath=/usr/bin/mongo/configlogs/mongodb.log
pidfilepath=/usr/bin/mongo/config/mongod.pid
directoryperdb=true
logappend=true
bind_ip=127.0.0.1,172.16.113.137
configsvr=true
port=27017
fork=true
oplogSize=100
4.2.2 启动
mongod -f /usr/bin/mongo/config.conf
4.3 mongos
4.3.1 配置文件
cat router.conf
logpath=/usr/bin/mongo/routerlogs/mongodb.log
pidfilepath=/usr/bin/mongo/router/mongod.pid
logappend=true
bind_ip=127.0.0.1,172.16.113.137
port=40000
configdb=172.16.113.137:27017 #关联配置服务
fork=true
chunkSize=200 #mongos启动参数中,chunkSize这一项是用来指定chunk的大小的,单位是MB,默认大小为200MB.
4.3.2 启动
[root@node02 mongo]# mongos -f /usr/bin/mongo/router.conf
2021-09-28T15:28:29.906+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.
about to fork child process, waiting until server is ready for connections.
forked process: 2693
child process started successfully, parent exiting
五、配置分片
5.1 添加分片节点
接下来,我们使用MongoDB Shell登录到mongos,添加Shard节点
mongos> use admin
switched to db admin
mongos> db.runCommand({addshard:"172.16.113.129:27017"})
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> db.runCommand({addshard:"172.16.113.129:27018"})
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> db.runCommand({addshard:"172.16.113.129:27019"})
{ "shardAdded" : "shard0001", "ok" : 1 }
查看状态
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("6152c41db4c49a97e985606f")
}
shards:
{ "_id" : "shard0001", "host" : "172.16.113.129:27019" }
{ "_id" : "shard0002", "host" : "172.16.113.129:27017" }
{ "_id" : "shard0003", "host" : "172.16.113.129:27018" }
active mongoses:
"3.2.10" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "gengjin", "primary" : "shard0001", "partitioned" : false }
{ "_id" : "hello", "primary" : "shard0001", "partitioned" : false }
{ "_id" : "kobe", "primary" : "shard0001", "partitioned" : false }
{ "_id" : "mydb", "primary" : "shard0001", "partitioned" : false }
{ "_id" : "nihao", "primary" : "shard0001", "partitioned" : false }
5.2 模拟测试
#1.创建用户
mongos> use admin
switched to db admin
mongos> db.createUser({"user":"admin","pwd":"admin","customData":{"name":"gengjin"},roles:[{role:"userAdminAnyDatabase",db:"admin"}]})
#2.设置分片存储的数据库
mongos> db.runCommand({ enablesharding:"test" })
{ "ok" : 1 }
mongos>
#3.设置分片存储的数据库集合
mongos> sh.shardCollection("test.user", { _id : "hashed" } )
{ "collectionsharded" : "test.user", "ok" : 1 }
mongos>
实例
mongos> db.gengjin.ensureIndex({name:1})
{
"raw" : {
"172.16.113.129:27019" : {
"createdCollectionAutomatically" : true,
"numIndexesBefore" : 1,
"numIndexesAfter" : 2,
"ok" : 1
}
},
"ok" : 1
}
mongos> sh.shardCollection("test.gengjin", {name:1})
{ "collectionsharded" : "test.gengjin", "ok" : 1 }
mongos>
#1.查看集合状态
mongos> db.gengjin.stats()
{
"sharded" : true,
"capped" : false,
"ns" : "test.gengjin",
"count" : 0,
"size" : 0,
"storageSize" : 4096,
"totalIndexSize" : 8192,
"indexSizes" : {
"_id_" : 4096,
"name_1" : 4096
},
.....
#2.查看分片状态
db.printShardingStatus()/sh.status()
#3.在mongos节点插入数据
for (var i=0; i<1000; i++) db.gengjin.insert({name:"zhangsan"+i, age:i})
#4.查看shard01、shard02、shard03三个节点,可以看到数据已经分布在这三个节点上了(mongos节点上是不保存插入数据的)
db.user.find()