MongoDB

最新推荐文章于 2022-10-17 20:18:42 发布

满眼う星辰

最新推荐文章于 2022-10-17 20:18:42 发布

阅读量402

点赞数 1

文章标签： mongodb

本文链接：https://blog.csdn.net/xyang0726/article/details/115245719

版权

MongoDB

MongoDB简介

MongoDB是一个基于分布式文件存储 [1] 的数据库。由C++语言编写。旨在为WEB应用提供可扩展的高性能数据存储解决方案。

MongoDB是一个介于关系数据库和非关系数据库之间的产品，是非关系数据库当中功能最丰富，最像关系数据库的。它支持的数据结构非常松散，是类似json的bson格式，因此可以存储比较复杂的数据类型。Mongo最大的特点是它支持的查询语言非常强大，其语法有点类似于面向对象的查询语言，几乎可以实现类似关系数据库单表查询的绝大部分功能，而且还支持对数据建立索引。

MongoDB 部署方案

1. 单实例部署

对于没有高可用需求的场景，可以使用单实例部署，即只需要运行一个 MongoDB 实例，并为该实例挂载一个可持久化的存储。

在这里插入图片描述

创建实例：

apiVersion: v1
kind: Secret
metadata:
  name: mongo-auth
  namespace: ohp
type: Opaque
data:
  username: cm9vdAo=
  password: cGFzc3dvcmQK
---
apiVersion: v1
kind: Service
metadata:
  name: mongo
  namespace: ohp
spec:
  ports:
  - port: 27017
  selector:
    app: mongo
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo
  namespace: ohp
spec:
  selector:
    matchLabels:
      app: mongo
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - image: mongo
        name: mongo
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongo-auth
              key: username
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongo-auth
              key: password
        ports:
        - containerPort: 27017
          name: mongo
        volumeMounts:
        - name: db-persistent-storage
          mountPath: /data/db
      volumes:
      - name: db-persistent-storage
        persistentVolumeClaim:
          claimName: mongo-pv-claim

其中有两点要注意：

Mongo 的认证用户名和密码配置在 Secret 中
SVC 指定了 ClusterIP 为 None，意味着 Service 将直接解析为 PodIP
Deployment 的发布策略为重建，因为 PV 只能挂载到一个 Pod 中，因此应该避免同时出现多个 Pod（该 Deployment 也不能扩容）

2. Replica set模式在kubernetes部署高可用集群

2.1 Namespace

kubectl create ns mongo

2.2 StorageClass

这里需要提前部署好NFS或者其他可提供SC的存储集群。
Kubernetes使用NFS做持久化存储

# mongo-clutser-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mongodb-data
provisioner: fuseim.pri/ifs

# create
kubectl create -f mongo-clutser-sc.yaml

2.3 Headless Service

apiVersion: v1
kind: Service
metadata:
  name: mongo
  namespace: mongo
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo

2.4 Statefulset

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
  namespace: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: prod
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: harbor.s.com/redis/mongo:3.4.22
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--bind_ip"
            - 0.0.0.0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: harbor.s.com/redis/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=prod"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: mongodb-data
      resources:
        requests:
          storage: 10Gi

2.5 RBAC

这时候查看集群状态，发现是不可用的。

kubectl exec -it mongo-0 -n mongo -- mongo
Defaulting container name to mongo.
Use 'kubectl describe pod/mongo-0 -n mongo' to see all of the containers in this pod.
MongoDB shell version v3.4.22
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.22
Server has startup warnings: 
2019-08-24T09:23:57.039+0000 I CONTROL  [initandlisten] 
2019-08-24T09:23:57.039+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-08-24T09:23:57.039+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-08-24T09:23:57.039+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-08-24T09:23:57.039+0000 I CONTROL  [initandlisten] 
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] 
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] 
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2019-08-24T09:23:57.040+0000 I CONTROL  [initandlisten] 
> rs.status()
{
        "info" : "run rs.initiate(...) if not yet done for the set",
        "ok" : 0,
        "errmsg" : "no replset config has been received",
        "code" : 94,
        "codeName" : "NotYetInitialized"
}
>

应该是mongo k8s sidecar没有正确的配置，查看其日志：

kubectl logs mongo-0 mongo-sidecar -n mongo

···
Error in workloop { [Error: [object Object]]
  message:
   { kind: 'Status',
     apiVersion: 'v1',
     metadata: {},
     status: 'Failure',
     message:
      'pods is forbidden: User "system:serviceaccount:mongo:default" cannot list resource "pods" in API group "" at the cluster scope',
     reason: 'Forbidden',
     details: { kind: 'pods' },
     code: 403 },
  statusCode: 403 }

信息显示默认分配的sa账号没有list此namespace下pods的权限，搜索了下这个问题早在很久之前在github上就有人提出，作者也给出了对应的解决方案，需要给默认的sa账号提权，增加list pods的权限，但是实际测试发现虽然给system:serviceaccount:mongo:dafault赋予pods的list权限，仍然会报错，以下是rbac配置：
mongo-k8s-sidecar/role.yaml at 2640ed1c2971b1279c2961efd257cde9fbe39574 · cvallance/mongo-k8s-sidecar · GitHub

# 使用后仍然无用的配置
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: mongo
  name: mongo-pod-read
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mongo-pod-read
  namespace: mongo
subjects:
- kind: ServiceAccount
  name: default
  namespace: mongo
roleRef:
  kind: Role
  name: mongo-pod-read
  apiGroup: rbac.authorization.k8s.io

所以我们需要重新想办法，给此sa更大的权限，这里使用默认的clusterrole view权限进行赋权，我们可以使用clusterrole对sa进行界定namespace的赋权，相当于clusterrole是一个可以进行clusterrole与role进行binding的模板：
GCE - K8s 1.8 - pods is forbidden - Cannot list pods - Unknown user “system:serviceaccount:default:default” · Issue #75 · cvallance/mongo-k8s-sidecar · GitHub

# 正确的rbac配置
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: mongo-default-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: mongo

但是pod在创建后，是无法更换sa账号与sa权限的，所以需要重建pod:

# 查看statefulset
kubectl get statefulset -n mongo
NAME    READY   AGE
mongo   3/3     23h

# scale
kubectl scale statefulset mongo -n mongo --replicas=0
statefulset.apps/mongo scaled

# 过会重新配置副本数为3
kubectl scale statefulset mongo -n mongo --replicas=3
statefulset.apps/mongo scaled

# 查看已经建立完毕
kubectl get all -n mongo
NAME          READY   STATUS    RESTARTS   AGE
pod/mongo-0   2/2     Running   0          21s
pod/mongo-1   2/2     Running   0          17s
pod/mongo-2   2/2     Running   0          12s


NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
service/mongo   ClusterIP   None         <none>        27017/TCP   23h




NAME                     READY   AGE
statefulset.apps/mongo   3/3     23h

再次查看集群状态，发现状态已经正常，集群创建成功：

kubectl exec -it mongo-0 -n mongo -- mongo
rs0:PRIMARY> rs.status()
{
        "set" : "rs0",
        "date" : ISODate("2019-08-25T08:58:12.550Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1566723485, 1),
                        "t" : NumberLong(2)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1566723485, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1566723485, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "10.244.4.87:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 19,
                        "optime" : {
                                "ts" : Timestamp(1566723485, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1566723485, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2019-08-25T08:58:05Z"),
                        "optimeDurableDate" : ISODate("2019-08-25T08:58:05Z"),
                        "lastHeartbeat" : ISODate("2019-08-25T08:58:11.877Z"),
                        "lastHeartbeatRecv" : ISODate("2019-08-25T08:58:11.192Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "10.244.3.65:27017",
                        "syncSourceHost" : "10.244.3.65:27017",
                        "syncSourceId" : 3,
                        "infoMessage" : "",
                        "configVersion" : 171757
                },
                {
                        "_id" : 1,
                        "name" : "10.244.5.9:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 19,
                        "optime" : {
                                "ts" : Timestamp(1566723485, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1566723485, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2019-08-25T08:58:05Z"),
                        "optimeDurableDate" : ISODate("2019-08-25T08:58:05Z"),
                        "lastHeartbeat" : ISODate("2019-08-25T08:58:11.875Z"),
                        "lastHeartbeatRecv" : ISODate("2019-08-25T08:58:11.478Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "10.244.4.87:27017",
                        "syncSourceHost" : "10.244.4.87:27017",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 171757
                },
                {
                        "_id" : 3,
                        "name" : "10.244.3.65:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 80,
                        "optime" : {
                                "ts" : Timestamp(1566723485, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2019-08-25T08:58:05Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "could not find member to sync from",
                        "electionTime" : Timestamp(1566723473, 1),
                        "electionDate" : ISODate("2019-08-25T08:57:53Z"),
                        "configVersion" : 171757,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                }
        ],
        "ok" : 1
}
rs0:PRIMARY>

2.6 扩容

如果需要对mongo扩容，只需要调整statefulset的replicas即可：

kubectl scale statefulset mongo --replicas=4 -n mongo

3. 使用/访问

mongo cluster访问默认连接为：

mongodb://mongo1,mongo2,mongo3:27017/dbname_?

在kubernetes中最常用的FQDN连接服务的连接为：

#appName.$HeadlessServiceName.$Namespace.svc.cluster.local

因为我们采用statefulset部署的pod，所以命名均有规则，所以实际上如果连接4副本的mongodb cluster，上面的默认连接该为（默认为namespace之外）：

mongodb://mongo-0.mongo.mongo.svc.cluster.local:27017,mongo-1.mongo.mongo.svc.cluster.local:27017,mongo-2.mongo.mongo.svc.cluster.local:27017,mongo-3.mongo.mongo.svc.cluster.local:27017/?replicaSet=rs0

4. 监控

使用helm chart prometheus-mongodb-exporter进行监控。

4.1 部署exporter

注意，这里的uri后如果是集群，必须使用“”，不然会各种告警，我在这踩了无数的坑
看起来uri是固定的，而不是自动发现，所以如果需要对集群的副本进行增加或删除，则需要helm修改uri，更新配置后重建pod。
image是为了内网容易部署，将默认image下载后放入harbor，并未做任何其他修改，可忽略。

# vi values.yaml 编辑定制参数
mongodb:
  uri: "mongodb://mongo-0.mongo.mongo.svc.cluster.local:27017,mongo-1.mongo.mongo.svc.cluster.local:27017,mongo-2.mongo.mongo.svc.cluster.local:27017,mongo-3.mongo.mongo.svc.cluster.local:27017/?replicaSet=rs0"
image:
  repository: harbor.s.com/mongo/mongodb-exporter
  tag: 0.7.0
# 部署
helm upgrade --install mongo-exporter stable/prometheus-mongodb-exporter -f values.yaml --namespace mongo --force
# 查看结果
kubectl port-forward service/mongo-exporter-prometheus-mongodb-exporter 9216
curl http://127.0.0.1:9216/metrics
.cluster.local:27017,mongo-1.mongo.mongo.svc.cluster.local:27017,mongo-2.mongo.mongo.svc.cluster.local:27017,mongo-3.mongo.mongo.svc.cluster.local:27017/?replicaSet=rs0"
image:
  repository: harbor.s.com/mongo/mongodb-exporter
  tag: 0.7.0
# 部署
helm upgrade --install mongo-exporter stable/prometheus-mongodb-exporter -f values.yaml --namespace mongo --force
# 查看结果
kubectl port-forward service/mongo-exporter-prometheus-mongodb-exporter 9216
curl http://127.0.0.1:9216/metrics

满眼う星辰

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
MongoDB

MongoDBMongoDB简介MongoDB是一个基于分布式文件存储 [1] 的数据库。由C++语言编写。旨在为WEB应用提供可扩展的高性能数据存储解决方案。MongoDB是一个介于关系数据库和非关系数据库之间的产品，是非关系数据库当中功能最丰富，最像关系数据库的。它支持的数据结构非常松散，是类似json的bson格式，因此可以存储比较复杂的数据类型。Mongo最大的特点是它支持的查询语言非常强大，其语法有点类似于面向对象的查询语言，几乎可以实现类似关系数据库单表查询的绝大部分功能，而且还支持对数
复制链接

扫一扫