K8S1.19.16部署MongoDB复制集集群并设置安全验证

csdn_FeiYu

已于 2022-07-03 15:38:30 修改

阅读量1.3k

点赞数 1

分类专栏：容器 k8s mongod 文章标签： mongodb kubernetes

于 2022-07-03 15:21:24 首次发布

本文链接：https://blog.csdn.net/CSDN_FeiYu/article/details/125583879

版权

容器同时被 3 个专栏收录

4 篇文章 1 订阅

订阅专栏

k8s

1 篇文章 0 订阅

订阅专栏

mongod

1 篇文章 0 订阅

订阅专栏

前提
三台k8s集群环境机器
安装了glusterfs文件系统（主要用于做持久化，省略的话需要调整一下持久化方式）
安装了docker

1.生成认证文件

openssl rand -base64 768 > mongodb.key

2.重新构建镜像
这里主要是加入认证文件和设置目录权限，为了省事直接设置了777，可以设置600

#镜像打包文件，置入认证文件、设置文件夹权限
cat >> Dockerfile << EOF
FROM mongo:4.4.4
RUN mkdir -p /data/logs
RUN mkdir -p /data/key
COPY ./mongodb.key /data/key/mongodb.key
RUN chmod 400 /data/key/mongodb.key
RUN chmod 777 /data/logs/
RUN chown -R mongodb:mongodb /data/
EOF

docker build -f Dockerfile -t 127.0.0.1:8089/kubernetes/mongo:4.4.4 .

3.部署服务
①创建持久化端点（可以省略或者使用其他任意持久化方式）

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
- addresses:
  - ip: 172.27.12.250
  - ip: 172.27.12.249
  - ip: 172.27.12.248
  ports:
  - port: 49152
    protocol: TCP

②创建MongoDB配置文件
注意配置目录与上面构建镜像设置的目录一致

apiVersion: v1
data:
  mongod.conf: |-
    dbpath=/data/db
    logpath=/data/logs/mongodb.log
    journal=true
    #pidfilepath=/data/run/mongodb.pid
    directoryperdb=true
    logappend=true
    bind_ip=0.0.0.0
    port=27017
    auth=true
    keyFile=/data/key/mongodb.key
kind: ConfigMap
metadata:
  name: mongodb-config
  namespace: default

③创建pv和pvc（可以根据自己的环境调整pv使用的端点）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: system-mongodb-replica-pv-1
  labels:
    pv: system-mongodb-replica-pv-1
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: system-mongodb-replica-1
    readOnly: false

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-1
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-replica-pv-2
  labels:
    pv: mongodb-replica-pv-2
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: mongodb-replica-2
    readOnly: false

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-2
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-2

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-3
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-3

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-3
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-3
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-replica-pv-3
  labels:
    pv: mongodb-replica-pv-3
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: mongodb-replica-3
    readOnly: false

④部署有状态应用和无头服务
这里注意使用的是前面构建好的镜像
注意踩坑点：挂载目录位置、开启ipv6（似乎不开启也没事）、无头服务（后面会提到）

apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-1
spec: 
  selector: 
    matchLabels: 
      name: mongodb-1
  serviceName: "mongodb-1"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-1
        app: mongodb-cluster-1
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-1
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-1
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-1

---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-2
spec: 
  selector: 
    matchLabels: 
      name: mongodb-2
  serviceName: "mongodb-2"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-2
        app: mongodb-cluster-2
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-2
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-2
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-2

---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-3
spec: 
  selector: 
    matchLabels: 
      name: mongodb-3
  serviceName: "mongodb-3"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-3
        app: mongodb-cluster-3
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-3
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-3
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-3

⑤测试运行
部署busybox测试无头服务网络，这里使用1.28.3的版本，注意有些版本有bug

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

进入到busybox容器中

kubectl exec -it busybox sh

#输入命令测试无头服务解析地址情况
nslookup mongodb-1

输出结果：
结果格式类似于下面，其中Name表示解析到的服务域名，Address表示绑定到的POD和其DNS主机地址

/ # nslookup mongodb-1
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mongodb-1
Address 1: 10.244.1.224 mongodb-1-0.mongodb-1.provide-service.svc.cluster.local

这里主要是用于负载均衡，详情可以自行百度无头服务

⑥创建外部访问服务

apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-1
  namespace: default
  labels:
    name: mongodb-cluster-1
spec:
  selector:
    app: mongodb-cluster-1
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30881
      port: 27017
      protocol: TCP
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-2
  namespace: default
  labels:
    name: mongodb-cluster-2
spec:
  selector:
    app: mongodb-cluster-2
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30882
      port: 27017
      protocol: TCP
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-3
  namespace: default
  labels:
    name: mongodb-cluster-3
spec:
  selector:
    app: mongodb-cluster-3
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30883
      port: 27017
      protocol: TCP
      targetPort: 27017

⑦初始化集群

#进入mongodb-1容器
kubectl exec -it mongodb-1-0 bash

#连接数据库
mongo

连接到数据库后

#初始化集群，arbiterOnly表示仲裁节点
config = {_id: 'MongoDBCluster', members: [{_id: 0, host: '172.27.12.250:30881'},{_id: 1, host: '172.27.12.250:30882'},{_id: 2, host: '172.27.12.250:30883','arbiterOnly':true}] }
rs.initiate(config)
#查看节点状态，会展示节点类型和地址等信息
rs.status()

#过几秒后本节点出现 MongoDBCluster:PRIMARY> 则表示本节点为主节点，在主节点上创建初始账号
use admin
db.createUser({user: "root",pwd: "123456",roles: [ { role: "root",db: "admin" } ]})
#测试登录
db.auth('root','123456')

注意：这里有个大坑
我在一开始的时候初始化集群使用的是无头服务域名地址，即

config = {_id: 'MongoDBCluster', members: [{_id: 0, host: 'mongodb-1:27017'},{_id: 1, host: 'mongodb-2:27017'},{_id: 2, host: 'mongodb-3:27017','arbiterOnly':true}] }

然后使用java或者Navicat数据库连接工具进行连接，连接使用的地址是外部访问地址，即172.27.12.250:30881,172.27.12.250:30882,172.27.12.250:30883这些。
结果单独使用主节点或者副节点进行连接是没问题的，但是使用集群方式进行访问就一直提无法解析服务地址。
Java连接提示：
Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@73ffe7b0. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongodb-3:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-3}, caused by {java.net.UnknownHostException: mongodb-3}}, {address=mongodb-1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-1}, caused by {java.net.UnknownHostException: mongodb-1}}, {address=mongodb-2:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-2}, caused by {java.net.UnknownHostException: mongodb-2}}]

Navicat连接显示：
no suitable servers found: serverSelectionTimeoutMS expired: [Failed to resolve ‘mongodb-1’] [Failed to resolve ‘mongodb-2’] [Failed to resolve ‘mongodb-3’]

查找资料，很多说是主机开启了ipv6但是mongodb没有开启，然后我开启ipv6再次尝试结果还是一样。然后找到一个答案说是mongodb集群的连接是由客户端指定的，java连接集群时需要调整连接串，我尝试了一下也是没用，这里我没有进行太多的尝试，有兴趣的可以自己百度看看，也许是我设置的不对。最后看到一个答案说是DNS解析失败，需要对主机的host进行调整，对照提示的错误信息，这个应该是最符合我这个情况的，经过诸多尝试最后还是没有解决使用无头服务初始化集群这个问题，无奈，只有将集群初始化地址改为外部可以访问的到的一个地址。
最后，java和navicat连接成功。

参考资料：