K8S1.19.16部署MongoDB复制集集群并设置安全验证

1 篇文章 0 订阅
1 篇文章 0 订阅

前提
三台k8s集群环境机器
安装了glusterfs文件系统(主要用于做持久化,省略的话需要调整一下持久化方式)
安装了docker

1.生成认证文件

openssl rand -base64 768 > mongodb.key

2.重新构建镜像
这里主要是加入认证文件和设置目录权限,为了省事直接设置了777,可以设置600

#镜像打包文件,置入认证文件、设置文件夹权限
cat >> Dockerfile << EOF
FROM mongo:4.4.4
RUN mkdir -p /data/logs
RUN mkdir -p /data/key
COPY ./mongodb.key /data/key/mongodb.key
RUN chmod 400 /data/key/mongodb.key
RUN chmod 777 /data/logs/
RUN chown -R mongodb:mongodb /data/
EOF

docker build -f Dockerfile -t 127.0.0.1:8089/kubernetes/mongo:4.4.4 .

3.部署服务
①创建持久化端点(可以省略或者使用其他任意持久化方式)

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
- addresses:
  - ip: 172.27.12.250
  - ip: 172.27.12.249
  - ip: 172.27.12.248
  ports:
  - port: 49152
    protocol: TCP

②创建MongoDB配置文件
注意配置目录与上面构建镜像设置的目录一致

apiVersion: v1
data:
  mongod.conf: |-
    dbpath=/data/db
    logpath=/data/logs/mongodb.log
    journal=true
    #pidfilepath=/data/run/mongodb.pid
    directoryperdb=true
    logappend=true
    bind_ip=0.0.0.0
    port=27017
    auth=true
    keyFile=/data/key/mongodb.key
kind: ConfigMap
metadata:
  name: mongodb-config
  namespace: default

③创建pv和pvc(可以根据自己的环境调整pv使用的端点)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: system-mongodb-replica-pv-1
  labels:
    pv: system-mongodb-replica-pv-1
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: system-mongodb-replica-1
    readOnly: false

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-1
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-replica-pv-2
  labels:
    pv: mongodb-replica-pv-2
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: mongodb-replica-2
    readOnly: false

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-2
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-2

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-3
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-3

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-3
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-3
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-replica-pv-3
  labels:
    pv: mongodb-replica-pv-3
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: mongodb-replica-3
    readOnly: false

④部署有状态应用和无头服务
这里注意使用的是前面构建好的镜像
注意踩坑点:挂载目录位置、开启ipv6(似乎不开启也没事)、无头服务(后面会提到)

apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-1
spec: 
  selector: 
    matchLabels: 
      name: mongodb-1
  serviceName: "mongodb-1"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-1
        app: mongodb-cluster-1
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-1
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-1
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-1

---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-2
spec: 
  selector: 
    matchLabels: 
      name: mongodb-2
  serviceName: "mongodb-2"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-2
        app: mongodb-cluster-2
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-2
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-2
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-2

---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-3
spec: 
  selector: 
    matchLabels: 
      name: mongodb-3
  serviceName: "mongodb-3"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-3
        app: mongodb-cluster-3
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-3
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-3
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-3

⑤测试运行
部署busybox测试无头服务网络,这里使用1.28.3的版本,注意有些版本有bug

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

进入到busybox容器中

kubectl exec -it busybox sh

#输入命令测试无头服务解析地址情况
nslookup mongodb-1

输出结果:
结果格式类似于下面,其中Name表示解析到的服务域名,Address表示绑定到的POD和其DNS主机地址

/ # nslookup mongodb-1
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mongodb-1
Address 1: 10.244.1.224 mongodb-1-0.mongodb-1.provide-service.svc.cluster.local

这里主要是用于负载均衡,详情可以自行百度无头服务

⑥创建外部访问服务

apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-1
  namespace: default
  labels:
    name: mongodb-cluster-1
spec:
  selector:
    app: mongodb-cluster-1
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30881
      port: 27017
      protocol: TCP
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-2
  namespace: default
  labels:
    name: mongodb-cluster-2
spec:
  selector:
    app: mongodb-cluster-2
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30882
      port: 27017
      protocol: TCP
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-3
  namespace: default
  labels:
    name: mongodb-cluster-3
spec:
  selector:
    app: mongodb-cluster-3
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30883
      port: 27017
      protocol: TCP
      targetPort: 27017      

⑦初始化集群

#进入mongodb-1容器
kubectl exec -it mongodb-1-0 bash

#连接数据库
mongo

连接到数据库后

#初始化集群,arbiterOnly表示仲裁节点
config = {_id: 'MongoDBCluster', members: [{_id: 0, host: '172.27.12.250:30881'},{_id: 1, host: '172.27.12.250:30882'},{_id: 2, host: '172.27.12.250:30883','arbiterOnly':true}] }
rs.initiate(config)
#查看节点状态,会展示节点类型和地址等信息
rs.status()

#过几秒后本节点出现 MongoDBCluster:PRIMARY> 则表示本节点为主节点,在主节点上创建初始账号
use admin
db.createUser({user: "root",pwd: "123456",roles: [ { role: "root",db: "admin" } ]})
#测试登录
db.auth('root','123456')

注意:这里有个大坑
我在一开始的时候初始化集群使用的是无头服务域名地址,即

config = {_id: 'MongoDBCluster', members: [{_id: 0, host: 'mongodb-1:27017'},{_id: 1, host: 'mongodb-2:27017'},{_id: 2, host: 'mongodb-3:27017','arbiterOnly':true}] }

然后使用java或者Navicat数据库连接工具进行连接,连接使用的地址是外部访问地址,即172.27.12.250:30881,172.27.12.250:30882,172.27.12.250:30883这些。
结果单独使用主节点或者副节点进行连接是没问题的,但是使用集群方式进行访问就一直提无法解析服务地址。
Java连接提示:
Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@73ffe7b0. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongodb-3:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-3}, caused by {java.net.UnknownHostException: mongodb-3}}, {address=mongodb-1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-1}, caused by {java.net.UnknownHostException: mongodb-1}}, {address=mongodb-2:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-2}, caused by {java.net.UnknownHostException: mongodb-2}}]

Navicat连接显示:
no suitable servers found: serverSelectionTimeoutMS expired: [Failed to resolve ‘mongodb-1’] [Failed to resolve ‘mongodb-2’] [Failed to resolve ‘mongodb-3’]

查找资料,很多说是主机开启了ipv6但是mongodb没有开启,然后我开启ipv6再次尝试结果还是一样。然后找到一个答案说是mongodb集群的连接是由客户端指定的,java连接集群时需要调整连接串,我尝试了一下也是没用,这里我没有进行太多的尝试,有兴趣的可以自己百度看看,也许是我设置的不对。最后看到一个答案说是DNS解析失败,需要对主机的host进行调整,对照提示的错误信息,这个应该是最符合我这个情况的,经过诸多尝试最后还是没有解决使用无头服务初始化集群这个问题,无奈,只有将集群初始化地址改为外部可以访问的到的一个地址
最后,java和navicat连接成功。

参考资料:

kuberntes集群不能解析service ip故障排查记录
k8s使用StatefulSet部署MongoDB集群
k8s部署mongo集群
Window Mongodb5.0 搭建复制集
Windows平台下安装MongoDB(集群)

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值