k8s nacos2.2.3 部署

前提条件

准备离线镜像

• 下载镜像

docker pull mysql:5.7.44
docker pull nacos/nacos-peer-finder-plugin:1.1
docker pull nacos/nacos-server:v2.2.3

• 重新打 tag

docker tag mysql:5.7.44 registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/mysql:5.7.44
docker tag nacos/nacos-peer-finder-plugin:1.1 registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/nacos-peer-finder-plugin:1.1
docker tag nacos/nacos-server:v2.2.3 registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/nacos-server:2.2.3

• 推送到阿里云镜像仓库

docker push registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/mysql:5.7.44
docker push registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/nacos-peer-finder-plugin:1.1
docker push registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/nacos-server:2.2.3

• 清理临时镜像

docker rmi mysql:5.7.44
docker rmi nacos/nacos-peer-finder-plugin:1.1
docker rmi nacos/nacos-server:v2.2.3

准备 Nacos 部署资源

• 官方资源配置清单

# 拉取官方的配置清单，各位参考着进行修改，本文不涉及修改过程
git clone https://github.com/nacos-group/nacos-k8s.git

• 初始化数据库文件

wget https://gitee.com/wangnaiqi/nacos/raw/master/nacos-mysql.sql

部署 MySQL

Nacos 需要使用 MySQL 存储配置数据，由于使用量不大，没有考虑高可用部署，直接在 K8s 上部署。也可以采用已有的 MySQL 数据库。

资源配置清单

• mysql-cm.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nacos-mysql-config
  namespace: nacos
data:
  custom.cnf: |-
    [mysqld]
    #performance setttings
    lock_wait_timeout = 3600
    open_files_limit    = 65535
    back_log = 1024
    max_connections = 1024
    max_connect_errors = 1000000
    table_open_cache = 1024
    table_definition_cache = 1024
    thread_stack = 512K
    sort_buffer_size = 4M
    join_buffer_size = 4M
    read_buffer_size = 8M
    read_rnd_buffer_size = 4M
    bulk_insert_buffer_size = 64M
    thread_cache_size = 768
    interactive_timeout = 600
    wait_timeout = 600
    tmp_table_size = 32M
    max_heap_table_size = 32M

• mysql-secret.yaml

kind: Secret
apiVersion: v1
metadata:
  name: nacos-mysql-secret
  namespace: nacos
data:
  MYSQL_ROOT_PASSWORD: UEA4OHcwcmQ=
  MYSQL_PASSWORD: UEA4OHcwcmQ=
type: Opaque

MYSQL_ROOT_PASSWORD 和 MYSQL_PASSWORD 是 MySQL 中 root 和 nacos 用户的密码。

密码需要使用 echo -n "P@88w0rd" | base64 加密的方式。

• mysql-sts.yaml

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: nacos-mysql
  namespace: nacos
  labels:
    app: nacos-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nacos-mysql
  template:
    metadata:
      labels:
        app: nacos-mysql
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
        - name: config
          configMap:
            name: nacos-mysql-config
            items:
              - key: custom.cnf
                path: custom.cnf
            defaultMode: 420
      containers:
        - name: nacos-mysql
          image: 'registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/mysql:5.7.44'
          ports:
            - name: tcp-3306
              containerPort: 3306
              protocol: TCP
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nacos-mysql-secret
                  key: MYSQL_ROOT_PASSWORD
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nacos-mysql-secret
                  key: MYSQL_PASSWORD
            - name: MYSQL_DATABASE
              value: "nacos"
            - name: MYSQL_USER
              value: "nacos" 
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 100m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              mountPath: /etc/localtime
            - name: data
              mountPath: /var/lib/mysql
            - name: config
              mountPath: /etc/mysql/conf.d/custom.cnf
              subPath: custom.cnf
  volumeClaimTemplates:
    - metadata:
        name: data
        namespace: nacos
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: glusterfs
  serviceName: mysql-headless

---
kind: Service
apiVersion: v1
metadata:
  name: nacos-mysql-headless
  namespace: nacos
  labels:
    app: nacos-mysql
spec:
  ports:
    - name: tcp-3306
      protocol: TCP
      port: 3306
      targetPort: 3306
  selector:
    app: nacos-mysql
  clusterIP: None
  type: ClusterIP

MYSQL_DATABASE 和 MYSQL_USER 是 MySQL 初始化时创建的 Nacos 数据库名称和 Nacos 用户名称。

• mysql-external.yaml

kind: Service
apiVersion: v1
metadata:
  name: nacos-mysql-external
  namespace: nacos
  labels:
    app: nacos-mysql-external
spec:
  ports:
    - name: tcp-mysql-external
      protocol: TCP
      port: 3306
      targetPort: 3306
      nodePort: 31006
  selector:
    app: nacos-mysql
  type: NodePort

部署资源

在运维管理服务器上操作：

• 部署资源

[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-cm.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-secret.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-sts.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-external.yaml

验证

• 查看资源状态

[root@zdevops-master k8s-yaml]# kubectl get cm,secrets,pvc,sts,pods -n nacos -o wide
NAME                             DATA   AGE
configmap/nacos-mysql-config     1      3h10m

NAME                           TYPE                                  DATA   AGE
secret/nacos-mysql-secret      Opaque                                2      3h10m

NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE     VOLUMEMODE
persistentvolumeclaim/data-nacos-mysql-0   Bound    pvc-13bec5c5-33e2-46df-bcdc-dc216c17f88a   5Gi        RWO            glusterfs      3h10m   Filesystem

NAME                           READY   AGE     CONTAINERS    IMAGES
statefulset.apps/nacos-mysql   1/1     3h10m   nacos-mysql   registry.zdevops.com.cn/library/mysql:5.7.38

NAME                READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES
pod/nacos-mysql-0   1/1     Running   0          32m   10.233.116.189   ks-k8s-master-2   <none>           <none>

• MySQL 登录验证

# 进入 MySQL POD
[root@zdevops-master k8s-yaml]# kubectl exec -it nacos-mysql-0 -n nacos -- /bin/bash

# 在 MySQL POD 内部以 root 用户连接 MySQL 服务器，并列出数据库
root@nacos-mysql-0:/# mysql -u root -pP@88w0rd
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| nacos              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

mysql> quit
Bye

# 在 MySQL POD 内部以 nacos 用户连接 MySQL 服务器，并列出数据库
root@nacos-mysql-0:/# mysql -u nacos -pP@88w0rd
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| nacos              |
+--------------------+
2 rows in set (0.00 sec)

mysql> quit
Bye

导入 Nacos 初始化数据

本文采用将 SQL 文件复制到容器内部执行的方式导入数据。实际使用中，可以直接在集群外部使用 MySQL 客户端连接 MySQL 的 NodePort 管理数据库并导入数据。

• 获取初始化数据库文件

# 官方
wget https://raw.githubusercontent.com/alibaba/nacos/develop/distribution/conf/nacos-mysql.sql

# Gitee备用
# wget https://gitee.com/wangnaiqi/nacos/raw/master/nacos-mysql.sql

• 将 SQL 文件复制到 MySQL 容器内部

# 拷贝文件
cd nacos
kubectl cp nacos-mysql.sql -n nacos nacos-mysql-0:home/

# 确认文件成功拷贝
kubectl exec -it nacos-mysql-0 -n nacos -- ls /home

• 登录容器内部，导入数据

# 进入 MySQL POD
[root@zdevops-master k8s-yaml]# kubectl exec -it nacos-mysql-0 -n nacos -- /bin/bash

# 导入数据， 需要输入 nacos 用户的密码
root@nacos-mysql-0:/# mysql -u nacos -p nacos < /home/nacos-mysql.sql 
Enter password:   P@88w0rd

集群模式 Nacos 部署

资源配置清单

• nacos-cm.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-config
  namespace: nacos
data:
  mysql.host: "nacos-mysql-external.nacos"
  mysql.db.name: "nacos"
  mysql.port: "3306"
  mysql.user: "nacos"
  mysql.password: "P@88w0rd"

• nacos-sts.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
  namespace: nacos
spec:
  serviceName: nacos-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      initContainers:
        - name: peer-finder-plugin-install
## nacos-peer-finder-plugin会给每个pod插入cluster.conf，形成集群
          image: registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/nacos-peer-finder-plugin:1.1
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /home/nacos/plugins/peer-finder
              name: data
              subPath: peer-finder
      containers:
        - name: nacos
          imagePullPolicy: Always
          image: registry.cn-zhangjiakou.aliyuncs.com/wnq-dev/nacos-server:2.2.3
          resources:
            requests:
              cpu: "100m"
              memory: "1Gi"
          ports:
            - containerPort: 8848
              name: client-port
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: NACOS_REPLICAS
              value: "3"
            - name: SERVICE_NAME
              value: "nacos-headless"
            - name: DOMAIN_NAME
              value: "cluster.local"
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: MYSQL_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-config
                  key: mysql.host
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-config
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-config
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-config
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-config
                  key: mysql.password
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: NACOS_APPLICATION_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: SPRING.DATASOURCE.PLATFORM
              value: "mysql"
          volumeMounts:
            - name: data
              mountPath: /home/nacos/plugins/peer-finder
              subPath: peer-finder
            - name: data
              mountPath: /home/nacos/data
              subPath: data
            - name: data
              mountPath: /home/nacos/logs
              subPath: logs
  volumeClaimTemplates:
    - metadata:
        name: data
        namespace: nacos
      spec:
        accessModes: [ "ReadWriteMany" ]
        storageClassName: "nfs-persist"
        resources:
          requests:
            storage: 5Gi
  selector:
    matchLabels:
      app: nacos
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  namespace: nacos
  labels:
    app: nacos
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  ports:
    - port: 8848
      name: server
      targetPort: 8848
    - port: 9848
      name: client-rpc
      targetPort: 9848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
    ## 兼容1.4.x版本的选举端口
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  clusterIP: None
  selector:
    app: nacos

DOMAIN_NAME 要改成实际的 K8s 的集群域名。

• nacos-external.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: nacos-external
  namespace: nacos
  labels:
    app: external
spec:
  type: NodePort
  ports:
    - name: tcp-nacos-external
      protocol: TCP
      port: 8848
      targetPort: 8848
      nodePort: 31848
  selector:
    app: nacos

部署资源

在运维管理服务器上操作

• 部署资源

[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/nacos-cm.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/nacos-sts.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/nacos-external.yaml

部署时遇到问题处理

1、pod报错
com.alibaba.nacos.api.exception.runtime.NacosRuntimeException: errCode: 102, errMsg: dataSource or tableName is null
解决方法:

nacos-sts.yaml添加

            - name: SPRING.DATASOURCE.PLATFORM
              value: "mysql"

2、重新部署时，遇到了无法给其中一个pod写入cluster.conf的情况，经排查该pod也无法ping通mysql。最后排查出原因为calico异常，重启后恢复。