kubernetes编排文件示例

sry1201

已于 2022-12-10 18:46:35 修改

阅读量1k

点赞数 1

分类专栏： dcoker和kubernetes 文章标签： kubernetes mysql java

于 2022-12-06 20:12:33 首次发布

本文链接：https://blog.csdn.net/qq_40182873/article/details/128209450

版权

dcoker和kubernetes 专栏收录该内容

9 篇文章 0 订阅

订阅专栏

kubernetes编排文件示例

编排文件生成网址：https://www.kubebiz.com/

mysql单机

需要一个配置文件,内容不会就用默认的即可

my.cnf

[mysqld]

pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql

symbolic-links=0
sql-mode=ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION

mysql-deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-mysql
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-mysql
  template:
    metadata:
      labels:
        app: my-mysql
    spec:
      containers:
        - name: my-mysql
          image: 'mysql:8.0.30'
          imagePullPolicy: IfNotPresent
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: Jeol@1201
            - name: MYSQL_USER
              value: test
            - name: MYSQL_PASSWORD
              value: Jeol@1201
          ports:
            - containerPort: 3306
              protocol: TCP
              name: http
          volumeMounts:
            - name: my-mysql-data
              mountPath: /var/lib/mysql
            - name: mysql-conf
              mountPath: /etc/mysql/mysql.conf.d
      volumes:
        - name: my-mysql-data
          hostPath:
            # 确保文件所在目录成功创建。
            path: /app/kube/mysql/data
            type: DirectoryOrCreate
        - name: mysql-conf
          hostPath:
            path: /app/kube/mysql/conf/mysql.conf.d
            type: DirectoryOrCreate

mysql-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-export
  namespace: default
spec:
  type: NodePort
  selector:
    app: my-mysql
  ports:
    - port: 3306
      targetPort: 3306
      nodePort: 32306

nacos集群

官方：https://nacos.io/zh-cn/docs/v2/quickstart/quick-start-kubernetes.html

其他参考：https://www.likecs.com/show-305484280.html https://blog.csdn.net/warrah/article/details/106488733

1、nfs服务端安装

nfs服务端安装完成，编辑好规则，共享目录为/data/nfs-volume nfs服务的ip为 k8s-node1

2、部署NFS StorageClass

1、项目地址(有部分yaml文件从此项目获取)，可以手动下载，再上传，下面的部分命令是在下载的项目下执行

git clone https://github.com/nacos-group/nacos-k8s.git

2、角色设置

kubectl create -f deploy/nfs/rbac.yaml

如果您的K8S命名空间不是默认的，请在创建RBAC之前执行以下脚本（不执行脚本，你直接再rbac.yaml中加上namesapce配置，效果一致，脚本更快）

kubectl create ns nacos

sed -i "s/namespace:.*/namespace: nacos/g" ./deploy/nfs/rbac.yaml

3.NFS-Client Provisioner部署，用于动态生成nfs类型的pv

kubectl create -f deploy/nfs/deployment.yaml -n nacos

# 执行前修改参照部署的nfs服务，修改如下
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: k8s-node1  #修改
            - name: NFS_PATH
              value: /data/nfs-volume/nacos/nfs-share #修改
      volumes:
        - name: nfs-client-root
          nfs:
            server: k8s-node1  #修改
            path: /data/nfs-volume/nacos/nfs-share # 修改

4、storageclass创建，用于配合nfs-client-provisioner动态创建持久卷

kubectl create -f deploy/nfs/class.yaml -n nacos

记录一下内容

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs
parameters:
  archiveOnDelete: "false"

5、验证（根据标签获取对应pod的详情）

kubectl get pod -l app=nfs-client-provisioner -n nacos

3、数据库

可以只部署一个数据库，也可以部署主从数据库，需要说明的是，如果你使用先有的数据库，那么建表语句你可以再安装包的conf目录下找到

一个数据库部署参考，注意修改共享服务端地址和共享目录

kubectl create -f deploy/mysql/mysql-nfs.yaml -n nacos

主从数据库，主要是镜像不同

主

apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql-master
  labels:
    name: mysql-master
spec:
  replicas: 1
  selector:
    name: mysql-master
  template:
    metadata:
      labels:
        name: mysql-master
    spec:
      containers:
      - name: mysql-master
        image: nacos/nacos-mysql-master:latest
        ports:
        - containerPort: 3306
        volumeMounts:
        - name: mysql-master-data
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "root"
        - name: MYSQL_DATABASE
          value: "nacos_devtest"
        - name: MYSQL_USER
          value: "nacos"
        - name: MYSQL_PASSWORD
          value: "nacos"
        - name: MYSQL_REPLICATION_USER
          value: 'nacos_ru'
        - name: MYSQL_REPLICATION_PASSWORD
          value: 'nacos_ru'          
      volumes:
      - name: mysql-master-data
        nfs:
          server: k8s-node1
          path: /data/nfs-volume/nacos/mysql-master
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-master
  labels:
    name: mysql-master
spec:
  type: NodePort
  ports:
  - port: 3306
    targetPort: 3306
    nodePort: 32308
  selector:
    name: mysql-master

从

apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql-slave
  labels:
    name: mysql-slave
spec:
  replicas: 1
  selector:
    name: mysql-slave
  template:
    metadata:
      labels:
        name: mysql-slave
    spec:
      containers:
      - name: mysql-slave
        image: nacos/nacos-mysql-slave:latest
        ports:
        - containerPort: 3307
        volumeMounts:
        - name: mysql-slave-data
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "root"
        - name: MYSQL_USER
          value: "nacos"
        - name: MYSQL_PASSWORD
          value: "nacos" 
        - name: MYSQL_REPLICATION_USER
          value: 'nacos_ru'
        - name: MYSQL_REPLICATION_PASSWORD
          value: 'nacos_ru'
      volumes:
      - name: mysql-slave-data
        nfs:
          server: k8s-node1
          path: /data/nfs-volume/nacos/mysql-slave
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-slave
  labels:
    name: mysql-slave
spec:
  type: NodePort
  ports:
  - port: 3307
    targetPort: 3307
    nodePort: 32307
  selector:
    name: mysql-slave

应用生效,需要说明的是，从库无法连接，可能需要进入从库容器内部设置权限，还有就是镜像内部相关的表是已经创建好的，不需要在创建

kubectl create -f deploy/mysql/mysql-nfs-master.yaml -n nacos
kubectl create -f deploy/mysql/mysql-nfs-slave.yaml -n nacos

4、部署nacos

修改deploy/nacos/nacos-pvc-nfs.yaml和数据库配置文件保持一致即可！

主从数据可配置，

data:
  mysql.master.db.name: "主库名称"
  mysql.master.port: "主库端口"
  mysql.slave.port: "从库端口"
  mysql.master.user: "主库用户名"
  mysql.master.password: "主库密码"

kubectl create -f deploy/nacos/nacos-pvc-nfs.yaml -n nacos

pvc一直处于pending状态，

#  pvc日志
 Normal  ExternalProvisioning  4s (x22 over 5m19s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "fuseim.pri/ifs" or manually created by system administrator

# pvc日志
provision "nacos/data-nacos-0" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

解决方式一

这样之后，我的kubectl命令直接启动执行不了了，放弃

# /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --feature-gates=RemoveSelfLink=false # 增加

解决方式二

使用不基于SelfLink的provisioner

参考：https://blog.csdn.net/echizao1839/article/details/125766826

# 国外镜像
gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0
# 使用替换的国内镜像
registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0
# 替换nfs deploy中的镜像

内存不足

0/3 nodes are available: 1 Insufficient memory, 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

# 其中一台机器内存不足，所以这一块调小一点，无语，并且k8s的节点需要有三个，当然又时候并不是这个的原因，机器内存不足，该启动不了还是启动不了
            requests:
              memory: "500Mi"
              cpu: "300m"

# 调小之后，其中一致pending的nacos终于进入了初始化阶段，但是一直无法进入就绪状态，可以强制部署在内存充足的机器上，在deploy上添加
    spec:
      nodeName: k8s-node1

成功截图

5、通过ingress 访问

$ vim nacos-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: nacos
spec:
  rules:
  - host: nacos-web.nacos-demo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nacos-headless
            port:
              number: 8848

我ingress似乎有问题，天翼云端口问题，总之访问不了，配置仅供参考

6、尝试直接通过service暴露端口

添加nodePort访问，访问无响应

nacos pod内部日志报错,这个似乎不行，应该和StatefulSet的模式有关吧，改成nodePort就不行，不改的话日志还是正常的，无语，所以还是需要通过ingress访问

service 访问StatefulSet 似乎需要设置clusterIP: None，然后service会生成StatefulSet 对应的DNS地址，然后sevice改了模式之后，导致nacos pod之间互相之间访问不了，构成不了集群

2022/11/05 10:42:11 Have not found myself in list yet.
My Hostname: nacos-0.nacos-headless.nacos.svc.cluster.local
Hosts in list: nacos-headless.nacos.svc.cluster.loca

nacos单机

yaml文件如下，注意，持久卷使用的是nfs,你可以替换成集群，部分配置可能多余，毕竟是通过集群的yaml文件改的，数据库也换了，初始化sql从安装包中获取

---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  labels:
    app: nacos
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  ports:
    - port: 8848
      name: server
      targetPort: 8848
      nodePort: 30848
    - port: 9848
      name: client-rpc
      targetPort: 9848
      nodePort: 31848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
    ## 兼容1.4.x版本的选举端口
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  type: NodePort
  selector:
    app: nacos
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
data:
  mysql.service.host: "61.171.5.6"
  mysql.db.name: "nacos"
  mysql.port: "32306"
  mysql.user: "root"
  mysql.password: "Jeol@1201"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nacos
spec:
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      serviceAccountName: nfs-client-provisioner
      initContainers:
        - name: peer-finder-plugin-install
          image: nacos/nacos-peer-finder-plugin:1.1
          imagePullPolicy: Always
          volumeMounts:
            - mountPath: /home/nacos/plugins/peer-finder
              name: data
              subPath: peer-finder
      containers:
        - name: nacos
          imagePullPolicy: Always
          image: nacos/nacos-server:v2.1.2
          resources:
            requests:
              memory: "1Gi"
              cpu: "300m"
          ports:
            - containerPort: 8848
              name: client-port
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: MYSQL_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.service.host                  
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: NACOS_APPLICATION_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"            
            - name: MODE
              value: "standalone"
            - name: NACOS_AUTH_ENABLE
              value: "true"
          volumeMounts:
            - name: data
              mountPath: /home/nacos/plugins/peer-finder
              subPath: peer-finder
            - name: data
              mountPath: /home/nacos/data
              subPath: data
            - name: data
              mountPath: /home/nacos/logs
              subPath: logs
      volumes:
        - name: data
          nfs:
            server: k8s-node1
            path: /data/nfs-volume/nacos/nfs-share
  selector:
    matchLabels:
      app: nacos

访问如图

开启权限认证

默认是允许所有客户端进行操作，无需密码，添加环境变量

            - name: NACOS_AUTH_ENABLE
              value: "true"

dubbo-admin

访问是成功了，nacos也连接上了，但是查询不到服务列表，可能版本不对，需要下载源码本地跑一下试试，为啥版本是0.3.0，这个0.5.0的不好使，而且其默认的最后一个版本就是0.3.0

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dubbo-admin
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dubbo-admin
  template:
    metadata:
      labels:
        app: dubbo-admin
    spec:
      containers:
        - name: dubbo-admin
          image: apache/dubbo-admin
          imagePullPolicy: IfNotPresent
          command: [ "/bin/bash", "-ce", "java -Dadmin.registry.address=nacos://nacos:nacos@61.171.5.6:30848 -Dadmin.config-center=nacos://nacos:nacos@61.171.5.6:30848 -Dadmin.metadata-report.address=nacos://nacos:nacos@61.171.5.6:30848 -jar /app.jar"]
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 60 
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: dubbo-admin-service
  namespace: default
spec:
  type: NodePort
  selector:
    app: dubbo-admin
  ports:
    - port: 8089
      targetPort: 8080
      nodePort: 31990