k8s之mysql容器数据持久化日志以及观测云日志分析

 思路：mysql数据库中重要日志有几种，错误日志，慢查询，二进制，常规日志，这些日志通过持久化，将数据存放在宿主机中。然后通过datakit将日志采集到观测云，然后通过pipeline进行切割

一、mysql部分

1、自定义my.cnf

因为需要这几种日志，通过configmap自定义my.cnf中错误日志，慢查询，常规日志的位置；datadir目录中的数据和二进制日志也很重要，此次不涉及。

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  my.cnf: |
    [mysqld]
    basedir=/var/log/mysql
    datadir=/var/log/mysql/data
    log-error=/var/lib/mysql/mysqld.log #错误日志
    log-bin=/var/lib/mysql/mysql-bin    #二进制日志
    server-id=1
    lc-messages-dir=/opt/rh/rh-mysql57/root/usr/share/rh-mysql57-mysql/english
    explicit_defaults_for_timestamp=1
    general_log=on
    general_log_file=/var/lib/mysql/server2.log #常规日志
    slow_query_log=ON
    slow_query_log_file=/var/lib/mysql/slow.log #慢日志
    long_query_time=0.5 
    log_queries_not_using_indexes
    explicit_defaults_for_timestamp=1
    [mysql]
    socket=/var/lib/mysql/mysql.sock
    #user=root
    #password=123

2、创建mysql中的pv

需要注意数据持久化存储在宿主机位置

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysqls-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"  #宿主机目录（持久化日志存放目录）

3、创建mysql中的pvc

需要与pv绑定

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysqls-pv-claim
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  volumeName: mysqls-pv  #关联相关pv

4、启动mysql.yaml

4.1查看下mysql.yaml整体部分（包含pv、pvc、cm）

[root@k8s-master ~]# cat mysqla.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysqls-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysqls-pv-claim
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  volumeName: mysqls-pv
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  my.cnf: |
    [mysqld]
    basedir=/var/log/mysql
    datadir=/var/log/mysql/data
    log-error=/var/lib/mysql/mysqld.log
    log-bin=/var/lib/mysql/mysql-bin
    server-id=1
    lc-messages-dir=/opt/rh/rh-mysql57/root/usr/share/rh-mysql57-mysql/english
    explicit_defaults_for_timestamp=1
    general_log=on
    general_log_file=/var/lib/mysql/server2.log
    slow_query_log=ON
    slow_query_log_file=/var/lib/mysql/slow.log
    long_query_time=0.5 
    log_queries_not_using_indexes
    explicit_defaults_for_timestamp=1
    [mysql]
    socket=/var/lib/mysql/mysql.sock
    #user=root
    #password=123
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:latest
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "admin123"
        ports:
        - containerPort: 3306
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
        - name: mysql-config-volume
          mountPath: /etc/my.cnf
          subPath: my.cnf
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysqls-pv-claim
      - name: mysql-config-volume
        configMap:
          name: mysql-config

4.2启动命令

[root@k8s-master ~]# kubectl apply -f mysqla.yaml 

5、验证(pv、pvc、cm)

[root@k8s-master ~]# kubectl get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                         STORAGECLASS   REASON   AGE
mysqls-pv      5Gi        RWO            Retain           Bound         default/mysqls-pv-claim                               110m
[root@k8s-master ~]# kubectl get pvc
NAME              STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysqls-pv-claim   Bound    mysqls-pv   5Gi        RWO                           110m
[root@k8s-master ~]# kubectl get cm
NAME               DATA   AGE
mysql-config       1      110m

5.1使用kubectl describe pods mysqlXXXXX(查看mounts部分和volumes部分)

[root@k8s-master ~]# kubectl describe pods   mysql-7fdf4bcfb-wqqmx   
Containers:
  mysql:
    Mounts:
      /etc/my.cnf from mysql-config-volume (rw,path="my.cnf")
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pvxkt (ro)
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysqls-pv-claim
    ReadOnly:   false
  mysql-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-config
    Optional:  false

5.2在宿主机上查看/mnt/data目录

[root@k8s-master ~]# cd /mnt/data/ 
[root@k8s-master data]# ll
总用量 29076
-rw-r----- 1 polkitd input      181 5月   5 13:02 mysql-bin.000001
-rw-r----- 1 polkitd input  3040348 5月   5 13:02 mysql-bin.000002
-rw-r----- 1 polkitd input      158 5月   5 13:02 mysql-bin.000003
-rw-r----- 1 polkitd input       96 5月   5 13:02 mysql-bin.index
-rw-r----- 1 polkitd input     8834 5月   5 13:02 mysqld.log
srwxrwxrwx 1 polkitd input        0 5月   5 13:02 mysql.sock
-rw------- 1 polkitd input        2 5月   5 13:02 mysql.sock.lock
-rw-r----- 1 polkitd input 26694041 5月   5 13:02 server2.log
-rw-r----- 1 polkitd input     2336 5月   5 13:02 slow.log

二、datakit部分

datakit通过pvc访问mysql中持久化数据，也就是存在本地目录中的/mnt/data

1、自定义mysql.conf

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit1
data:
    mysql.conf: |-
      [[inputs.mysql]]
        interval = "10s"
        innodb = true
        tables = []
        users = []
        dbm = true
        election = true

        [inputs.mysql.log]
          files = ["/mnt/*.log"] #将宿主机数据放入/mnt中

2、创建datakit中的pv

[root@k8s-master ~]# cat nfs-pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: shareda-data
spec:
  capacity:
    storage: 3Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.10.102 #自己nfs服务地址
    path: "/mnt/data/"

3、创建datakit中的pvc

[root@k8s-master ~]# cat nfs-pvca.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shareda-data-claim
  namespace: datakit1         #自己datakit的命名空间
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 3Gi
  volumeName: shareda-data    #关联pv

4、修改pv、pvc、cm部分

[root@k8s-master ~]# cat datakit.yaml
        volumeMounts:
        - mountPath: /usr/local/datakit/cache
          name: cache
          readOnly: false
        - mountPath: /rootfs
          name: rootfs
        - mountPath: /var/run
          name: run
        - mountPath: /sys/kernel/debug
          name: debugfs
        - mountPath: /var/lib/containerd/container_logs
          name: container-logs
        # # ---iploc-start
        #- mountPath: /usr/local/datakit/data/ipdb/iploc/
        #  name: datakit-ipdb
        # # ---iploc-end
        - mountPath: /usr/local/datakit/conf.d/db/mysql.conf
          name: datakit-conf
          subPath: mysql.conf
          readOnly: true
        - mountPath: /mnt           #将nfs挂载容器内目录
          name: shareda-data-volume 
        workingDir: /usr/local/datakit
      hostIPC: true
      hostPID: true
      restartPolicy: Always
      serviceAccount: datakit
      serviceAccountName: datakit
      tolerations:
      - operator: Exists
      volumes:
      - persistentVolumeClaim:
          claimName: shareda-data-claim  #自己的pvc
        name: shareda-data-volume
      - configMap:                       #mysql的cm
          name: datakit-conf
        name: datakit-conf
      - hostPath:
          path: /
        name: rootfs
      - hostPath:
          path: /var/run
        name: run
      - hostPath:
          path: /sys/kernel/debug
        name: debugfs
      - hostPath:
          path: /root/datakit_cache
        name: cache
      - hostPath:
          path: /var/lib/containerd/container_logs
        name: container-logs
      # # ---iploc-start
      #- emptyDir: {}
      #  name: datakit-ipdb
      # # ---iploc-end
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

5、启动datakit.yaml

[root@k8s-master ~]# kubectl apply -f datakit.yaml

6、验证

需要查看pv、pvc、cm、/mnt、datakit monitor

6.1验证pv

[root@k8s-master ~]# kubectl  get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                         STORAGECLASS   REASON   AGE
shareda-data   3Gi        RWX            Retain           Bound         datakit1/shareda-data-claim                           4h27m

6.2验证pvc

[root@k8s-master ~]# kubectl  get pvc -n datakit1
NAME                 STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
shareda-data-claim   Bound    shareda-data   3Gi        RWX                           4h28m

6.3验证cm

[root@k8s-master ~]# kubectl get cm -n datakit1
NAME               DATA   AGE
datakit-conf       1      4h30m

6.4验证/mnt，进入容器，查看/mnt目录

[root@k8s-master ~]# kubectl exec -it datakit-cwxck -n datakit1 /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@k8s-master:/usr/local/datakit# cd /mnt
root@k8s-master:/mnt# ls
mysql-bin.000001  mysql-bin.000002  mysql-bin.000003  mysql-bin.index  mysql.sock  mysql.sock.lock  mysqld.log  server2.log  slow.log
root@k8s-master:/mnt# 

6.5验证是否采集成功，datakit monitor在Inputs Info部分会有logging/mysql标志

[root@k8s-master ~]# kubectl exec -it datakit-cwxck -n datakit1 /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@k8s-master:/usr/local/datakit# datakit monitor

三、观测云部分

1、验证是否采集成功

2、通过pipeline进行切割

pipeline切割是他人提供