观测云之容器datakit采集宿主机日志

当DataKit部署在Kubernetes（k8s）容器内部，需要采集位于宿主机上的日志时，可遵循以下思路采集日志，以本地日志为例（mysqld.log）：
思路：nfs确保日志的一致性，用软连接将日志放到nfs里面，用pvc将nfs挂到datakit里面，通过configmap将datakit里面的log配置文件进行更改并挂载。以下为具体步骤

1、利用NFS共享存储：首先，将宿主机上的目标日志目录通过网络文件系统（NFS）服务共享出来，使得日志数据可从集群内外部访问。配置NFS服务器以便将宿主机上的日志文件路径暴露给集群内其他节点。（详情见：自建基础设施部署中的nfs部署）
1.2验证

[root@k8s-master ~]# showmount -e 192.168.10.102
Export list for 192.168.10.102:
/mnt/data *
[root@k8s-node01 ~]# showmount -e 192.168.10.102
Export list for 192.168.10.102:
/mnt/data *

2、创建PersistentVolume和PersistentVolumeClaim：接下来，在Kubernetes集群内部，为共享的NFS存储创建一个PersistentVolume（PV），确保其挂载选项与NFS共享路径相匹配。随后，创建一个对应的PersistentVolumeClaim（PVC），以声明所需的存储资源。这样，PVC会自动绑定到先前创建的PV上，使得日志数据能够在Pod之间持久化共享。

2.1创建pv:nfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: shared-data
spec:
  capacity:
    storage: 3Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.10.102 #自己nfs服务地址
    path: "/mnt/data/test" 

2.2创建pvc:nfs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data-claim
  namespace: datakit         #自己datakit的命名空间
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 3Gi
  volumeName: shared-data    #关联pv

2.3启动

kubectl apply -f nfs-pv.yaml
kubectl apply -f nfs-pvc.yaml

2.4验证(status变成了bound)

[root@k8s-master ~]# kubectl get pv
NAME          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
shared-data   3Gi        RWX            Retain           Bound    datakit/shared-data-claim                           3h54m

[root@k8s-master ~]# kubectl get pvc -n datakit
NAME                STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
shared-data-claim   Bound    shared-data   3Gi        RWX                           3h54m

3、配置DataKit容器以挂载存储卷：在DataKit所在的Pod配置中，将上述创建的PVC挂载至DataKit容器的适当路径，确保日志数据可以从挂载的NFS存储卷中被DataKit容器访问。

3.1修改datakit.yaml中volumeMounts部分

volumeMounts:
 - mountPath: /usr/local/datakit/cache
   name: cache
   readOnly: false
 - mountPath: /rootfs
   name: rootfs
 - mountPath: /var/run
   name: run
 - mountPath: /sys/kernel/debug
   name: debugfs
 - mountPath: /var/lib/containerd/container_logs
   name: container-logs
 - mountPath: /mnt           #将nfs挂载容器内目录
   name: shared-data-volume  

3.2修改datakit.yaml中 volumes部分

volumes:
- persistentVolumeClaim:
    claimName: shared-data-claim  #自己的pvc
  name: shared-data-volume

3.3验证使用kubectl describe pod xxx -n x 命令查看Volumes部分

[root@k8s-master ~]# kubectl describe pod datakit-j2j8d  -n datakit
Volumes:
  shared-data-volume:
    Type:       PersistentVolumeClaim
    ClaimName:  shared-data-claim
    ReadOnly:   false

4、在DataKit开启相关配置：在DataKit中通过ConfigMap传递配置，开启并配置日志采集模块。

4.1配置configmap部分

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
    logging.conf: |-
      [[inputs.logging]]
        ## Required
        ## File names or a pattern to tail.
        logfiles = [
          "/mnt/xxxx.log",      #容器内的路径
        ]

        # Only two protocols are supported:TCP and UDP.
        # sockets = [
        #	 "tcp://0.0.0.0:9530",
        #	 "udp://0.0.0.0:9531",
        # ]
        ## glob filteer
        ignore = [""]

        ## Your logging source, if it's empty, use 'default'.
        source = ""

        ## Add service tag, if it's empty, use $source.
        service = ""

        ## Grok pipeline script name.
        pipeline = ""

        ## optional status:
        ##   "emerg","alert","critical","error","warning","info","debug","OK"
        ignore_status = []

        ## optional encodings:
        ##    "utf-8", "utf-16le", "utf-16be", "gbk", "gb18030" or ""
        character_encoding = ""

        ## The pattern should be a regexp. Note the use of '''this regexp'''.
        ## regexp link: https://golang.org/pkg/regexp/syntax/#hdr-Syntax
        # multiline_match = '''^\S'''

        auto_multiline_detection = true
        auto_multiline_extra_patterns = []

        ## Removes ANSI escape codes from text strings.
        remove_ansi_escape_codes = false

        ## If the data sent failure, will retry forevery.
        blocking_mode = true

        ## If file is inactive, it is ignored.
        ## time units are "ms", "s", "m", "h"
        ignore_dead_log = "1h"

        ## Read file from beginning.
        from_beginning = false

        [inputs.logging.tags]
        # some_tag = "some_value"
        # more_tag = "some_other_value"

4.2开启volumeMounts:

 volumeMounts:
 - mountPath: /usr/local/datakit/cache
   name: cache
   readOnly: false
 - mountPath: /rootfs
   name: rootfs
 - mountPath: /var/run
   name: run
 - mountPath: /sys/kernel/debug
   name: debugfs
 - mountPath: /var/lib/containerd/container_logs
   name: container-logs
 - mountPath: /usr/local/datakit/conf.d/log/logging.conf
   name: datakit-conf
   subPath: logging.conf
   readOnly: true

4.3启动

[root@k8s-master ~]#  kubectl apply -f datakit.yaml

5、在观测云上在日志部分查看default部分

点击日志查看路径：/mnt/mysqld.log(容器内路径)