K8s 日志采集 (Daemonset模式)

K8s 日志采集 (Daemonset模式)

业务日志落盘到Node → Filebeat → ES → Kibana/Grafana

1、定好规范, 日志写入目录, 如/beta/logs/项目名

2、落盘到本地, Pod 开启hostPath 挂载/beta/logs/项目名到宿主机

3、Daemonset Filebeat 抓取宿主机/beta/logs/项目名日志

---

1、Filebeat 部署

Filebeat v7.10.1、ES v7.10.0

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: filebeat
    component: client
  name: filebeta
  namespace: default
spec:
  selector:
    matchLabels:
      app: filebeat
      commponent: client
  template:
    metadata:
      labels:
        app: filebeat
        commponent: client
    spec:
      containers:
        - env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
          image: 'xxxxx/elasticsearch/filebeat:7.10.1'
          imagePullPolicy: IfNotPresent
          name: filebeat-client
          resources:
            limits:
              cpu: '1'
              memory: 2Gi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/share/filebeat/filebeat.yml
              name: filebeat-config
              subPath: filebeat.yml
            - mountPath: /usr/share/filebeat/config
              name: filebeat-input-config
            - mountPath: /beta/logs
              name: hostlog
            - mountPath: /usr/share/filebeat/data
              name: filebeatdata
            - mountPath: /var/log/messages
              name: message
      hostNetwork: true
      imagePullSecrets:
        - name: xxxxxsecret
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      securityContext:
        runAsUser: 0
      volumes:
        - hostPath:
            path: /beta/logs
            type: ''
          name: hostlog
        - hostPath:
            path: /var/log/messages
            type: ''
          name: message
        - configMap:
            defaultMode: 420
            items:
              - key: filebeat.yml
                path: filebeat.yml
            name: filebeat
          name: filebeat-config
        - configMap:
            defaultMode: 420
            items:
              - key: filebeat-input.yml
                path: filebeat-input.yml
            name: filebeatinput
          name: filebeat-input-config
        - emptyDir: {}
          name: filebeatdata

filebeat, Configmap

filebeat.inputs:

filebeat.config.inputs:
  path: config/*.yml
  reload.enabled: true
  close_inactive: 10m

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

setup.template.settings:
  index.number_of_shards: 1
  
setup.kibana:
  host: "https://es-cn-xxxxxx-kibana.cn-shanghai.elasticsearch.aliyuncs.com"
  user: "elastic"
  password: "密码"
output.elasticsearch:
  # Array of hosts to connect to.
  username: filebeta_log
  password: 密码
  hosts: ["http://es-cn-xxxxx.elasticsearch.aliyuncs.com:9200"]
  indices:
    - index: "%{[fields.servicename]}-%{+yyyy.ww}"
processors:
  - add_fields:
      target: "fields"
      fields:
        environment: "prd"
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
    #      - add_kubernetes_metadata: ~
  - drop_fields:
      fields: ["agent.ephemeral_id","agent.hostname","agent.id","agent.name", "agent.type", "agent.version","host.id","host.ip","host.mac","host.os.build","host.os.family","host.os.kernel","ecs.version","log.offset","log.flags"]
      ignore_missing: false

filebeatinput, Configmap

# 2种维护方式:

# 方式一、1个业务对应1个索引, 这种在后期维护并不方便
- type: log
  enabled: false
  paths:
    - /beta/logs/job-esdatatransfer/*.log
  encoding: UTF-8
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  fields:
    servicename: "job-esdatatransfer"
    
# 方式二、所有日志写入到一个索引, 这种模式维护起来比较简单, 适用于日志数量不大的情况
- type: log
  enabled: false
  paths:
    - /beta/logs/*/*.log
  encoding: UTF-8
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  fields:
    servicename: "beta-pro-service" 

---

2、Pod 日志落盘

所有Pod统一写入, 规范目录 /beta/logs/项目名

# .spec.template.spec.containers.env
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
# .spec.template.spec.volumes
          volumeMounts:
            - mountPath: /beta/logs/beta-credit-h5api
              name: logs
              subPathExpr: $(MY_POD_IP)
# .spec.template.spec.containers[*].volumeMounts
      volumes:
        - hostPath:
            path: /beta/logs/beta-credit-h5api
            type: DirectoryOrCreate
          name: logs

3、ES使用的阿里云, 部署略