kubernetes-sidecar方式日志收集(filebeat->kafka)

最新推荐文章于 2024-07-24 16:54:22 发布

大飞哥2

最新推荐文章于 2024-07-24 16:54:22 发布

阅读量2k

点赞数

分类专栏： kubernetes kafka ELK 文章标签： kubernetes filebeat kafka

本文链接：https://blog.csdn.net/kozazyh/article/details/79388182

版权

kubernetes 同时被 3 个专栏收录

127 篇文章 11 订阅

订阅专栏

kafka

10 篇文章 0 订阅

订阅专栏

ELK

4 篇文章 0 订阅

订阅专栏

version: filebeat:6.2.4

容器日志的默认存储方式是：json-file
1. docker 所有日志实际存放路径为：/var/lib/docker/containers/ ，日志文件的名称跟k8s的命名方式并没有关联；
2. 为了方便管理（使用deployment,container的名称来命名日志），会创建一些links，指向日志实际存储路径 /var/lib/docker/containers/ ；
a.在 /var/log/pods/pod_uuid/ 目录下为每个pod 创建links --> /var/lib/docker/containers/
b.在 /var/log/containers/ 目录下创建links --> /var/log/pods/
3. 可以看到实际日志存放在/var/lib/docker/containers/...下，并在/var/log/pods/和/var/log/containers/创建一些软links指向实际日志存放位置。这样是为了可以按pod和pod中的container找到自己的日志文件位置。

其中命名方式,下面以一个例子说明：
1.定义一个deployment:

test@local:~/k8s/test/testlog$ cat testlogs.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: testlogs
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: testlogs
    spec:
      containers:
      - name: a
        image: koza/echo-message:latest
        imagePullPolicy: IfNotPresent

2.查询运行状态：

test@local:~/k8s/test/testlog$ kubectl get pod |grep testlogs
testlogs-3220469847-sc1kx                                   2/2       Running   0          1h

3.根据testlogs的日志存放，可以看到规则如下(/var/log/containers/下的命名规则)：
格式1：pod_name _default_ container_name
格式2：container_hostname _default_ container_name
testlogs-3220469847-sc1kx_default_a
pod_name 和 container_hostname 是一样的。

[root@node1~]#ls -la /var/log/containers/testlogs-3220469847-sc1kx_default_a-ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f.log
lrwxrwxrwx 1 root root 58 2月  27 11:25 /var/log/containers/testlogs-3220469847-sc1kx_default_a-ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f.log -> /var/log/pods/e84c2a12-1b6d-11e8-8d36-fefcfe08185c/a_0.log

[root@node1 ~]# ls -la  /var/log/pods/e84c2a12-1b6d-11e8-8d36-fefcfe08185c/a_0.log
lrwxrwxrwx 1 root root 163 2月  27 11:25 /var/log/pods/e84c2a12-1b6d-11e8-8d36-fefcfe08185c/a_0.log -> /var/lib/docker/containers/ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f/ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f-json.log

[root@node1 ~]# ls -la /var/lib/docker/containers/ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f/ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f-json.log
-rw-r----- 1 root root 544318 2月  27 13:01 /var/lib/docker/containers/ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f/ad4d2f104bf5a45f9d9f13c1ed5ed4685408bbd2d4e0746dc45736035e66214f-json.log
[root@k8s-node ~]#

4.有了这些规则(/var/log/containers/${POD_NAME}-*_default_${CONTAINER_NAME}-*.log)，假设容器的日志输出都是stdout以及使用filebeat来收集日志,为了方便维护日志，我们单独创建一个日志收集组件（filebeat）跟app的容器一起运行在同一个pod中。这样，我们就可以在filebeat.yam中根据pod的命名规则，指定自己的日志。

filebeat.yaml:

filebeat.prospectors:
- input_type: log
  paths:
    - "/var/log/containers/${POD_NAME}-*_default_${CONTAINER_NAME}-*.log"
  symlinks: true
  json.message_key: log
  json.keys_under_root: true
  json.add_error_key: true
  multiline.pattern: '^\['
  multiline.negate: false
  multiline.match: after
  document_type: kube-logs
  fields:
    host: ${FILEBEAT_HOST:${HOSTNAME}}
    cluster_name: ${CLUSTER_NAME:default}
  fields_under_root: true

#output.logstash:
#    hosts: ${LOGSTASH_HOSTS:?No logstash host configured. Use env var LOGSTASH_HOSTS to set hosts.}
#    timeout: 15

output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ${KAFKA_URL}
  # message topic selection + partitioning
  topic: '${TOPIC_NAME}'
  partition.round_robin:
    reachable_only: false
  required_acks: ${ACKS:1}
  compression: gzip
  max_message_bytes: 1000000

logging.level: ${LOG_LEVEL:error}

Dockerfile

FROM debian:jessie

ENV FILEBEAT_VERSION 6.2.4

RUN apt-get update && \
    apt-get -y install wget && \
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz && \
    tar xzf filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz && \
    mv filebeat-${FILEBEAT_VERSION}-linux-x86_64/filebeat /usr/local/bin && \
    rm -rf /filebeat* && \
    apt-get -y remove wget && \
    apt-get -y autoremove

COPY filebeat.yml /etc/filebeat/
RUN chmod go-w /etc/filebeat/filebeat.yml

CMD ["/usr/local/bin/filebeat", "-e", "-c", "/etc/filebeat/filebeat.yml"]

build image

build --tag koza/filebeat-kafka:latest .

在k8s中应用(deployment.yaml)：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: testlogs
  namespace: default
  labels:
    app: testlogs
spec:
  template:
    metadata:
      labels:
        app: testlogs
      name: testlogs
    spec:
      containers:
      - name: filebeat
        image: koza/filebeat-kafka:latest
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 50m
            memory: 50Mi
        env:
          - name: KAFKA_URL
            value: "[\"kafka-01.server:9092\",\"kafka-02.server:9092\",\"kafka-03.server:9092\"]"
          - name: TOPIC_NAME
            value: testlogs
          - name: LOG_LEVEL
            value: info
          - name: ACKS
            value: 1
          - name: POD_NAME
            value: testlogs
          - name: CONTAINER_NAME
            value: mylogs
          - name: CLUSTER_NAME
            value: my_cluster
          - name: FILEBEAT_HOST
            valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumeMounts:
        - name: varlog
          mountPath: /var/log/containers
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true

      - name: mylogs
        image: koza/echo-message:latest
        imagePullPolicy: IfNotPresent

      volumes:
      - name: varlog
        hostPath:
          path: /var/log/containers
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

5.备注

1.容器日志输出到stdout,也会写一份json-file.log ,如果定义log-driver为none，就不会记录日志,docker logs 容器名称，也不会查询到日志。
2.k8s中每个POD中的container的hostname跟POD name是一样的。
3.filebeat 的配置文件filebeat.yml 支持读取系统的环境变量,像${POD_NAME}、${CONTAINER_NAME}都是从环境变量传入的变量（k8s可以定义env来传入变量）。

参考：

kubernetes日志架构(建议先看看这篇文章)：https://kubernetes.io/docs/concepts/cluster-administration/logging/

filebeat处理k8s stdout日志：https://github.com/ApsOps/filebeat-kubernetes

https://hub.docker.com/r/apsops/filebeat-kubernetes/

filebeat处理文件日志：https://jimmysong.io/posts/kubernetes-filebeat/

官方教程：https://github.com/elastic/beats/tree/master/deploy/kubernetes/filebeat

kubernetes日志存储机制： https://blog.csdn.net/liukuan73/article/details/79015610