filebeat使用docker-compose启动，收集docker和nginx日志

安装elasticsearch

# docker-compose.yml
version: '3'
services:
  es:
    image: "elasticsearch:7.12.0"
    container_name: es
    restart: always
    volumes:
      - ./data:/usr/share/elasticsearch/data
      - ./logs:/user/share/elasticsearch/logs
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - default

  kibana:
      image: kibana:7.12.0
      container_name: kibana
      environment:
        - ELASTICSEARCH_HOSTS=http://es:9200
        - I18N_LOCALE=zh-CN
        - xpack.monitoring.ui.container.elasticsearch.enabled=false
      ports:
        - 5601:5601
      restart: always
      networks:
        - default

networks:
  default:
    external:
      name: mynet

安装filebeat

# docker-compose.yml
version: '3'
services:
  nginx:
    image: "docker.elastic.co/beats/filebeat:7.12.0"
    container_name: filebeat
    restart: always
    privileged: true
    user: root
    volumes:
      - ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
      - /root/nginx/nginx/log:/root/nginx/nginx/log
      - /var/lib/docker/containers:/var/lib/docker/containers
      - ./modules.d:/usr/share/filebeat/modules.d
      - ./inputs.d:/usr/share/filebeat/inputs.d
      - ./logs:/usr/share/filebeat/logs
      - ./data:/usr/share/filebeat/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - default
networks:
  default:
    external:
      name: mynet

这个是filebeat的配置文件
参考：https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

# filebeat.docker.yml
filebeat.config.inputs:
  enabled: true
  #这里可以自定义一些要录入的日志
  path: ${path.config}/inputs.d/*.yml

filebeat.config:
  modules:
    #这里是module，一些内置的模块，在这里开启
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              #你自己的容器的名称
              docker.container.name: container_name
          config:
            - type: log
              paths:
              #这里会自动监听容器下的日志
                - /var/lib/docker/containers/${data.docker.container.id}/*-json.log
              #这里可以自定义保存日志的索引名称
              index: "%{[agent.name]}-chinew-%{+yyyy.MM.dd}"
              #加上tag，方便搜索
              tags: ["docker", "chipro"]

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: 'es:9200'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

setup.kibana:
  host: "kibana:5601"

使用nginx模块配置收集nginx日志
参考官方文档：https://www.elastic.co/guide/en/beats/filebeat/7.12/filebeat-module-nginx.html

# modules.d/nginx.yml
- module: nginx
  access:
    enabled: true
    var.paths: ["/root/nginx/nginx/log/*.access.log"]
  error:
    enabled: true
    var.paths: ["/root/nginx/nginx/log/*.error.log"]

问题

有可能监听了nginx或者docker的日志，但是没有收集成功，
这时候，可能需要执行下
chmod 777 -R /var/lib/docker/containers/
chmod 777 -R /root/nginx/nginx/log/

这样改智能临时获得权限，如果容器重启，依旧没有权限
正确的做法是在
filebeat的docker-compose里面，使用 user: root，使得filebeat启动用户是root

如果要收集所有docker容器的，配置

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

如果只需要收集部分容器日志，则配置多个condition即可

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: chinew_ney_app_1
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*-json.log

        - condition:
            contains:
              docker.container.name: auth
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*-json.log

如果需要将收集到的容器日志，放到自己命名的index怎么弄
参考：https://www.elastic.co/guide/en/beats/filebeat/7.12/filebeat-input-container.html

indexedit
If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: “%{[agent.name]}-myindex-%{+yyyy.MM.dd}” might expand to “filebeat-myindex-2019.11.01”.

publisher_pipeline.disable_hostedit
By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

也就是只需要自己定义index名称规则就行，命名还可以使用容器名称

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: chinew_ney_app_1
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*-json.log
              index:
                %{[agent.name]}-myindex-${data.docker.container.name}-%{+yyyy.MM.dd}