ELK技术4--收集docker日志+kibana自定义画图

一.架构图

二.安装docker

1.安装依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2
2.yum源准备
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
yum makecache fast
3.安装docker-ce
yum install docker-ce
4.启动docker服务
systemctl start docker

使得下载镜像更加快速
阿里云Docker-hub
https://cr.console.aliyum.com/cn-hangzhou/mirrors

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
	"registry-mirrors": ["https://uoggbpok.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker

三.运行nginx镜像

1.运行nginx镜像

docker pull nginx
[root@m01 ~]#  docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS        PORTS               NAMES
[root@m01 ~]# systemctl stop nginx
[root@m01 ~]# docker run --name nginx -p 80:80 -d nginx
0e0c9217852da129259f83714a6115576d797dfbe68c58a91bd2afa982068cb6
[root@m01 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS             PORTS                NAMES
0e0c9217852d        nginx               "nginx -g 'daemon of…"   17 seconds ago      Up 15 seconds       0.0.0.0:80->80/tcp   nginx
[root@m01 ~]# docker start 0e0c9217852d
0e0c9217852d
[root@m01 ~]# docker exec -it 0e0c9217852d /bin/bash
root@0e0c9217852d:/# cat /etc/debian_version
10.4
root@0e0c9217852d:/# 

http://10.0.0.51/

2.查看日志

[root@m01 ~]# docker logs -f nginx
10.0.0.1 - - [19/Jul/2020:00:52:27 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "-"

日志存放的地方：
[root@m01 ~]# cd /var/lib/docker/containers
[root@m01 containers]# ll
total 0
drwx------ 4 root root 237 Jul 19 08:43 244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1
[root@m01 containers]# cd  244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1
[root@m01 244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1]# ll
total 28
-rw-r----- 1 root root 1952 Jul 19 08:52 244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1-json.log

四.配置filebeat通过标签收集多个容器日志

假如我们有多个docker镜像或者重新提交了新镜像，那么直接指定ID的就不是太方便了。
我们从当前的容器提交一个新的镜像并且运行起来
docker commit nginx nginx:v2
docker images
docker run --name nginx-v2 -p 8080:80 -d nginx:v2
此时我们的容器目录下就有了两个不同的容器目录
[root@m01 ~]# ls /var/lib/docker/containers/
244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1
f1ed338bba38a7dc45524d03a8ce2b47dcd243d0b310315ad459faa3776e1064
如果直接配置filebeat存到es里本台机器所有的容器日志都会混在一起没有办法区分
多容器日志收集处理：
其实收集的日志本质来说还是文件，而这个日志是以容器-json.log命名存放在默认目录下的json格式的文件：
[root@m01 ~]# head -1 /var/lib/docker/containers/244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1/244b956d56659374b72c392f38dbd0290ad74094bb68ed0d2312635f79c6aeb1-json.log 
{"log":"10.0.0.1 - - [19/Jul/2020:00:52:31 +0000] \"GET /favicon.ico HTTP/1.1\" 404 555 \"http://10.0.0.51/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2020-07-19T00:52:31.714101121Z"}

但是每个容器的ID都不一样，为了区分不同服务运行的不同容器，可以使用docker-compose通过给容器添加labels标签来作为区分
然后filbeat把容器日志当作普通的json格式来解析并传输到es

1.安装docker-compose

yum install -y python2-pip

2.pip安装

默认源为国外，可以使用国内加速，相关网站
https://mirrors.tuna.tsinghua.edu.cn/help/pypi/
pip加速操作命令
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

3.继续安装docker-compose

[root@m01 ~]# yum install -y gcc
[root@m01 ~]# yum install -y libffi-devel python-devel openssl-devel
[root@m01 ~]#pip install docker-compose

4.检查

docker-compose version

5.编写docker-compose.yml

[root@m01 ~]# vi docker-compose.yml （注意要用vi而不是vim，vim复制的代码会乱）
version: '3'
services:
  nginx:
    image: nginx:v2
    # 设置labels
    labels:
      service: nginx
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"
  db:
    image: nginx:latest
    # 设置labels
    labels:
      service: db 
    # logging设置增加labels.service
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"

6.清理镜像

[root@m01 ~]#  docker stop nginx
nginx
[root@m01 ~]#  docker stop nginx-v2
nginx-v2
[root@m01 ~]# docker rm nginx
nginx
[root@m01 ~]# docker rm nginx-v2
nginx-v2 
[root@m01 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS        PORTS               NAMES

7.运行docker-compose.yml

docker-compose up 
会卡住
打开另一个窗口：
[root@m01 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS             PORTS                  NAMES
17ff5d3bc2d2        nginx:latest        "nginx -g 'daemon of…"   7 minutes ago       Up 2 seconds        0.0.0.0:80->80/tcp     root_db_1
0624dc20ff14        nginx:v2            "nginx -g 'daemon of…"   7 minutes ago       Up 2 seconds        0.0.0.0:8080->80/tcp   root_nginx_1
[root@m01 ~]# systemctl restart docker 

访问：
http://10.0.0.51/nginx

http://10.0.0.51:8080/db

8.检查日志是否增加了lable标签

[root@m01 ~]# cd /var/lib/docker/containers/
[root@m01 containers]# ll
total 0
drwx------ 4 root root 237 Jul 19 10:58 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009
drwx------ 4 root root 237 Jul 19 10:58 9838b5324afb43b6544afcef841429db9176eeb89025d7586231d2513d19302d
[root@m01 containers]# cd 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009
[root@m01 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009]# ll
total 44
-rw-r----- 1 root root 18732 Jul 19 10:58 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009-json.log
[root@m01 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009]# cat 47174401e9bfe79d53accb4199233a65c1d6806f328224a4bc32d69739011009-json.log#查看得到日志

9.配置filebeat通过标签收集多个容器日志

[root@m01 ~]# cat /etc/filebeat/filebeat.yml    
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        attrs.service: "nginx"
    - index: "docker-db-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        attrs.service: "db"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

五.配置filebeat通过服务类型和日志类型多条件创建不同索引

目前为止，已经可以按服务来收集日志了，但是错误日志和正确日志混在了一起，不好区分，所以可以进一步进行条件判断，根据服务和日志类型创建不同的索引
filebeat配置文件开始：docker容器拆解日志
[root@elk-176 ~]# cat /etc/filebeat/filebeat.yml        
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "nginx"
          stream: "stdout"
    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "nginx"
          stream: "stderr"
    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "db"
          stream: "stdout"
    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
          attrs.service: "db"
          stream: "stderr"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
[root@m01 ~]# systemctl restart filebeat

查看：

http://10.0.0.51:5601

六.使用fillebeat modules配置

1.实验准备

[root@m01 ~]# >/etc/filebeat/filebeat.yml
配置文件
[root@m01 ~]# vi /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "docker-access-%{[beat.version]}-%{+yyyy.MM}"
[root@m01 ~]# filebeat modules list

2.激活nginx模块

（2种方法）
查看状态：
[root@m01 ~]# cd /etc/filebeat/modules.d/
[root@m01 modules.d]# ll
第一种：
[root@m01 ~]# filebeat modules enable nginx
Enabled nginx
[root@m01 ~]# filebeat modules list
Enabled:
nginx
第二种：
[root@m01 modules.d]# mv nginx.yml.disabled nginx.yml
[root@m01 modules.d]# filebeat modules list
Enabled:
nginx

3.修改nginx日志为普通格式

[root@m01 modules.d]# vim /etc/nginx/nginx.conf
access_log  /var/log/nginx/access.log  main;
出现问题：
[root@m01 modules.d]# systemctl start nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
解决方法：
[root@m01 modules.d]# systemctl stop docker
[root@m01 modules.d]# systemctl disable docker
[root@m01 modules.d]# iptables -F
[root@m01 modules.d]# iptables -X
[root@m01 modules.d]# iptables -Z
[root@m01 modules.d]# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[root@m01 modules.d]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@m01 modules.d]# systemctl start nginx

4.验证

[root@m01 modules.d]# curl 10.0.0.51
[root@m01 modules.d]# tail -f /var/log/nginx/access.log
10.0.0.51 - - [19/Jul/2020:11:26:53 +0800] "GET / HTTP/1.1" 200 4833 "-" "curl/7.29.0" "-"
[root@m01 modules.d]# ab -n 20 -c 20 http://10.0.0.51/test
配置文件
[root@m01 modules.d]# vi nginx.yml
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]
"nginx.yml" 16L, 426C written
[root@m01 modules.d]# vim /etc/filebeat/filebeat.yml
#index: "docker-access-%{[beat.version]}-%{+yyyy.MM}"注释掉这一行
[root@m01 modules.d]# systemctl restart filebeat
[root@m01 modules.d]# find / -name "elasticsearch-plugin"
/usr/share/elasticsearch/bin/elasticsearch-plugin
[root@m01 modules.d]# /usr/share/elasticsearch/bin/elasticsearch-plugin  install ingest-user-agent
[root@m01 modules.d]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[root@m01 modules.d]# systemctl restart filebeat

[root@m01 modules.d]# vi /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

  setup.kibana:
    host: "10.0.0.51:5601"
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
  - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.contains:
        fileset.name: "access"
  - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.contains:
        fileset.name: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

"/etc/filebeat/filebeat.yml" 23L, 558C written
[root@m01 modules.d]# systemctl restart nginx
[root@m01 modules.d]# systemctl restart elasticsearch
[root@m01 modules.d]# systemctl restart filebeat

查看：

七.导入kibana视图

1.实验配置

默认如果使用filbeat模版导入视图会把所有的服务都导入进去,而我们实际上并不需要这么多视图，
而且默认的视图模版只能匹配filebeat-*开头的索引，所以这里我们有2个需要解决：
1.通过一定处理只导入我们需要的模版
2.导入的视图模版索引名称可以自定义
解决方法：
1.备份一份filebeat的kibana视图，删除不需要的视图模版文件
2.修改视图文件里默认的索引名称为我们需要的索引名称
[root@m01 ~]# cp -a /usr/share/filebeat/kibana /root
[root@m01 ~]# cd kibana/
[root@m01 kibana]# rm -rf 5
[root@m01 kibana]# cd 6/dashboard/
[root@m01 dashboard]# find . -type f ! -name "*nginx*"|xargs rm -rf
[root@m01 dashboard]# rm -fr ml-nginx-*
[root@m01 dashboard]# sed -i 's#filebeat\-\*#nginx\-\*#g' Filebeat-nginx-overview.json
[root@m01 dashboard]# sed -i 's#filebeat\-\*#nginx\-\*#g' Filebeat-nginx-logs.json
[root@m01 dashboard]# cd ..
[root@m01 6]# ls
dashboard  index-pattern
[root@m01 6]# cd index-pattern/
[root@m01 index-pattern]# sed -i 's#filebeat\-\*#nginx\-\*#g' filebeat.json
[root@m01 index-pattern]# cd ..
[root@m01 6]# ls
dashboard  index-pattern
[root@m01 6]# cd
[root@m01 ~]# filebeat setup --dashboards -E setup.dashboards.directory=/root/kibana/

2.使用kibana画图

画图实验准备：
[root@m01 ~]# >/var/log/nginx/access.log
上传nginx_access.tar.gz（准备好的日志文件）
[root@m01 ~]# tar -zxvf nginx_access.tar.gz
var/log/nginx/access.log
[root@m01 ~]# mv var/log/nginx/access.log /var/log/nginx/access.log
[root@m01 ~]#vi /etc/filebeat/filebeat.yml
[root@m01 ~]# cat  /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

setup.kibana:
  host: "10.0.0.51:5601"

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  #index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

[root@m01 ~]# systemctl restart filebeat
[root@m01 ~]# systemctl restart nginx

实验操作：

删除之前的项目

新建一个项目

把之前的模板删除，自己画图

调整ip地址方向


保存画图