ELK+Filebeat + kibana 容器化

最新推荐文章于 2024-08-02 15:47:16 发布

运维涛涛

最新推荐文章于 2024-08-02 15:47:16 发布

阅读量415

点赞数

分类专栏： docker elk+filebeat+kibana 文章标签： docker

本文链接：https://blog.csdn.net/weixin_46545831/article/details/112967747

版权

docker elk+filebeat+kibana 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

声明

本文是以下面的文章知识作为基础的
1.filebeat+ logstash容器化
2.elaticsearch 容器化

工作流程图

在这里插入图片描述
编写Dockerfile
创建环境
先为这个小测试创建一个目录，作为各个容器的上下文环境

1 | mkdir  -p elk/{filebeat,logstash,elasticsearch}
2 | cd elk/

创建自定义的网络
当在一台宿主机上创建多个可以网络互通的容器时，建议不要使用默认的网桥，
因为我们之前在基础部分说过，默认的网桥，容器之间的通信需要使用彼此的 IP，这样比较麻烦。
所以我们使用自己创建的网桥，就可以使用彼此的容器名进行互相通信了，容器的名称会被转换为容器的主机名。

1 | docker network create -d bridge elk-net   #创建
2 | docker network ls |grep elk-net           # 查看
3 | docker ps |grep ela
4 | docker stop 2097af7a9e7e b67664c72256
5 | docker rm 2097af7a9e7e b67664c72256   #  删除之前创建的logstash和filebeat

运行容器

1 | docker run -d --rm --name=elasticsearch --network=elk-net -e "discovery.type=single-node" -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:7.10.2

-d 后台运行此容器
–rm 由于是测试，所以当停止这个容器的时候同时删除这个容器
–network 连接到指定的网络
-p 映射端口到宿主机

检查 elasticserch 集群状态

1 | curl http://127.0.0.1:9200/_cat/health   # 输出结果如下图   ok！

在这里插入图片描述

2.logstash

a.准备管道配置文件
下面的配置是监听本机的 5044 端口接收 Fielbeat 的输入
并且将处理清洗过的事件数据输出到 elasticsearch

1 | cd elk/
2 | vim logstash/logstash_nginx.conf

代码如下

input {
  beats {
    port => 5044
    host => "0.0.0.0"
  }
}
filter {
  if ([fileset][module] == "nginx") {
    if ([fileset][name] == "access") {
      grok {
        match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
        remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
      }
    }
    else if [fileset][name] == "error" {
      grok {
        match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
        remove_field => "message"
      }
      mutate {
        rename => { "@timestamp" => "read_timestamp" }
      }
      date {
        match => [ "[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
        remove_field => "[nginx][error][time]"
      }
    }
  }
}
output {
  elasticsearch {
    hosts => ["elasticsearch:9200"] 
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }  # 假如有问题，可以同时进行调试
}

b.Dockerfile

1 | cd elk/logstash/
2 | vim Dockerfile
  FROM docker.elastic.co/beats/filebeat:7.10.0
  COPY ./filebeat./sky.log /example.log
  COPY ./filebeat/filebeat.yml  /usr/share/filebeat/filebeat.yml

3 | docker build -t elk_logstash .
4 | docker run -d --rm --name=logstash71 --network=elk-net -v $PWD/logstash/logstash_nginx.conf:/usr/share/logstash/pipeline/logstash.conf docker.elastic.co/logstash/logstash:7.10.2

3.filebeat

1 | cd elk/filebeat/
2 | vim sky.log

a.日志文件如下

95.213.177.126 - - [18/Jul/2017:00:01:09 +0800] "POST http://check.proxyradar.com/azenv.php HTTP/1.1" 404 326 "https://proxyradar.com/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)" "-"
202.108.211.56 - - [18/Jul/2017:00:03:23 +0800] "GET http://1.1.1.1/ HTTP/1.1" 200 6228 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21" "-"
221.228.109.90 - - [18/Jul/2017:01:52:17 +0800] "GET http://www.sharkyun.com/ HTTP/1.1" 200 6228 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:17 +0800] "GET http://www.sharkyun.com/css/style_eeoweb.css HTTP/1.1" 200 11988 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:18 +0800] "GET http://www.sharkyun.com/mobile/js/deviceType.js HTTP/1.1" 200 1055 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:18 +0800] "GET http://www.sharkyun.com/js/jplayer/skin/black/css/style.css HTTP/1.1" 200 3339 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:18 +0800] "GET http://www.sharkyun.com/js/index_eeoweb.js HTTP/1.1" 200 910 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:18 +0800] "GET http://www.sharkyun.com/js/easySlider.js HTTP/1.1" 200 2431 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:18 +0800] "GET http://www.sharkyun.com/js/require_eeoweb.js HTTP/1.1" 200 7161 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"
221.228.109.90 - - [18/Jul/2017:01:52:18 +0800] "GET http://www.sharkyun.com/js/jquery.js HTTP/1.1" 200 46467 "https://www.sharkyun.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "119.61.20.114"

b.准备配置文件

1 | cd elk/filebeat/
2 | vim filebeat.yml
filebeat.inputs:
- type: log
  paths:
    - /*.log

output.logstash
  hosts: ["logstash71:5044"]

构建和运行

1 | docker build -t filebeat:1.0 .
2 | docker run -d  filebeat:1.0

4.kibana

a. 默认的配置
kibana 的容器其实处于测试性的目的就可以直接运行了。
因为默认的配置文件中集群的 url 就是 http://elasticsearch:9200
下面是容器内默认的配置文件内容
/usr/share/kibana/config/kibana.yml

运行容器

1 | docker run -d  --name=kibana71 --network=elk-net -p 8090:5601 docker.elastic.co/kibana/kibana:7.10.2

检查是否自动发现了我们创建的索引
使用浏览器访问宿主机的 5601 端口
比如 http://127.0.0.1:5601

1 | curl http://127.0.0.1:5601
HTTP/1.1 302 Found
location: /spaces/enter
kbn-name: kibana
kbn-license-sig: ba0fe4d935c514ddecae8d90f73cdb6e886bfe16d04ecc1bdc01cedaf0e15766
cache-control: private, no-cache, no-store, must-revalidate
content-length: 0
Date: Wed, 20 Jan 2021 14:29:15 GMT
Connection: keep-alive

去浏览器使用自己服务器ip+5601
在这里插入图片描述