大数据日志监控filebeat+kafka+elk+docker部署

部署架构

日志格式说明

系统日志

{"file":"entry.go:224","level":"info","msg":"set sensive words: [呵呵呵呵 测试一下 你好]","time":"2020-04-24T11:20:32+08:00"}

访问日志，包含登录日志（请求类型值为2则为登录日志，登录日志才有额外信息）

# IP - 用户名 [时间] "方法 路径 协议" 响应状态码 内容大小 "refere" "User-Agent" 请求次数 "路由名称" "后端服务地址" 花费时间ms "后端服务名称" "参数" 请求类型 |||请求内容||| |||响应内容||| |||额外信息|||  

#例子
127.0.0.1 - admin [07/May/2020:08:52:52 +0000] "GET /login HTTP/1.1" 200 711 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.14 Safari/537.36 Edg/83.0.478.13" 18 "old@internal" "-" 57ms "-" "username=admin&password=123456" 2 |||-||| |||"{\"meta\":{\"success\":true,\"statusCode\":200,\"message\":\"\"},\"data\":{\"userId\":\"admin\",\"userName\":\"超级管理员\",\"userNickname\":\"联奕大学管理员\",\"email\":\"123456@qq.com\",\"orgId\":\"1\",\"departmentId\":\"000000\",\"departmentName\":\"院领导\",\"userType\":\"teacher\",\"sex\":\"1\",\"skin\":\"blue\",\"startTimestamp\":\"2020-05-07T16:52:52.9985147+08:00\",\"lastAccessTime\":\"2020-05-07T16:52:52.9985147+08:00\",\"authorization\":{},\"menus\":{},\"apps\":{\"ly-gtc\":\"ly-gtc\"},\"accessToken\":\"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODg4NDg3NzIsImlhdCI6MTU4ODg0MTU3MiwidXNlcm5hbWUiOiJhZG1pbiIsImFjY2Vzc1Rva2VuIjoiIn0.z1ZAQpcivtLD81vKy1Nvi_7NRw25wMLDPhv-2glMb5k\",\"refreshToken\":\"dca4655e-3ec2-4c91-9864-f440748ff934\",\"expiresIn\":7200}}\n"||| |||{"username":"超级管理员","usertype":"teacher","login_access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODg4NDg3NzIsImlhdCI6MTU4ODg0MTU3MiwidXNlcm5hbWUiOiJhZG1pbiIsImFjY2Vzc1Rva2VuIjoiIn0.z1ZAQpcivtLD81vKy1Nvi_7NRw25wMLDPhv-2glMb5k"}|||

安全日志

{"clientAddr":"127.0.0.1:24377","clientHost":"127.0.0.1","clientPort":"24377","clientUsername":"-","downstreamContentSize":17,"downstreamStatus":429,"duration":4982900,"originContentSize":17,"originDuration":4982900,"originStatus":429,"overhead":0,"requestAddr":"127.0.0.1:81","requestContentSize":0,"requestCount":16,"requestHost":"127.0.0.1","requestMethod":"GET","requestPath":"/hello/a","requestPort":"81","requestProtocol":"HTTP/1.1","requestScheme":"http","retryAttempts":0,"routerName":"hello@mongodb","startLocal":"2020-05-06T15:03:04.403529+08:00","startUTC":"2020-05-06T07:03:04.403529Z","entryPointName":"web","level":"info","msg":"","request_Body":"\"\"","request_Params":"","request_User_Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.14 Safari/537.36 Edg/83.0.478.13","response_Body":"\"Too Many Requests\"","safety_Info":"127.0.0.1 访问/hello/a 限制为每秒2","safety_Type":"ratelimit","time":"2020-05-06T15:03:11+08:00"}

docker日志

安全日志：logType=1

登录日志：logType=2

操作日志：logType=3

普通日志：除了上面三种情况都是

类型 时间 日志等级 用户名 请求地址和端口 请求参数 请求方法 请求次数 响应状态码 响应信息

#示例
{"logType":1,"time":"2020-05-14T15:03:04.403529+08:00","level":"info","username":"-","requestAddr":"127.0.0.1:81","requestParam":"id=9","requestMethod":"GET","requestCount":16,"status":200,"msg":"Too Many Requests"}
{"logType":2,"time":"2020-05-14T15:03:04.403529+08:00","level":"warn","username":"-","requestAddr":"127.0.0.2:81","requestParam":"id=9","requestMethod":"GET","requestCount":16,"status":200,"msg":"Too Many Requests"}
{"logType":3,"time":"2020-05-14T15:03:04.403529+08:00","level":"error","username":"-","requestAddr":"127.0.0.3:81","requestParam":"id=9","requestMethod":"GET","requestCount":16,"status":200,"msg":"Too Many Requests"}
{"logType":4,"time": "2020-05-14T15:03:04.403529+08:00" "level":"error","username":"-","requestAddr":"127.0.0.3:81","requestParam":"id=9","requestMethod":"GET","requestCount":16,"status":200,"msg":"Too Many Requests"

环境准备

系统环境

Centos7.0及以上

基础软件环境

安装Docker 19及以上版本

软件环境

docker pull filebeat:7.6.1

docker pull wurstmeister/zookeeper:latest
docker pull wurstmeister/kafka:latest

docker pull logstash:7.6.1

docker pull elasticsearch:7.6.1

#可选安装，目的用来查看日志
docker pull kibana:7.6.1

部署过程

Filebeat

在opt/elk/config/目录下新建filebeat.yml

filebeat.yml

# 日志输入配置
#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /home/logs/nginx/*.log
  fields:
    log_topic: nginx_log
    #- c:\programdata\elasticsearch\logs\*

- type: log
  enabled: true
  paths:
    - /opt/elk/logs/*.log
  fields:
    log_topic: traefik_log

- type: log
  enabled: true
  paths:
    - /opt/elk/securityLogs/*.log
  fields:
    log_topic: security_log

#日志输出配置(采用kafka缓冲日志数据)
output.kafka:
  enabled: true
  hosts: ["192.168.30.23:9092"]
  topic: 'filebeat'

input输入配置说明

• type 监控类型

• enabled 是否开启，默认为false，需要设置为true

• paths 指定要监控的日志，可以指定具体得文件或者目录

• fields 向输出的每一条日志添加额外的信息

output输出配置说明

• enabled 是否开启
• hosts kafka的ip和port
• topic 推送到kafka新建的topic

启动

重点：需要监控的日志目录也需要挂载进容器，这样filebeat才能读取到

docker run -d --name=filebeat \
--user=root \
-v /opt/elk/config/filebeat.yml:/usr/share/filebeat/filebeat.yml \
-v /home/logs/nginx/:/home/logs/nginx/ \
-v /opt/elk/logs/:/opt/elk/logs/ \
-v /opt/elk/securityLogs/:/opt/elk/securityLogs/ \
elastic/filebeat:7.6.1 

Kafka（先启动zookeeper）

启动

#zookeeper port:2181
docker run -d --name zookeeper -p 2181:2181 --log-opt max-size=100m wurstmeister/zookeeper:latest

#kafka	port:9092
docker run -d --name kafka --publish 9092:9092 --log-opt max-size=100m \
--link zookeeper \
--env KAFKA_ZOOKEEPER_CONNECT=192.168.30.23:2181 \
--env KAFKA_ADVERTISED_HOST_NAME=192.168.30.23 \
--env KAFKA_ADVERTISED_PORT=9092 \
--volume /etc/localtime:/etc/localtime \
wurstmeister/kafka:latest

为了防止kafka日志文件过大，我们需要设置合适的日志保存策略

server.properties

测试kafak是否安装成功

#进入kafka容器
docker exec -it kafka /bin/bash

#之后进入bin目录
cd /opt/kafka_2.12-2.4.0/bin

#创建test这个topic
./kafka-topics.sh --create --zookeeper 192.168.30.23:2181 --replication-factor 1 --partitions 1 --topic test

#之后创建生产者，运行命令之后，随便输点啥
./kafka-console-producer.sh --broker-list 192.168.30.23:9092 --topic test

#重开一个窗口，新建消费者，运行命令之后，就可以看到生产者发过来的消息了
./kafka-console-consumer.sh --bootstrap-server 192.168.30.23:9092 --topic test --from-beginning

2、使用 kafka-run-class 指令，获取topic的最小offset和最大offset

#查看各个分区的最小offset(这个意思就是，这个offset之前的消息已经被清除了，现在consumer是从这个offset之后开始消费):
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic filebeat --time -2
#查看各个分区的最大offset(这个意思就是，producer下一次写入信息时的offset):
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic filebeat --time -1

3、获取最近N条数据，offset = MaxOffset - N

比如想要获取最近10条数据，根据第二个步骤获取到的MaxOffset为52884，则这条命令最后的--offset参数为52884 - 10 = 52874

./kafka-console-consumer.sh --bootstrap-server 192.168.35.14:9092 --topic filebeat \
	--property print.key=true --partition 0 --offset 52874

elasticsearch

配置

• 需要设置系统内核参数，否则会因为内存不足无法启动

  sysctl -w vm.max_map_count=262144

  使之立即生效

  sysctl -p

创建/opt/elk/elasticsearch.yml配置文件，内容如下：

cluster.name: "elasticsearch"
network.host: 0.0.0.0
#快照仓库地址
path.repo: /opt/elk/es/snapshot

启动

docker run -d --name es -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" -e "cluster.name=elasticsearch" \
-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
-v /opt/elk/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /opt/elk/es/snapshot:/opt/elk/es/snapshot \
--log-opt max-size=100m -d elasticsearch:7.6.1

创建es快照仓库文件夹夹并改变权限：

mkdir -p /opt/elk/es/snapshot
chmod 777 snapshot

#进入容器
docker exec -it es /bin/bash

#安装中文分词插件
elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.1/elasticsearch-analysis-ik-7.6.1.zip
#重启se
docker restart es

验证es是否安装成功(curl ip:9200)

[root@localhost ~]# curl localhost:9200
{
  "name" : "a33593fc9be0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "LyoRC3pKRHmJnigMyr8eeQ",
  "version" : {
    "number" : "7.6.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "aa751e09be0a5072e8570670309b1f12348f023b",
    "build_date" : "2020-02-29T00:15:25.529771Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

logstash

在/opt/elk/logstash目录下新建logstash.conf文件

input {
  kafka {
    bootstrap_servers => "192.168.35.14:9092"
    topics => ["filebeat"]
    group_id => "test-consumer-group"
    codec => "json"
    consumer_threads => 1
    decorate_events => true
  }
}

filter {
  if [fields][log_topic] == "nginx_log"{
    json {
      source => "message"
      remove_field => ["message"]
    }

    date {
      match => ["time", "yyyy-MM-dd HH:mm:ss ZZ", "ISO8601"]
      target => "@timestamp"
    }
  }

  if [fields][log_topic] == "security_log"{
    json {
      source => "message"
      remove_field => ["message"]
    }

    date {
      match => ["time", "yyyy-MM-dd HH:mm:ss ZZ", "ISO8601"]
      target => "@timestamp"
    }
  }

  if [fields][log_topic] == "traefik_log"{  
    
    dissect{
      mapping => {
	"message" => '%{source_address} %{traefik_access_user_identifier} %{user_name} [%{timestamp}] "%{http_request_method} %{url_original} HTTP/%{http_version}" %{http_response_status_code} %{traefik_access_message}'
      }
    }

    grok {
	match => { "traefik_access_message" =>  '(?:%{NUMBER:http_response_body_bytes:long}|-)( (?:"%{DATA:http_request_referrer}"|-)?( (?:"%{DATA:user_agent_original}"|-)?)?( (?:%{NUMBER:traefik_access_request_count:long}|-)?)?( (?:"%{DATA:traefik_access_frontend_name}"|-)?)?( "%{DATA:traefik_access_backend_url}")?( %{NUMBER:temp_duration:long}ms)?)?( (?:"%{DATA:traefik_service_name}"|-)?)?( (?:"%{DATA:appid}"|-)?)?( (?:"%{DATA:traefik_service_paramter}"|-)?)?( %{NUMBER:request_type:long})?( (?:\|\|\|%{DATA:request_body}\|\|\||-)?)?( (?:\|\|\|%{DATA:response_body}\|\|\||-)?)?( (?:\|\|\|%{DATA:extra_info}\|\|\||-)?)?' }
	remove_field => [ "message", "traefik_access_message" ]
    }

    if [request_type] == "2" {
    	json {
      		source => "extra_info"
      		remove_field => ["extra_info"]
    	}	
    } 
    if [request_type] == "3" {
    	json {
      		source => "extra_info"
      		remove_field => ["extra_info"]
    	}	
    } 

    date {
	match => [ "timestamp" , "dd/MMM/yyyy:H:m:s Z" ]
	#remove_field => [ "traefik_access_time" ]
	target => "@timestamp"
    }

    mutate {
      convert => {
        "temp_duration" => "integer" 
        "traefik_access_request_count" => "integer"
        "http_response_status_code" => "integer"
      }
    }

    useragent {
      source=>"user_agent_original"
    }
    grok {
      match => { "source_address" =>  '^(%{IP:source_ip}|%{HOSTNAME:source_domain})$' }
    }

    geoip {
      source=>"source_ip"
      target=>"source_geo"
    }
 
  }
  
}


output {
  stdout{ codec=>rubydebug}
  
  if [fields][log_topic] == "traefik_log"{
	if [request_type] == "2" {
		elasticsearch {
         		hosts => ["192.168.35.14:9200"]
         		index => "login_log-%{+YYYY.MM.dd}"
      		}
	} else if [request_type] == "3" {
		elasticsearch {
         		hosts => ["192.168.35.14:9200"]
         		index => "logout_log-%{+YYYY.MM.dd}"
      		}
	} else {
		elasticsearch {
                        hosts => ["192.168.35.14:9200"]
                        index => "traefik_log-%{+YYYY.MM.dd}"
                }
	}
  }

  if [fields][log_topic] == "nginx_log"{
      	elasticsearch {
        	hosts => ["192.168.35.14:9200"]
         	index => "nginx_log-%{+YYYY.MM.dd}"
      	}
  }

  if [fields][log_topic] == "security_log"{
      	elasticsearch {
         	hosts => ["192.168.35.14:9200"]
         	index => "security_log-%{+YYYY.MM.dd}"
      	}
  }
}

日志说明

nginx_log：网关系统日志

syslog: 服务器系统日志

security_log 安全日志

k8s_container_log：容器日志(包括以下三种日志，根据日志中logType可分类)

• 安全日志：logType=1
• 登录日志：logType=2
• 操作日志：logType=3
• 容器日志：除了以上三种

traefik_log 访问日志（包括以下日志，根据request_type分类）

• 登录日志：request_type=2
• 登出日志：request_type=3
• 访问日志：除了以上都是

解析语法说明：

#将message直接解析，应为message是json格式，解析起来跟简单
json {
	source => "message"
	#解析之后移除message字段
	remove_field => ["message"]
}

#logstash自动生成的timestamp，不是日志的生成时间，需要将日志里的时间提取出来，替换timestamp
date {
	match => ["time", "yyyy-MM-dd HH:mm:ss ZZ", "ISO8601"]
	target => "@timestamp"
}

#重命名、添加字段、移除字段
mutate {
        rename => {
			"timestamp"=>"time" 
		}
		add_field => {
			#这里message是个数组，我们把它转化成字段
			"msg" => "%{[message][0]}" 
		}
		remove_field => ["message"]
}

在/opt/elk/目录下新建logstash.yml，将修改es所在服务器的ip

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://192.168.30.23:9200" ]

启动

docker run -d --name=logstash -p 5044:5044 -p 9600:9600 \
-v /opt/elk/logstash.yml:/usr/share/logstash/config/logstash.yml \
-v /opt/elk/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \
-v /opt/elk/logstash-container.conf:/usr/share/logstash/pipeline/logstash-container.conf \
--log-opt max-size=100m \
logstash:7.6.1

查看logstash启动日志（日志数据推送过来的时候，可以看到具体日志数据）

docker logs -f logstash

---

kibana（可选启动，为了更直观的查看测试日志数据，建议启动）

启动

docker run -d -p 5601:5601 --name kibana -e ELASTICSEARCH_HOSTS=http://192.168.30.23:9200 --log-opt max-size=100m kibana:7.6.1

效果

上传日志文件测试，在浏览器可以查看elasticsearch中各个索引库的数据，索引库的生成在logstash.conf配置文件output模块，输出到es，会生成对应的索引库

一：在浏览器中输入ip:port/index/_search

ip：es的ip

port：一般默认为9200

index：es的索引库

如图：

二：使用kibana查看效果

输入ip:5601