- 环境说明
按照之前的文章，搭建2台CentOS6.5 服务器，主机 172.16.1.176 安装ELK+redis并配置好，主机 172.16.1.177 安装EL并配置
附带一下本文Elasticsearch 的配置文件
[root@elk logstash]# grep '^[a-z]' /usr/share/elasticsearch-6.2.2/config/elasticsearch.yml
cluster.name: li-application #集群名字
node.name: linux-1 #节点名称
path.data: /data/es-data #数据存储路径
path.logs: /var/log/es-log #日志存放路径
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
network.host: 172.16.1.176
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.1.176", "172.16.1.177"] #集群的配置，这里选择单播
http.cors.enabled: true
http.cors.allow-origin: "*"
2.配置插件HEAD
HEAD 插件主要可以比较直观看到Elasticsearch集群状态，拓扑，分片，以及数据浏览，测试调用等一些功能
这里将HEAD插件安装到176主机上
https://github.com/mobz/elasticsearch-head
- 安装NodeJS
yum install -y nodejs- 安装npm
npm install -g cnpm --registry=https://registry.npm.taobao.org- 使用npm安装grunt
npm install -g gruntnpm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy- 下载head
https://github.com/mobz/elasticsearch-head/archive/master.zip
wget https://github.com/mobz/elasticsearch-head/archive/master.zip- 解压插件，然后配置
unzip master.zip #解压插件包
npm install #安装一下依赖
#先停止ELasticsearch
vi $ES_HOME/config/elasticsearch.yml #修改一下es的配置文件，$ES_HOME为你安装ES的路径，本人笔记默认在/usr/share/elasticsearch-6.2.2/，添加如下内容：
http.cors.enabled: true
http.cors.allow-origin: "*"
vi Gruntfile.js #修改Head插件文件夹里的配置文件 ，修改如下内容：
connect: {
server: {
options: {
hostname: '172.16.1.176',
port: 9100,
base: '.',
keepalive: true
}
}
}
/usr/share/elasticsearch-6.2.2/bin/elasticsearch #重新将ES启动起来- 启动HEAD插件
进入head插件根目录，使用命令 grnut server 跑起来
或者通过命令 npm run start 也可以启动 head
[root@elk elasticsearch-head-master]# grunt server
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://172.16.1.176:9100打开浏览器测试访问：http://172.16.1.176:9100/
如上图，两个es节点均成功建立，集群状态右上角显示，安装成功。
3.安装redis
redis消息队列作用说明：1、防止Logstash和ES无法正常通信，从而丢失日志。
2、防止日志量过大导致ES无法承受大量写操作从而丢失日志。
3、应用程序（php，java）在输出日志时，可以直接输出到消息队列，从而完成日志收集。
补充：如果redis使用的消息队列出现扩展瓶颈，可以使用更加强大的kafka，或者mq来代替。
[root@elk ~]# yum install -y redis
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* epel: mirror01.idc.hinet.net
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Setting up Install Process
Package redis-3.2.11-1.el6.x86_64 already installed and latest version
Nothing to do
本文使用yum安装，版本 3.2.11
修改一下redis 的配置文件，修改如下内容
[root@elk ~]# vim /etc/redis.conf
bind 172.16.1.176 #绑定IP
port 6379 #端口
daemonize yes #后台运行开启Redis
[root@elk ~]# service redis start
启动 ： [确定]
检查redis服务运行正常，使用netstat 查看监听端口，使用 redis-cli -h 连接redis
[root@elk ~]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.16.1.176:6379 0.0.0.0:* LISTEN 14270/redis-server
[root@elk ~]# redis-cli -h 172.16.1.176
172.16.1.176:6379> info
# Server
redis_version:3.2.11
redis_git_sha1:00000000
redis_git_dirty:04.logstash 收集各类日志
4.1 收集syslog 日志 （syslog input ；redis output input；）
- 在172.16.1.176 主机上 创建logstash 配置文件 ： shipper.conf 用作读取各类日志，写入redis
[root@elk logstash]# vim shipper.confinput{
syslog {
type => "system" #定义一个type 用于下方output的时候做匹配
host => "172.16.1.176" #syslog的IP地址
port => "514" #syslog端口
}
}
output{
if [type] == "system" {
redis {
host => "172.16.1.176" #redis的地址
port => "6379" #端口
db => "3" #创建一个数据库3
data_type => "list" #以list存
key => "system" #key名称，自己定义，最好对应上日志的类型
}
}
}
input 使用 syslog plugins 收集
syslog plugins介绍：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html
output 使用 redis plugins 输出到 redis中
redis plugins 介绍 ：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html
- 修改本机syslog配置
[root@elk logstash]# vim /etc/rsyslog.conf# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@172.16.1.176:514修改remote-host 为本机IP地址
重启rsyslog 进程生效
[root@elk logstash]# service rsyslog restart
关闭系统日志记录器： [确定]
启动系统日志记录器： [确定]
[root@elk logstash]# 以配置文件shipper.conf 运行logstash （ctrl+c 可以终止运行）
[root@elk logstash]# bin/logstash -f shipper.conf 使用 logger 命令 手动产生一些日志 ，然后进入redis查看是否生效
[root@elk ~]# logger "test LOG"
[root@elk ~]# redis-cli -h 172.16.1.176
172.16.1.176:6379> SELECT 3
OK
172.16.1.176:6379[3]>
172.16.1.176:6379[3]> KEYS *
1) "system"
172.16.1.176:6379[3]> LINDEX system -1
"{\"facility\":1,\"facility_label\":\"user-level\",\"timestamp\":\"Mar 6 16:18:04\",\"program\":\"root\",\"type\":\"system\",\"severity\":5,\"@timestamp\":\"2018-03-06T08:18:04.000Z\",\"@version\":\"1\",\"host\":\"172.16.1.176\",\"logsource\":\"elk\",\"message\":\"test LOG\\n\",\"severity_label\":\"Notice\",\"priority\":13}"
172.16.1.176:6379[3]> exit
数据写入成功
- 在172.16.1.177 主机上 创建logstash 配置文件 ： indexer.conf 用作 从redis读取数据，写入es
创建配置文件indexer.conf
[root@elk2 logstash]# vim indexer.confinput{
redis {
host => "172.16.1.176" #redis主机地址 ,下面的配置对应上172.16.1.176 主机写入redis的配置
port => "6379" #端口
db => "3" #数据库名字
data_type => "list" #数据类型
key => "system"
type => "system"
}
}
output{
if [type] == "system" {
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "system-%{+YYYY.MM.dd}" #使用system+ 日期，这个格式的索引
}
}
}
input 使用 redis plugins 收集
redis plugins介绍：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html
既然ELK 是三个好基友，output当然使用 esplugins 输出到 es 中
es plugins 介绍：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
在主机172.16.1.177 将logstash 跑起来 （ctrl+c 可以终止运行）
[root@elk2 logstash]# bin/logstash -f indexer.conf 回到redis主机，可以看到redis的数据已经给消化了
[root@elk ~]# redis-cli -h 172.16.1.176
172.16.1.176:6379> SELECT 3
OK
172.16.1.176:6379[3]> KEYS *
(empty list or set)
172.16.1.176:6379[3]>
进入es 的head 插件页面，可以看到数据已经写入了es 中
登录kibana，创建一个默认的索引 ： system*
使用time作为数据过滤
数据日志已经呈现在kibana中了
4.2 收集 nginx 日志（file input plugin）
本小节，file input plugin 的文档： https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html- 将nginx日志格式化为json ，方便logstash收集
默认access.log的日志格式为：
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
修改成json格式输出，我们需要新增log格式配置，修改nginx 的配置文件
[root@elk logstash]# vim /etc/nginx/nginx.conf将配置添加到配置文件
log_format access_log_json '{"user_ip":"$remote_addr","@timestamp":"$time_iso8601","user_req":"$request","http_code":"$status","body_bytes_sents":"$body_bytes_sent","referer":"$http_referer","responsetime":"$request_time","user_ua":"$http_user_agent"}';
access_log /var/log/nginx/access.log access_log_json;http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format access_log_json '{"user_ip":"$remote_addr","@timestamp":"$time_iso8601","user_req":"$request","http_code":"$status","body_bytes_sents":"$body_bytes_sent","referer":"$http_referer","responsetime":"$request_time","user_ua":"$http_user_agent"}';
access_log /var/log/nginx/access.log access_log_json;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
}保存并重新加载nginx配置
[root@elk logstash]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@elk logstash]# nginx -s reload
测试访问，查看访问日志
[root@elk logstash]# cat /var/log/nginx/access.log
{"user_ip":"172.16.38.18","@timestamp":"2018-03-07T10:44:27+08:00","user_req":"GET /nginx-logo.png HTTP/1.1","http_code":"200","body_bytes_sents":"368","referer":"http://172.16.1.176/","responsetime":"0.000","user_ua":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"}
{"user_ip":"172.16.38.18","@timestamp":"2018-03-07T10:44:27+08:00","user_req":"GET /poweredby.png HTTP/1.1","http_code":"200","body_bytes_sents":"2811","referer":"http://172.16.1.176/","responsetime":"0.000","user_ua":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"}
{"user_ip":"172.16.38.18","@timestamp":"2018-03-07T10:44:28+08:00","user_req":"GET /favicon.ico HTTP/1.1","http_code":"404","body_bytes_sents":"3652","referer":"http://172.16.1.176/","responsetime":"0.000","user_ua":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"}
修改172.16.1.176 主机的 logstash 配置文件 shipper.conf （本文放在 /usr/share/logstash ）
[root@elk logstash]# vim shipper.confinput{
syslog {
type => "system" #定义一个type 用于下方output的时候做匹配
host => "172.16.1.176" #syslog的IP地址
port => "514" #syslog端口
}
file { #以文件形式读取
path => "/var/log/nginx/access.log" #文件路径
codec => "json" #利用codec插件，读取json数据
start_position => "beginning" #记录位置，从头
type => "nginx-log"
}
}
output{
if [type] == "system" { #写入system日志到redis
redis {
host => "172.16.1.176" #redis的地址
port => "6379" #端口
db => "3" #创建一个数据库3
data_type => "list" #以list存
key => "system" #key名称，自己定义，最好对应上日志的类型
}
}
if [type] == "nginx-log" { #写入nginx日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log"
}
}
}修改172.16.1.176 主机上的 logstash 配置文件 indexer.conf
[root@elk2 logstash]# pwd
/usr/share/logstash
[root@elk2 logstash]# vim indexer.confinput{
redis {
host => "172.16.1.176" #redis主机地址 ,下面的配置对应上172.16.1.176 主机写入redis的配置
port => "6379" #端口
db => "3" #数据库名字
data_type => "list" #数据类型
key => "system"
type => "system"
}
redis { #从redis读取nginx日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log" #需要对应上redis的key
type => "nginx-log"
}
}
output{
if [type] == "system" { #写入system 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "system-%{+YYYY.MM.dd}" #使用system+ 日期，这个格式的索引
}
}
if [type] == "nginx-log" { #写入nginx 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
}在172.16.1.176 以及 172.16.1.177 主机上，分别将logstash跑起
[root@elk logstash]# bin/logstash -f shipper.conf[root@elk2 logstash]# bin/logstash -f indexer.conf访问一下nginx产生日志，日志记录在kibana中可以看到了
4.3 收集 es 日志（codec multiline ）
multiline 使用文档： https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html按照本人前面文章配置es，默认的es日志是存放在
[root@elk logstash]# cat /var/log/es-log/li-application.log日志样例：
[root@elk logstash]# cat /var/log/es-log/li-application.log
[2018-03-07T08:00:01,741][INFO ][o.e.c.m.MetaDataCreateIndexService] [linux-1] [system-2018.03.07] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2018-03-07T08:00:02,871][INFO ][o.e.c.m.MetaDataMappingService] [linux-1] [system-2018.03.07/30wHARyNQOWmA8M7m5VhAg] create_mapping [doc]
[2018-03-07T08:00:03,902][INFO ][o.e.c.r.a.AllocationService] [linux-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[system-2018.03.07][4]] ...]).
[2018-03-07T11:43:35,996][INFO ][o.e.c.m.MetaDataCreateIndexService] [linux-1] [nginx-log-2018.03.07] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2018-03-07T11:43:36,861][INFO ][o.e.c.m.MetaDataMappingService] [linux-1] [nginx-log-2018.03.07/EEPYUpQ5SFehkouvUrWSEQ] create_mapping [doc]
[2018-03-07T11:43:38,362][INFO ][o.e.c.r.a.AllocationService] [linux-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[nginx-log-2018.03.07][4]] ...]).
[2018-03-07T11:50:28,465][INFO ][o.e.c.r.a.AllocationService] [linux-1] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300} left]).
[2018-03-07T11:50:28,466][INFO ][o.e.c.s.MasterService ] [linux-1] zen-disco-node-left({linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300}), reason(left)[{linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300} left], reason: removed {{linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300},}
[2018-03-07T11:50:28,466][INFO ][o.e.c.s.ClusterApplierService] [linux-1] removed {{linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300},}, reason: apply cluster state (from master [master {linux-1}{RY3FNOHwT9-CPjSb8nM8Tw}{RgVROHXBR26GVXabtyRZ3Q}{172.16.1.176}{172.16.1.176:9300} committed version [54] source [zen-disco-node-left({linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300}), reason(left)[{linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300} left]]])
[2018-03-07T11:50:28,639][INFO ][o.e.i.s.IndexShard ] [linux-1] [nginx-log-2018.03.07][1] primary-replica resync completed with 0 operations
[2018-03-07T11:50:28,699][INFO ][o.e.i.s.IndexShard ] [linux-1] [nginx-log-2018.03.07][3] primary-replica resync completed with 0 operations
[2018-03-07T11:50:28,718][WARN ][o.e.c.NodeConnectionsService] [linux-1] failed to connect to node {linux-2}{6bO0Kw5aTx6nYZN_0Y4mwg}{63mbD-B4Q6ueOw3OtevIGg}{172.16.1.177}{172.16.1.177:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [linux-2][172.16.1.177:9300] connect_exception
at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:616) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:513) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_161]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_161]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: 拒绝连接: 172.16.1.177/172.16.1.177:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: 拒绝连接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2018-03-07T11:50:28,752][INFO ][o.e.i.s.IndexShard ] [linux-1] [system-2018.03.07][1] primary-replica resync completed with 0 operations
类似于java的日志，所以这边收集的时候，就需要一定的规则，将不是以 "[" 开头的日志记录，都归回到上一行。方便查看的时候，不会分开java日志的报错
- 修改172.16.1.176 主机的 logstash 配置文件 shipper.conf （本文放在 /usr/share/logstash ）
input{
syslog {
type => "system" #定义一个type 用于下方output的时候做匹配
host => "172.16.1.176" #syslog的IP地址
port => "514" #syslog端口
}
file { #以文件形式读取
path => "/var/log/nginx/access.log" #文件路径
codec => "json" #利用codec插件，读取json数据
start_position => "beginning" #记录位置，从头
type => "nginx-log"
}
file { #以文件形式读取
path => "/var/log/es-log/li-application.log"
start_position => "beginning"
codec => multiline{ #使用meltiline 插件
pattern => "^\[" #意思即为匹配开头是[
negate => true # negate = 否定
what => "previous" # 上面配置意思即为，非开头是“[” 的行数，归为上一行记录
}
type => "es-log"
}
}
output{
if [type] == "system" { #写入system日志到redis
redis {
host => "172.16.1.176" #redis的地址
port => "6379" #端口
db => "3" #创建一个数据库3
data_type => "list" #以list存
key => "system" #key名称，自己定义，最好对应上日志的类型
}
}
if [type] == "nginx-log" { #写入nginx日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log"
}
}
if [type] == "es-log"{ #写入es日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "es-log"
}
}
}修改172.16.1.176 主机上的 logstash 配置文件 indexer.conf
[root@elk2 logstash]# vim indexer.confinput{
redis {
host => "172.16.1.176" #redis主机地址 ,下面的配置对应上172.16.1.176 主机写入redis的配置
port => "6379" #端口
db => "3" #数据库名字
data_type => "list" #数据类型
key => "system"
type => "system"
}
redis { #从redis读取nginx日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log" #需要对应上redis的key
type => "nginx-log"
}
redis { #从redis读取es日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "es-log"
type => "es-log"
}
}
output{
if [type] == "system" { #写入system 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "system-%{+YYYY.MM.dd}" #使用system+ 日期，这个格式的索引
}
}
if [type] == "nginx-log" { #写入nginx 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
if [type] == "es-log"{ #写入es 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "es-log-%{+YYYY.MM.dd}"
}
}
}
4.4 收集 tcp 日志 （input tcp）
本小节，input tcp 文档：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html
修改172.16.1.176 主机上 shipper.conf
[root@elk logstash]# vim shipper.confinput{
syslog {
type => "system" #定义一个type 用于下方output的时候做匹配
host => "172.16.1.176" #syslog的IP地址
port => "514" #syslog端口
}
file { #以文件形式读取
path => "/var/log/nginx/access.log" #文件路径
codec => "json" #利用codec插件，读取json数据
start_position => "beginning" #记录位置，从头
type => "nginx-log"
}
file { #以文件形式读取
path => "/var/log/es-log/li-application.log"
start_position => "beginning"
codec => multiline{ #使用meltiline 插件
pattern => "^\[" #意思即为匹配开头是[
negate => true # negate = 否定
what => "previous" # 上面配置意思即为，非开头是“[” 的行数，归为上一行记录
}
type => "es-log"
}
tcp { #以TCP协议打开端口收集日志
host => "172.16.1.176"
port => "6666"
type => "tcp-log"
}
}
output{
if [type] == "system" { #写入system日志到redis
redis {
host => "172.16.1.176" #redis的地址
port => "6379" #端口
db => "3" #创建一个数据库3
data_type => "list" #以list存
key => "system" #key名称，自己定义，最好对应上日志的类型
}
}
if [type] == "nginx-log" { #写入nginx日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log"
}
}
if [type] == "es-log"{ #写入es日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "es-log"
}
}
if [type] == "tcp-log"{ #写入tcp日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "tcp-log"
}
}
}
跑起logstash ，检查打开的端口
[root@elk logstash]# bin/logstash -f shipper.conf
修改172.16.1.177 主机的 indexer.conf 配置文件
[root@elk2 logstash]# vim indexer.confinput{
redis {
host => "172.16.1.176" #redis主机地址 ,下面的配置对应上172.16.1.176 主机写入redis的配置
port => "6379" #端口
db => "3" #数据库名字
data_type => "list" #数据类型
key => "system"
type => "system"
}
redis { #从redis读取nginx日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log" #需要对应上redis的key
type => "nginx-log"
}
redis { #从redis读取es日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "es-log"
type => "es-log"
}
redis { #从redis读取tcp日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "tcp-log"
type => "tcp-log"
}
}
output{
if [type] == "system" { #写入system 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "system-%{+YYYY.MM.dd}" #使用system+ 日期，这个格式的索引
}
}
if [type] == "nginx-log" { #写入nginx 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
if [type] == "es-log"{ #写入es 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "es-log-%{+YYYY.MM.dd}"
}
}
if [type] == "tcp-log"{ #写入tcp日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "tcp-log-%{+YYYY.MM.dd}"
}
}
}[root@elk2 logstash]# bin/logstash -f indexer.conf利用 工具 nc 发送一些记录到 tcp端口中，随意发一个log（没有安装nc的可以yum安装一下），也可以输出一些信息到设备的tcp端口中，如下命令
[root@elk ~]# nc 172.16.1.176 6666 < install.log
[root@elk ~]# echo "test tcp log 1 " | nc 172.16.1.176 6666
[root@elk ~]# echo "test tcp log 2 " > /dev/tcp/172.16.1.176/6666配置好kibana的索引后，可以看到收集的日志
4.5 收集 mysql slowlog 日志 （grok plugin ）
本小节，gork plugin应用：https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
由于本机没有mysql 环境，手动在/var/log/目录下创建一个 mysql slowlog 日志 作为测试
[root@elk logstash]# cat /var/log/slowlog.log
# User@Host: taobao[taobao] @ regular_exp [192.168.35.23] Id: 1235
# Schema: bat_db Last_errno: 0 Killed: 0
# Query_time: 3.101086 Lock_time: 0.181175 Rows_sent: 0 Rows_examined: 360321 Rows_affected: 103560
# Bytes_sent: 58
SET timestamp=1450288856;
create table just_for_temp_case as
select '2015-12-16' as the_date,
t2.user_id,
t1.stuff_no,
count(*) as buy_times
from stuff_entries as t1
join bill as t2
on t1.orderserino = t2.id
where t2.notification_ts < '2015-12-17 00:00:00'
and t2.notification_ts >= '2015-09-18 00:00:00'
group by t2.user_id,
t1.stuff_no;
# Time: 151217 18:03:47
# User@Host: taobao[taobao] @ regular_exp [192.168.35.23] Id: 1235
# Schema: bat_db Last_errno: 0 Killed: 0
# Query_time: 3.101086 Lock_time: 0.181175 Rows_sent: 0 Rows_examined: 360321 Rows_affected: 103560
# Bytes_sent: 58
SET timestamp=1450288856;
create table just_for_temp_case as
select '2015-12-16' as the_date,
t2.user_id,
t1.stuff_no,
count(*) as buy_times
from stuff_entries as t1
join bill as t2
on t1.orderserino = t2.id
where t2.notification_ts < '2015-12-17 00:00:00'
and t2.notification_ts >= '2015-09-18 00:00:00'
group by t2.user_id,
t1.stuff_no;
# Time: 151217 18:03:47
# User@Host: taobao[taobao] @ regular_exp [192.168.35.23] Id: 1235
# Schema: bat_db Last_errno: 0 Killed: 0
# Query_time: 3.101086 Lock_time: 0.181175 Rows_sent: 0 Rows_examined: 360321 Rows_affected: 103560
# Bytes_sent: 58
SET timestamp=1450288856;
create table just_for_temp_case as
select '2015-12-16' as the_date,
t2.user_id,
t1.stuff_no,
count(*) as buy_times
from stuff_entries as t1
join bill as t2
on t1.orderserino = t2.id
where t2.notification_ts < '2015-12-17 00:00:00'
and t2.notification_ts >= '2015-09-18 00:00:00'
group by t2.user_id,
t1.stuff_no;
# Time: 151217 18:03:47编辑172.16.1.176 主机上的shipper.conf ，添加上读取mysql的慢日志配置，并且添加一个filter ，使用grok插件
Grok负责解析文本模式，使用正则表达式并将其分配至标识符。
Grok模式的语法结构为%{PATTERN:IDENTIFIER}。每个Logstash filter包含多种能够与日志信息相匹配并将其分配给不同标识符的grok模式，而这也正是让日志内容转化为结构化信息的关键所在。
[root@elk logstash]# vim shipper.conf input{
syslog {
type => "system" #定义一个type 用于下方output的时候做匹配
host => "172.16.1.176" #syslog的IP地址
port => "514" #syslog端口
}
file { #以文件形式读取
path => "/var/log/nginx/access.log" #文件路径
codec => "json" #利用codec插件，读取json数据
start_position => "beginning" #记录位置，从头
type => "nginx-log"
}
file { #以文件形式读取
path => "/var/log/es-log/li-application.log"
start_position => "beginning"
codec => multiline{ #使用meltiline 插件
pattern => "^\[" #意思即为匹配开头是[
negate => true # negate = 否定
what => "previous" # 上面配置意思即为，非开头是“[” 的行数，归为上一行记录
}
type => "es-log"
}
tcp { #以TCP协议打开端口收集日志
host => "172.16.1.176"
port => "6666"
type => "tcp-log"
}
file {
path => "/var/log/slowlog.log" #这里填写测试的slowlog地址，生产环境填写为自己的具体路径
start_position => "beginning"
codec => multiline { #与es读取一样，需要合并多行
pattern => "^# User@Host:"
negate => true
what => "previous"
}
type => "mysql-slowlog"
}
}
filter{
if [type] == "mysql-slowlog"{ #匹配type为mysql-slowlog 生效
grok {
match => [ "message", "#这里正则表达式，自己根据slowlog去编写，编写的规则以及语法可以参考上方连接#"]
}
match => [ "timestamp", "UNIX" ]
#remove_field => [ "timestamp" ]
}
}
}
output{
if [type] == "system" { #写入system日志到redis
redis {
host => "172.16.1.176" #redis的地址
port => "6379" #端口
db => "3" #创建一个数据库3
data_type => "list" #以list存
key => "system" #key名称，自己定义，最好对应上日志的类型
}
}
if [type] == "nginx-log" { #写入nginx日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log"
}
}
if [type] == "es-log"{ #写入es日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "es-log"
}
}
if [type] == "tcp-log"{ #写入tcp日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "tcp-log"
}
}
if [type] == "mysql-slowlog"{ #写入slowlog日志到redis
redis {
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "mysql-slowlog"
}
}
}编辑172.16.1.177配置文件 indexer.conf
[root@elk2 logstash]# vim indexer.confinput{
redis {
host => "172.16.1.176" #redis主机地址 ,下面的配置对应上172.16.1.176 主机写入redis的配置
port => "6379" #端口
db => "3" #数据库名字
data_type => "list" #数据类型
key => "system"
type => "system"
}
redis { #从redis读取nginx日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "nginx-log" #需要对应上redis的key
type => "nginx-log"
}
redis { #从redis读取es日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "es-log"
type => "es-log"
}
redis { #从redis读取tcp日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "tcp-log"
type => "tcp-log"
}
redis { #从redis读取slowlog日志
host => "172.16.1.176"
port => "6379"
db => "3"
data_type => "list"
key => "mysql-slowlog"
type => "mysql-slowlog"
}
}
output{
if [type] == "system" { #写入system 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "system-%{+YYYY.MM.dd}" #使用system+ 日期，这个格式的索引
}
}
if [type] == "nginx-log" { #写入nginx 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
if [type] == "es-log"{ #写入es 日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "es-log-%{+YYYY.MM.dd}"
}
}
if [type] == "tcp-log"{ #写入tcp日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "tcp-log-%{+YYYY.MM.dd}"
}
}
if [type] == "mysql-slowlog"{ #写入slowlog日志到 es
elasticsearch{
hosts => ["172.16.1.176:9200"]
index => "mysqlslow-log-%{+YYYY.MM.dd}"
}
}
}分别运行logstash起来
可以看到HEAD插件中，数据写入到es中了，filter 控件生效
kibana中同样看到日志记录了 因为我这里测试的slowlog日志是2015年 所以，注意选择好查看的日期，选择最近5年
2862
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
