ELK简介:
ELK是三个开源软件的缩写,分别表示:Elasticsearch , Logstash, Kibana , 它们都是开源软件。新增了一个FileBeat,它是一个轻量级的日志收集处理工具(Agent),Filebeat占用资源少,适合于在各个服务器上搜集日志后传输给Logstash,官方也推荐此工具。
Elasticsearch是个开源分布式搜索引擎,提供搜集、分析、存储数据三大功能。它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。
Logstash 主要是用来日志的搜集、分析、过滤日志的工具,支持大量的数据获取方式。一般工作方式为c/s架构,client端安装在需要收集日志的主机上,server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。
Kibana 也是一个开源和免费的工具,Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以帮助汇总、分析和搜索重要数据日志。
官方文档:
Logstash
Kibana:
Elasticsearch
elasticsearch中文社区
[root@server1 ~]# ls
elk jdk-8u121-linux-x64.rpm
[root@server1 ~]# yum install jdk-8u121-linux-x64.rpm #由于shijava应用需要安装jdk的包
[root@server1 ~]# cd elk/
[root@server1 elk]# ls
bigdesk-master.zip jemalloc-devel-3.6.0-1.el6.x86_64.rpm
elasticsearch-2.3.3.rpm kibana-4.5.1-1.x86_64.rpm
elasticsearch-head-master.zip logstash-2.3.3-1.noarch.rpm
elk日志分析平台.pdf nginx-1.8.0-1.el6.ngx.x86_64.rpm
jemalloc-3.6.0-1.el6.x86_64.rpm redis-3.0.6.tar.gz
[root@server1 elk]# yum install elasticsearch-2.3.3.rpm -y #下载安装es
[root@server1 elasticsearch]# cd /etc/elasticsearch/
[root@server1 elasticsearch]# ls
elasticsearch.yml logging.yml scripts #配置文件在/etc/elasticsearch/ 下 elasticsearch.yml 为配置文件
[root@server1 elasticsearch]# vim elasticsearch.yml
[root@server1 elasticsearch]# /etc/init.d/elasticsearch start
Starting elasticsearch: [ OK ]
[root@server1 bin]# cd /usr/share/elasticsearch/bin/
[root@server1 bin]# ls
elasticsearch elasticsearch.in.sh elasticsearch-systemd-pre-exec plugin
[root@server1 bin]# ./plugin install file:/root/elk/elasticsearch-head-master.zip
-> Installing from file:/root/elk/elasticsearch-head-master.zip...
Trying file:/root/elk/elasticsearch-head-master.zip ...
Downloading .........DONE
Verifying file:/root/elk/elasticsearch-head-master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/share/elasticsearch/plugins/head
[root@server1 bin]# ./plugin ls
ERROR: unknown command [ls]. Use [-h] option to list available commands
[root@server1 bin]# ./plugin list
Installed plugins in /usr/share/elasticsearch/plugins:
- head
[root@server1 bin]# cd /var/lib/elasticsearch/
绿色代表良好
黄色分片为丢失
红色主分片丢失
配置集群节点端配置方法只要更改’node,name id端口等即可
我这里有三个节点,负责的功能可以自己自由组合,在配置文件中加入
node.master: true 为主节点
node.data: false 为数据节点
http.enable: true 可以负责查询
首先elasticsearch有丰富的api接口
这样时我们对其的操控变得容易的多
logstash日志查询
[root@server1 elk]# rpm -ivh logstash-2.3.3-1.noarch.rpm
Preparing... ########################################### [100%]
1:logstash ########################################### [100%]
[root@server1 conf.d]# /opt/logstash/bin/logstash -e 'input { stdin{}} output {stdout{ codec => rubydebug }}'
Settings: Default pipeline workers: 1
Pipeline main started
good
{
"message" => "good",
"@version" => "1",
"@timestamp" => "2018-08-24T02:45:19.601Z",
"host" => "server1"
}
boy
{
"message" => "boy",
"@version" => "1",
"@timestamp" => "2018-08-24T02:45:27.111Z",
"host" => "server1"
}
yes
{
"message" => "yes",
"@version" => "1",
"@timestamp" => "2018-08-24T02:45:30.421Z",
"host" => "server1"
}
cool
{
"message" => "cool",
"@version" => "1",
"@timestamp" => "2018-08-24T02:45:34.267Z",
"host" => "server1"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
[root@server1 conf.d]# cd /etc/logstash/conf.d/
[root@server1 conf.d]# ls
[root@server1 conf.d]# vim es.conf #也可以写入文件中,以.conf结尾的
[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/log
login.defs logrotate.conf logrotate.d/ logstash/
[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/log
login.defs logrotate.conf logrotate.d/ logstash/
[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf #-f指定配置文件
Settings: Default pipeline workers: 1
Pipeline main started
hello
{
"message" => "hello",
"@version" => "1",
"@timestamp" => "2018-08-24T02:49:48.930Z",
"host" => "server1"
}
boy
{
"message" => "boy",
"@version" => "1",
"@timestamp" => "2018-08-24T02:49:57.033Z",
"host" => "server1"
}
internet
{
"message" => "internet",
"@version" => "1",
"@timestamp" => "2018-08-24T02:50:12.046Z",
"host" => "server1"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
es.conf内的内容
结果查询
文件模块的使用方法
将日志导入,在els页面查看
input {
file {
path => "/var/log/messages" #这里需要注意的是在后端运行的时候需要增加可读权限
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => ["172.25.19.1"]
index => "message-%{+YYYY.MM.dd}"
}
}
~
执行一便
[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/es.conf
Settings: Default pipeline workers: 1
Pipeline main started
可以在页面看到
日志文件都已经上传成了
其实在程序运行的情况下,以下写入日志也会上传过来,当程序打断再次运行起来的时候不会从头加载了,否则会有许多的重复,对资源来讲是一种浪费,
syslog模块
input {
syslog {
port => 514
}
}
output {
stdout {
codec => rubydebug
}
}
[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/aa.conf #指定配置文件启动
,aa.conf的内容为上述内容
Settings: Default pipeline workers: 1
Pipeline main started
查看端口514已经打开
在做一个远程日志同步 将server2的日志传通过514端口输给server1
在这里终端页面已经显示出来了
在myes.log里面的日志是分行的
也可以通过插件来实现在一行输出
在文件里面写入filter的multiline插件
在终端显示出来,在一行输出
还有更多好玩的插件当然需要符合自己的logstash
日志的拆分截取
您可以选择将数据类型转换添加到grok模式。 默认情况下,所有语义都保存为字符串。 如果您希望转换语义的数据类型,例如将字符串更改为整数,则使用目标数据类型将其后缀。 例如%{NUMBER:num:int},它将num语义从字符串转换为整数。 目前唯一支持的转换是int和float。
input {
stdin {
}
}
filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
output {
stdout {
codec => rubydebug
}
}
[root@server1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/apache.conf
Settings: Default pipeline workers: 1
Pipeline main started
55.3.244.1 GET /index.html 15824 0.043
{
"message" => "55.3.244.1 GET /index.html 15824 0.043",
"@version" => "1",
"@timestamp" => "2018-08-24T06:33:30.626Z",
"host" => "server1",
"client" => "55.3.244.1",
"method" => "GET",
"request" => "/index.html",
"bytes" => "15824",
"duration" => "0.043"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout {
codec => rubydebug
}
}
[root@server1 conf.d]# cd /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/
[root@server1 logstash-patterns-core-2.0.5]# ls
CHANGELOG.md lib NOTICE.TXT
CONTRIBUTORS LICENSE patterns
Gemfile logstash-patterns-core.gemspec README.md
[root@server1 logstash-patterns-core-2.0.5]# cd patterns/
[root@server1 patterns]# ls
aws exim haproxy linux-syslog mongodb rails
bacula firewalls java mcollective nagios redis
bro grok-patterns junos mcollective-patterns postgresql ruby
[root@server1 patterns]# vim grok-patterns
搭建
由nginx的访问日志传递给redis ,并且redis传递给clasticsearch 并由kibana从clas进行数据的可视化
在server1端
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}
}
}
output {
redis {
host => ["172.25.19.2"]
port => 6379
data_type => "list"
key => "logstash:redis"
}
}
在后端启动,所以nginx的日志需要可读权限
我的server2端是redis
input {
redis {
host => "172.25.19.2"
port => 6379
data_type => "list"
key => "logstash:redis"
}
}
output {
elasticsearch {
hosts => ["172.25.19.1"]
index => "nginx-%{+YYYY.MM.dd}"
}
}
server3端负责kibana
配置文件所在位置
[root@server3 elk]# vim /opt/kibana/config/kibana.yml
输入任意一个节点的ip即可
先看看数据都没有elasticsearch
在kibana进行操作
在里匹配关键字
在这里进行配置各种的图像