转载自:http://467754239.blog.51cto.com/4878013/1700828
一、简介
1、 核心组成
ELK由Elasticsearch、Logstash和Kibana三部分组件组成;
Elasticsearch是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。
Logstash是一个完全开源的工具,它可以对你的日志进行收集、分析,并将其存储供以后使用
kibana 是一个开源和免费的工具,它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以帮助您汇总、分析和搜索重要数据日志。
2、 四大组件
Logstash: logstash server端用来搜集日志;
Elasticsearch: 存储各类日志;
Kibana: web化接口用作查寻和可视化日志;
Logstash Forwarder:logstash client端用来通过lumberjack 网络协议发送日志到logstash server;
3、 ELK工作流程
在需要收集日志的所有服务上部署logstash,作为logstash agent(logstash shipper)用于监控并过滤收集日志,将过滤后的内容发送到Redis,然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch,可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。
4、 ELK的帮助手册
ELK官网:https://www.elastic.co/
ELK官网文档:https://www.elastic.co/guide/index.html
ELK中文手册:http://kibana.logstash.es/content/elasticsearch/monitor/logging.html
注释:
ELK有两种安装方式
(1)集成环境:Logstash有一个集成包,里面包括了其全套的三个组件;也就是安装一个集成包。
(2)独立环境:三个组件分别单独安装、运行、各司其职。(比较常用)
本实验也以第二种方式独立环境来进行演示;单机版主机地址为:192.168.1.104
二、Logstash
1、 安装jdk
Logstash的运行依赖于Java运行环境
yum -y install java-1.8.0
java -version
2、 安装Logstash
wgethttps://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz -O/usr/local/src/logstash-1.5.4.tar.gz
tar zxf/usr/local/src/logstash-1.5.4.tar.gz -C /usr/local/
配置logstash环境变量
echo "export PATH=\$PATH:/usr/local/logstash-1.5.4/bin" > /etc/profile.d/logstash.sh
. /etc/profile
3、 Logstash常用参数
-e :指定logstash的配置信息,可以用于快速测试;
-f :指定logstash的配置文件;可以用于生产环境;
4、 启动logstash
4.1通过-e参数指定logstash的配置信息,用于快速测试,直接输出到屏幕。
logstash -e "input {stdin{}} output {stdout{}}"
myname is steven //
手动输入后回车,等待10秒后会有返回结果
2017-06-08T04:11:17.444Zlocalhost.localdomain my nname is steven
这种方式是原封不动的输出
4.2通过-e参数指定logstash的配置信息,用于快速测试,以json格式输出到屏幕。
logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
my name is steven //
手动输入后回车,等待10秒后会有返回结果
{
"message" => "my name is steven",
"@version" => "1",
"@timestamp" => "2017-06-08T04:16:02.713Z",
"host" => "localhost.localdomain"
}
5、 Logstash以配置文件方式启动
5.1输出信息到屏幕
vim logstash-simple.conf
input { stdin {} }
output {
stdout { codec=>rubydebug }
}
logstash -f logstash-simple.conf //普通方式启动
Logstash startup completed
logstash agent -f logstash-simple.conf --verbose//开启debug模式
Pipeline started {:level=>:info}
Logstash startup completed
hello world. //手动输入hello world.
{
"message" => "helloworld.",
"@version" => "1",
"@timestamp" => "2017-06-08T05:45:58.867Z",
"host" =>"localhost.localdomain"
}
效果同命令行配置参数一样...
5.2logstash输出信息存储到redis数据库中
刚才我们是将信息直接显示在屏幕上了,现在我们将logstash的输出信息保存到redis数据库中,如下:
前提是本地(192.168.5.204)有redis数据库,那么下一步我们就是安装redis数据库.
cat logstash_to_redis.conf
input { stdin { } }
output {
stdout { codec => rubydebug }
redis {
host => '192.168.5.204:6379'
data_type => 'list'
key => 'logstash:redis'
}
}
如果提示Failed to send event to Redis,表示连接Redis失败或者没有安装,请检查...
6、 查看logstash的监听端口
三、Redis
四、Elasticsearch
1、 安装Elasticsearch
wgethttps://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz-O /usr/local/src/elasticsearch-1.7.2.tar.gz
tar zxf/usr/local/src/elasticsearch-1.7.2.tar.gz -C /usr/local/
2、 修改elasticsearch配置文件elasticsearch.yml并且做以下修改
vim /usr/local/elasticsearch-1.7.2/config/elasticsearch.yml
discovery.zen.ping.multicast.enabled: false #关闭广播,如果局域网有机器开9300 端口,服务会启动不了
network.host: 192.168.5.204 #指定主机地址,其实是可选的,但是最好指定因为后面跟kibana集成的时候会报http连接出错(直观体现好像是监听了:::9200 而不是0.0.0.0:9200)
http.cors.allow-origin: "/.*/"
http.cors.enabled: true #这2项都是解决跟kibana集成的问题,错误体现是 你的 elasticsearch 版本过低,其实不是
3、 启动elasticsearch服务
/usr/local/elasticsearch-1.7.2/bin/elasticsearch #日志会输出到stdout
/usr/local/elasticsearch-1.7.2/bin/elasticsearch -d#表示以daemon的方式启动nohup /usr/local/elasticsearch-1.7.2/bin/elasticsearch > /var/log/logstash.log 2>&1 &
4、 查看elasticsearch的监听端口
netstat -tnlp |grep java
tcp 0 0 ::ffff:192.168.5.204:9200 :::* LISTEN 3119/java
tcp 0 0 ::ffff:192.168.5.204:9300 :::* LISTEN 3119/java
5、 Elasticsearch和Logstash结合
将logstash的信息输出到elasticsearch中
cat logstash-elasticsearch.conf
input { stdin {} }
output {
elasticsearch { host => "192.168.5.204" }
stdout{ codec=> rubydebug }
}
6、 基于配置文件启动logstash
/usr/local/logstash-1.5.4/bin/logstash agent -f logstash-elasticsearch.conf
Logstash startup completed
python linux java c++ //手动输入
{
"message" => "pythonlinux java c++",
"@version" => "1",
"@timestamp" => "2017-06-08T07:44:36.907Z",
"host" =>"localhost.localdomain"
}
7、 浏览器发送请求来查看elasticsearch是否接收到了数据
http://localhost:9200/_search?pretty
{
"took" : 44,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 6,
"max_score" : 1.0,
"hits" : [ {
"_index" :"logstash-2017.06.08",
"_type" : "logs",
"_id" :"AVyGqbpc8-7ivsGY27gZ",
"_score" : 1.0,
"_source":{"message":"","@version":"1","@timestamp":"2017-06-08T07:44:48.444Z","host":"localhost.localdomain"}
}, {
"_index" :"logstash-2017.06.08",
"_type" : "logs",
"_id" :"AVyGqb5M8-7ivsGY27ga",
"_score" : 1.0,
"_source":{"message":"","@version":"1","@timestamp":"2017-06-08T07:44:49.707Z","host":"localhost.localdomain"}
}, {
"_index" :"logstash-2017.06.08",
"_type" : "logs",
"_id" :"AVyGq03A8-7ivsGY27gb",
"_score" : 1.0,
"_source":{"message":"","@version":"1","@timestamp":"2017-06-08T07:46:32.506Z","host":"localhost.localdomain"}
}, {
"_index" :"logstash-2017.06.08",
"_type" : "logs",
"_id" :"AVyGqYw98-7ivsGY27gW",
"_score" : 1.0,
"_source":{"message":"pythonlinux javac++","@version":"1","@timestamp":"2017-06-08T07:44:36.907Z","host":"localhost.localdomain"}
}, {
"_index" :"logstash-2017.06.08",
"_type" : "logs",
"_id" :"AVyGqbJy8-7ivsGY27gX",
"_score" : 1.0,
"_source":{"message":"","@version":"1","@timestamp":"2017-06-08T07:44:47.217Z","host":"localhost.localdomain"}
}, {
"_index" :"logstash-2017.06.08",
"_type" : "logs",
"_id" :"AVyGqbZq8-7ivsGY27gY",
"_score" : 1.0,
"_source":{"message":"","@version":"1","@timestamp":"2017-06-08T07:44:47.858Z","host":"localhost.localdomain"}
} ]
}
}
8、 安装elasticsearch插件
Elasticsearch-kopf插件可以查询Elasticsearch中的数据,安装elasticsearch-kopf,只要在你安装Elasticsearch的目录中执行以下命令即可:
cd /usr/local/elasticsearch-1.7.2/bin/
./plugin install lmenezes/elasticsearch-kopf
-> Installinglmenezes/elasticsearch-kopf...
Tryinghttps://github.com/lmenezes/elasticsearch-kopf/archive/master.zip...
Downloading .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed lmenezes/elasticsearch-kopf into/usr/local/elasticsearch-1.7.2/plugins/kopf
执行插件安装后会提示失败,很有可能是网络等情况...
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip...
Failed to install lmenezes/elasticsearch-kopf, reason: failed to download out of all possible locations..., use --verbose to get detailed information
解决办法就是手动下载该软件,不通过插件安装命令...
cd /usr/local/elasticsearch-1.7.2/plugins
wget https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip
unzip master.zip
mv elasticsearch-kopf-master kopf
以上操作就完全等价于插件的安装命令
9、 浏览器访问kop页面访问elaticsearch保存的数据
netstat -tnlp |grep java
http://192.168.5.204:9200/_plugin/kopf/#!/cluster
10、 从redis数据库中读取然后输出到elasticsearch中
cat logstash-redis.conf
input {
redis {
host => '192.168.5.204' # 我方便测试没有指定password,最好指定password
data_type => 'list'
port => "6379"
key => 'logstash:redis' #自定义
type => 'redis-input' #自定义
}
}
output {
elasticsearch {
host => "192.168.5.204"
codec => "json"
protocol => "http" #版本1.0+ 必须指定协议http
}
}
11、
五、Kinaba
1、 安装Kinaba
wgethttps://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz -O /usr/local/src/kibana-4.1.2-linux-x64.tar.gz
tar zxf/usr/local/src/kibana-4.1.2-linux-x64.tar.gz -C /usr/local
2、 修改kinaba配置文件kinaba.yml
vim /usr/local/kibana-4.1.2-linux-x64/config/kibana.yml
elasticsearch_url: "http://192.168.5.204:9200"
3、 启动kinaba
/usr/local/kibana-4.1.2-linux-x64/bin/kibana
输出以下信息,表明kinaba成功.
{"name":"Kibana","hostname":"localhost.localdomain","pid":3473,"level":30,"msg":"Noexisting kibana indexfound","time":"2017-06-08T08:09:55.424Z","v":0}
{"name":"Kibana","hostname":"localhost.localdomain","pid":3473,"level":30,"msg":"Listeningon0.0.0.0:5601","time":"2017-06-08T08:09:55.433Z","v":0}
kinaba默认监听在本地的5601端口上
4、 浏览器访问kinaba
4.1使用默认的logstash-*的索引名称,并且是基于时间的,点击“Create”即可。
http://192.168.5.204:5601/#/settings/indices/?_g=()
4.2看到如下界面说明索引创建完成。
4.3点击“Discover”,可以搜索和浏览Elasticsearch中的数据。
>>>结束<<<
1、ELK默认端口号
elasticsearch:9200 9300
logstash : 9301
kinaba : 5601
2、错误汇总
(1)java版本过低
[2015-10-07 18:39:18.071] WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
(2)Kibana提示Elasticsearch版本过低...
This version of Kibana requires Elasticsearch 2.0.0 or higher on all nodes. I found the following incompatible nodes in your cluster:
Elasticsearch v1.7.2 @ inet[/192.168.5.204:9200] (127.0.0.1)
解决办法: