2015-10-08 13:18:58
ELK 日志分析系统
标签：
elk
原创作品，允许转载，转载时请务必以超链接形式标明文章
原始出处 、作者信息和本声明。否则将追究法律责任。
http://467754239.blog.51cto.com/4878013/1700828
|
1
2
3
4
5
6
|
Logstash的运行依赖于Java运行环境。
# yum -y install java-1.8.0
# java -version
openjdk version
"1.8.0_51"
OpenJDK Runtime Environment (build 1.8.0_51-b16)
OpenJDK 64-Bit Server VM (build 25.51-b03, mixed mode)
|
|
1
2
3
4
5
6
|
# wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz
# tar zxf logstash-1.5.4.tar.gz -C /usr/local/
配置logstash的环境变量
# echo "export PATH=\$PATH:/usr/local/logstash-1.5.4/bin" > /etc/profile.d/logstash.sh
# . /etc/profile
|
|
1
2
|
-e :指定logstash的配置信息，可以用于快速测试;
-f :指定logstash的配置文件；可以用于生产环境;
|
|
1
2
3
4
5
|
# logstash -e "input {stdin{}} output {stdout{}}"
my name is zhengyansheng.
//
手动输入后回车，等待10秒后会有返回结果
Logstash startup completed
2015-10-08T13:55:50.660Z 0.0.0.0 my name is zhengyansheng.
这种输出是直接原封不动的返回...
|
|
1
2
3
4
5
6
7
8
9
10
|
# logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
my name is zhengyansheng.
//
手动输入后回车，等待10秒后会有返回结果
Logstash startup completed
{
"message"
=>
"my name is zhengyansheng."
,
"@version"
=>
"1"
,
"@timestamp"
=>
"2015-10-08T13:57:31.851Z"
,
"host"
=>
"0.0.0.0"
}
这种输出是以json格式的返回...
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# vim logstash-simple.conf
input { stdin {} }
output {
stdout { codec=> rubydebug }
}
# logstash -f logstash-simple.conf //普通方式启动
Logstash startup completed
# logstash agent -f logstash-simple.conf --verbose //开启debug模式
Pipeline started {:level=>:info}
Logstash startup completed
hello world.
//
手动输入hello world.
{
"message"
=>
"hello world."
,
"@version"
=>
"1"
,
"@timestamp"
=>
"2015-10-08T14:01:43.724Z"
,
"host"
=>
"0.0.0.0"
}
效果同命令行配置参数一样...
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
前提是本地(192.168.1.104)有redis数据库，那么下一步我们就是安装redis数据库.
# cat logstash_to_redis.conf
input { stdin { } }
output {
stdout { codec => rubydebug }
redis {
host =>
'192.168.1.104'
data_type =>
'list'
key =>
'logstash:redis'
}
}
如果提示Failed to send event to Redis，表示连接Redis失败或者没有安装，请检查...
|
|
1
2
3
|
# logstash agent -f logstash_to_redis.conf --verbose
# netstat -tnlp |grep java
tcp 0 0 :::9301 :::* LISTEN 1326
/java
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
wget http:
//download
.redis.io
/releases/redis-2
.8.19.
tar
.gz
yum
install
tcl -y
tar
zxf redis-2.8.19.
tar
.gz
cd
redis-2.8.19
make
MALLOC=libc
make
test
//
这一步时间会稍久点...
make
install
cd
utils/
.
/install_server
.sh
//
脚本执行后，所有选项都以默认参数为准即可
Welcome to the redis service installer
This script will help you easily
set
up a running redis server
Please
select
the redis port
for
this instance: [6379]
Selecting default: 6379
Please
select
the redis config
file
name [
/etc/redis/6379
.conf]
Selected default -
/etc/redis/6379
.conf
Please
select
the redis log
file
name [
/var/log/redis_6379
.log]
Selected default -
/var/log/redis_6379
.log
Please
select
the data directory
for
this instance [
/var/lib/redis/6379
]
Selected default -
/var/lib/redis/6379
Please
select
the redis executable path [
/usr/local/bin/redis-server
]
Selected config:
Port : 6379
Config
file
:
/etc/redis/6379
.conf
Log
file
:
/var/log/redis_6379
.log
Data
dir
:
/var/lib/redis/6379
Executable :
/usr/local/bin/redis-server
Cli Executable :
/usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied
/tmp/6379
.conf =>
/etc/init
.d
/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
|
|
1
2
3
4
|
# netstat -tnlp |grep redis
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 3843
/redis-server
*
tcp 0 0 127.0.0.1:21365 0.0.0.0:* LISTEN 2290
/src/redis-serv
tcp 0 0 :::6379 :::* LISTEN 3843
/redis-server
*
|
|
1
2
3
4
5
6
7
8
9
|
# cd redis-2.8.19/src/
# ./redis-cli -h 192.168.1.104 -p 6379 //连接redis
192.168.1.104:6379>
ping
PONG
192.168.1.104:6379>
set
name zhengyansheng
OK
192.168.1.104:6379> get name
"zhengyansheng"
192.168.1.104:6379> quit
|
|
1
2
|
# ps -ef |grep redis
root 3963 1 0 08:42 ? 00:00:00
/usr/local/bin/redis-server
*:6379
|
|
1
2
|
# cd redis-2.8.19/src/
# ./redis-cli monitor //reids动态监控
|
|
1
2
3
4
|
# netstat -tnlp |grep redis
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 3843
/redis-server
*
tcp 0 0 127.0.0.1:21365 0.0.0.0:* LISTEN 2290
/src/redis-serv
tcp 0 0 :::6379 :::* LISTEN 3843
/redis-server
*
|
|
1
2
3
|
# cd redis-2.8.19/src/
# ./redis-cli monitor
OK
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# cat logstash_to_redis.conf
input { stdin { } }
output {
stdout { codec => rubydebug }
redis {
host =>
'192.168.1.104'
data_type =>
'list'
key =>
'logstash:redis'
}
}
# logstash agent -f logstash_to_redis.conf --verbose
Pipeline started {:level=>:info}
Logstash startup completed
dajihao linux
{
"message"
=>
"dajihao linux"
,
"@version"
=>
"1"
,
"@timestamp"
=>
"2015-10-08T14:42:07.550Z"
,
"host"
=>
"0.0.0.0"
}
|
|
1
2
3
4
5
|
# ./redis-cli monitor
OK
1444315328.103928 [0 192.168.1.104:56211]
"rpush"
"logstash:redis"
"{\"message\":\"dajihao linux\",\"@version\":\"1\",\"@timestamp\":\"2015-10-08T14:42:07.550Z\",\"host\":\"0.0.0.0\"}"
如果redis的监控上也有以上信息输出，表明logstash和redis的结合是正常的。
|
|
1
2
|
# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz
# tar zxf elasticsearch-1.7.2.tar.gz -C /usr/local/
|
|
1
2
3
4
5
|
# vim /usr/local/elasticsearch-1.7.2/config/elasticsearch.yml
discovery.zen.
ping
.multicast.enabled:
false
#关闭广播，如果局域网有机器开9300 端口，服务会启动不了
network.host: 192.168.1.104
#指定主机地址，其实是可选的，但是最好指定因为后面跟kibana集成的时候会报http连接出错（直观体现好像是监听了:::9200 而不是0.0.0.0:9200）
http.cors.allow-origin:
"/.*/"
http.cors.enabled:
true
#这2项都是解决跟kibana集成的问题，错误体现是 你的 elasticsearch 版本过低，其实不是
|
|
1
2
3
|
# /usr/local/elasticsearch-1.7.2/bin/elasticsearch #日志会输出到stdout
# /usr/local/elasticsearch-1.7.2/bin/elasticsearch -d #表示以daemon的方式启动
# nohup /usr/local/elasticsearch-1.7.2/bin/elasticsearch > /var/log/logstash.log 2>&1 &
|
|
1
2
3
|
# netstat -tnlp |grep java
tcp 0 0 :::9200 :::* LISTEN 7407
/java
tcp 0 0 :::9300 :::* LISTEN 7407
/java
|
|
1
2
3
4
5
6
7
|
将logstash的信息输出到elasticsearch中
# cat logstash-elasticsearch.conf
input { stdin {} }
output {
elasticsearch { host =>
"192.168.1.104"
}
stdout { codec=> rubydebug }
}
|
|
1
2
3
4
5
6
7
8
9
10
|
# /usr/local/logstash-1.5.4/bin/logstash agent -f logstash-elasticsearch.conf
Pipeline started {:level=>:info}
Logstash startup completed
python linux java c++
//
手动输入
{
"message"
=>
"python linux java c++"
,
"@version"
=>
"1"
,
"@timestamp"
=>
"2015-10-08T14:51:56.899Z"
,
"host"
=>
"0.0.0.0"
}
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
# curl http://localhost:9200/_search?pretty
{
"took"
: 28,
"timed_out"
:
false
,
"_shards"
: {
"total"
: 5,
"successful"
: 5,
"failed"
: 0
},
"hits"
: {
"total"
: 1,
"max_score"
: 1.0,
"hits"
: [ {
"_index"
:
"logstash-2015.10.08"
,
"_type"
:
"logs"
,
"_id"
:
"AVBH7-6MOwimSJSPcXjb"
,
"_score"
: 1.0,
"_source"
:{
"message"
:
"python linux java c++"
,
"@version"
:
"1"
,
"@timestamp"
:
"2015-10-08T14:51:56.899Z"
,
"host"
:
"0.0.0.0"
}
} ]
}
}
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
#Elasticsearch-kopf插件可以查询Elasticsearch中的数据，安装elasticsearch-kopf，只要在你安装Elasticsearch的目录中执行以下命令即可：
# cd /usr/local/elasticsearch-1.7.2/bin/
# ./plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes
/elasticsearch-kopf
...
Trying https:
//github
.com
/lmenezes/elasticsearch-kopf/archive/master
.zip...
Downloading .............................................................................................
Installed lmenezes
/elasticsearch-kopf
into
/usr/local/elasticsearch-1
.7.2
/plugins/kopf
执行插件安装后会提示失败，很有可能是网络等情况...
-> Installing lmenezes
/elasticsearch-kopf
...
Trying https:
//github
.com
/lmenezes/elasticsearch-kopf/archive/master
.zip...
Failed to
install
lmenezes
/elasticsearch-kopf
, reason: failed to download out of all possible locations..., use --verbose to get detailed information
解决办法就是手动下载该软件，不通过插件安装命令...
cd
/usr/local/elasticsearch-1
.7.2
/plugins
wget https:
//github
.com
/lmenezes/elasticsearch-kopf/archive/master
.zip
unzip master.zip
mv
elasticsearch-kopf-master kopf
以上操作就完全等价于插件的安装命令
|
|
1
2
3
4
|
# netstat -tnlp |grep java
tcp 0 0 :::9200 :::* LISTEN 7969
/java
tcp 0 0 :::9300 :::* LISTEN 7969
/java
tcp 0 0 :::9301 :::* LISTEN 8015
/java
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
# cat logstash-redis.conf
input {
redis {
host =>
'192.168.1.104'
# 我方便测试没有指定password，最好指定password
data_type =>
'list'
port =>
"6379"
key =>
'logstash:redis'
#自定义
type
=>
'redis-input'
#自定义
}
}
output {
elasticsearch {
host =>
"192.168.1.104"
codec =>
"json"
protocol =>
"http"
#版本1.0+ 必须指定协议http
}
}
|
|
1
2
|
# wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
# tar zxf kibana-4.1.2-linux-x64.tar.gz -C /usr/local
|
|
1
2
|
# vim /usr/local/kibana-
4.1
.
2
-linux-x64/config/kibana.yml
elasticsearch_url:
"http://192.168.1.104:9200"
|
|
1
2
3
4
5
6
|
/usr/local/kibana-4
.1.2-linux-x64
/bin/kibana
输出以下信息，表明kinaba成功.
{
"name"
:
"Kibana"
,
"hostname"
:
"localhost.localdomain"
,
"pid"
:1943,
"level"
:30,
"msg"
:
"No existing kibana index found"
,
"time"
:
"2015-10-08T00:39:21.617Z"
,
"v"
:0}
{
"name"
:
"Kibana"
,
"hostname"
:
"localhost.localdomain"
,
"pid"
:1943,
"level"
:30,
"msg"
:
"Listening on 0.0.0.0:5601"
,
"time"
:
"2015-10-08T00:39:21.637Z"
,
"v"
:0}
kinaba默认监听在本地的5601端口上
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
1、ELK默认端口号
elasticsearch：9200 9300
logstash : 9301
kinaba : 5601
2、错误汇总
(1)java版本过低
[2015-10-07 18:39:18.071] WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
(2)Kibana提示Elasticsearch版本过低...
This version of Kibana requires Elasticsearch 2.0.0 or higher on all nodes. I found the following incompatible nodes
in
your cluster:
Elasticsearch v1.7.2 @ inet[
/192
.168.1.104:9200] (127.0.0.1)
解决办法：
|
扫一扫
5万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
