1. 下载
https://www.elastic.co/downloads/kibana
https://www.elastic.co/downloads/elasticsearch
https://www.elastic.co/downloads/logstash
2. 解压到制定的文件夹下面并重命名
mv elasticsearch-2.3.3 elasticsearch
mv kibana-4.5.1 kibana
mv logstash-2.3.2 logstash
3. 安装插件
- $cd elasticsearch
- head
- $bin/plugin install mobz/elasticsearch-head
- watcher
- $bin/plugin install license
- $bin/plugin install watcher
- shield
- $bin/plugin install shield
4. 配置shield插件
- #创建管理员用户
- $bin/shield/esusers useradd es_admin -r admin
- 创建Logstash用户
- $bin/shield/esusers useradd logstashserver -r logstash
- 创建kibana用户
- $bin/shield/esusers useradd kibanaserver -r kibana4_server
5. 配置kibana
- $cd cabana
- $ vi config/kibana.yml
- 设置server.host #例如 "127.0.0.1"
- 设置elasticsearch.username # 例如： kibanaserver
- 设置elasticsearch.password
- 保存退出
6. 启动
- $cd elasticsearch
- $bin/elasticsearch
- $cd kibana
- $bin/kibana
7. 测试
es: http://127.0.0.1:9200/_plugin/head/
kibana: http://127.0.0.1:5601
输入管理员账号和密码 es_admin/password
如果一切正常。说明elasticsearch 和 kibana搭建成功
8. 配置logstash
cd logstash
(1) hello world
vi logstash-simple.conf
input {
stdin { }
}
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "logstashserver"
password => "newpass"
}
stdout {
codec => rubydebug }
}
bin/logstash -f logstash-simple.conf
接着输入Hello World
去http://127.0.0.1:9200/_plugin/head/可以看到以logstash开头的indices
(2) 复杂点的logstash配置
vi logstash-filter.conf
input { stdin { } }
filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
output { elasticsearch {
hosts => ["localhost:9200"]
user => "logstashserver"
password => "newpass"
}
stdout { codec => rubydebug }
}
bin/logstash -f logstash-filter.conf
输入以下
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
(3) syslog
vi logstash-syslog.conf
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
} }
filter {
if [type] == "syslog”
{grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ] }
date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
} } }
output { elasticsearch {
hosts => ["localhost:9200”]
user => "logstashserver"
password => "newpass"
}
stdout { codec => rubydebug } }
再另外打开一个终端
telnet localhost 5000
输入hello logstash
(4) File Input
logstash-tutorial-dataset 文件下载链接
https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz
vi logstash-first.conf
input {
file {
path => "/Users/fdrong/LogProject/logstash-tutorial-dataset"
start_position => beginning
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "logstashserver"
password => "newpass"
}
stdout {
codec => rubydebug }
}
测试以下语法是否有错
bin/logstash -f logstash-first.conf —configtest
如果没有错误
bin/logstash -f logstash-first.conf
如果看到控制台打印解析后的日志（JSON格式）说明配置成功
9. 配置Watcher
(1). add a condition that simply checks to see if the search input returned any hits
curl -u es_admin —XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{
"trigger" : { "schedule" : { "interval" : "10s" } },
"input" : {
"search" : {
"request" : {
"indices" : [ "logs" ],
"body" : {
"query" : {
"match" : { "message": "error" }}}}}},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
}
}'
(2) 观察结果
curl —u es_admin -XGET 'http://localhost:9200/.watch_history*/_search?pretty' -d '{
"query" : {
"bool" : {
"must" : [
{ "match" : { "result.condition.met" : true }},
{ "range" : { "result.execution_time" : { "from" : "now-10s"}}}
]
}
}
}'
（3）添加错误日志
curl -u es_admin —XPOST 'http://localhost:9200/logs/event' -d '{
"timestamp" : "2015-05-17T18:12:07.613Z",
"request" : "GET index.html",
"status_code" : 404,
"message" : "Error: File not found"
}’
然后再用第二步的命令查看出现很多hints说明执行成功
（4）添加邮件提醒功能
a. 首先cd elasticsearch
vi config/elasticsearch.yml
在文件的最末尾添加发件服务器设置
watcher.actions.email.service.account:
account:
profile: qq
email_defaults:
from: '<xxxxxx@qq.com>'
smtp:
auth: true
starttls.enable: true
host: smtp.qq.com
port: 25
user: xxxxx
password: xxxxxx
b. 添加邮件提醒event
curl -u es_admin —XPUT 'http://localhost:9200/_watcher/watch/log_error_watch_email' -d '{
"trigger" : { "schedule" : { "interval" : "10s" } },
"input" : {
"search" : {
"request" : {
"indices" : [ "logs" ],
"body" : {
"query" : {
"match" : { "message": "error" }}}}}},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
}
"actions" : {
"email_administrator" : {
"throttle_period": "15m", #发送邮件间隔
"email" : {
"to" : "接收邮件地址",
"subject" : "found {{ctx.payload.hits.total}} errors in logs",
"body" : "Too many error in the system, see attached data",
"attachments" : {
"attached_data" : {
"data" : {
"format" : "json"
}
}
},
"priority" : "high"
}
}
}
}'
40秒之后就会收到一个提醒邮件，提示发现错误日志
（4）删除
curl -XDELETE 'http://localhost:9200/_watcher/watch/log_error_watch'
curl -XDELETE 'http://localhost:9200/_watcher/watch/log_error_watch_email'
10. 相关链接
logstash document
https://www.elastic.co/guide/index.html
config
http://kibana.logstash.es/content/logstash/plugins/output/elasticsearch.html
elastic download
https://www.elastic.co/downloads
wacher
https://www.elastic.co/guide/en/watcher/current/watch-log-data.html#log-add-condition
shield
https://eligao.com/shield-on-elasticsearch/
http://blog.sina.com.cn/s/blog_8ea8e9d50102wudw.html
http://blog.csdn.net/july_2/article/details/24481935
扫一扫
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
