mac && elasticsearch statck install
环境 mac
1 get Elasticsearch
2 get kibana
3 get filebeat
4 install X-Pack
5 go for launch
open Kibana
1 install elasticsearch
1 查看环境 jdk1.8以上
java -version
echo $JAVA_HOME
2 下载elasticsearch
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.tar.gz
3 解压启动[不能root启动]
tar -xvf elasticsearch-5.1.1.tar.gz
sudo chown -Rf kobe /usr/local/application/
cd usr/local/application/elasticsearch-5.1.1/bin
./elasticsearch
4 启动
[2016-12-10T20:48:16,414][INFO ][o.e.n.Node ] [] initializing ...
[2016-12-10T20:48:16,570][INFO ][o.e.e.NodeEnvironment ] [JCy-1Ry] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [63.7gb], net total_space [111.8gb], spins? [unknown], types [hfs]
[2016-12-10T20:48:16,570][INFO ][o.e.e.NodeEnvironment ] [JCy-1Ry] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-12-10T20:48:16,572][INFO ][o.e.n.Node ] [JCy-1Ry] node name [JCy-1Ry] derived from node ID; set [node.name] to override
[2016-12-10T20:48:16,580][INFO ][o.e.n.Node ] [JCy-1Ry] version[5.0.0], pid[1054], build[253032b/2016-10-26T04:37:51.531Z], OS[Mac OS X/10.12/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [aggs-matrix-stats]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [ingest-common]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [lang-expression]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [lang-groovy]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [lang-mustache]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [lang-painless]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [percolator]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [reindex]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [transport-netty3]
[2016-12-10T20:48:18,328][INFO ][o.e.p.PluginsService ] [JCy-1Ry] loaded module [transport-netty4]
[2016-12-10T20:48:18,328][INFO ][o.e.p.PluginsService ] [JCy-1Ry] no plugins loaded
[2016-12-10T20:48:22,371][INFO ][o.e.n.Node ] [JCy-1Ry] initialized
[2016-12-10T20:48:22,372][INFO ][o.e.n.Node ] [JCy-1Ry] starting ...
[2016-12-10T20:48:22,701][INFO ][o.e.t.TransportService ] [JCy-1Ry] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2016-12-10T20:48:25,814][INFO ][o.e.c.s.ClusterService ] [JCy-1Ry] new_master {JCy-1Ry}{JCy-1RyUTKeObYvr_uHlCw}{93GbHutRQPm3p5EUS
指定name启动
./elasticsearch -Ecluster.name=kobehaha -Enode.name=nodename
5 cluster health
两种方式
1 curl
curl -XGET 'localhost:9200/_cat/health?v&pretty'
2 kibana console
GET /_cat/health?v
result
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1481374854 21:00:54 kobehaha green 1 1 0 0 0 0 0 0 - 100.0%
6 创建索引
curl -XPUT 'localhost:9200/customer?pretty&pretty'
{
"acknowledged" : true,
"shards_acknowledged" : true
}
curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open customer i8YhESMiQgykO7fYxMN4kQ 5 1 0 0 260b 260b
7 索引添加数据
curl -XPUT 'localhost:9200/customer/external/1?pretty&pretty' -d'
kabana
1 down load and unzip kibana
2 open config/kibana.yml and set elasticsearch.url to point at elasticsearch instance
3 bin/kibana
borower at http://localhost:5601
logstash
download and unzip logstash 5.0.2
https://www.elastic.co/downloads/past-releases/logstash-5-0-2
watch logstash how to config
https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
this is very helpful for log
this conclude many plugins for logstash
因为和下面filebeat所以我直接上最终的配置文件
input {
beats {
port => "5043"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
9200是输出
filter是处理
input是入口
install filebeat[作为输入Log的处理]
我就不做详细的介绍了 这主要是对本地文件的一个filter
配置文件看一下就知道什么意思了
我处理的都是最常规的apach日志
/usr/local/logs/logstash-tutorial.log
86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.haskell.org/haskellwiki/Xmonad/Frequently_asked_questions" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
启动
sudo ./filebeat -e -c filebeat.yml -d "publish"
配置文件
filebeat.prospectors:
- input_type: log
paths:
- /usr/local/logs/logstash-tutorial.log
output.logstash:
hosts: ["localhost:5043"]
输入为/usr/loal/.....
输出为logstash的入口5043....
测试
1 查看elasticsearch是否有数据
curl 'localhost:9200/_cat/indices?v'
很明显在所以中看到了logstash今天的索引数据
2 打开 http://localhost:5601/ 就可以看到kibana的数据面板了