elasticsearch stack install

mac && elasticsearch statck install

环境 mac

1 get Elasticsearch

2 get kibana

3 get filebeat

4 install X-Pack

5 go for launch

open Kibana

1 install elasticsearch

1 查看环境 jdk1.8以上

    java -version

    echo $JAVA_HOME

2 下载elasticsearch

    curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.tar.gz

3 解压启动[不能root启动]

    tar -xvf elasticsearch-5.1.1.tar.gz

    sudo chown -Rf kobe /usr/local/application/

    cd usr/local/application/elasticsearch-5.1.1/bin

    ./elasticsearch

4 启动

[2016-12-10T20:48:16,414][INFO ][o.e.n.Node               ] [] initializing ...
[2016-12-10T20:48:16,570][INFO ][o.e.e.NodeEnvironment    ] [JCy-1Ry] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [63.7gb], net total_space [111.8gb], spins? [unknown], types [hfs]
[2016-12-10T20:48:16,570][INFO ][o.e.e.NodeEnvironment    ] [JCy-1Ry] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-12-10T20:48:16,572][INFO ][o.e.n.Node               ] [JCy-1Ry] node name [JCy-1Ry] derived from node ID; set [node.name] to override
[2016-12-10T20:48:16,580][INFO ][o.e.n.Node               ] [JCy-1Ry] version[5.0.0], pid[1054], build[253032b/2016-10-26T04:37:51.531Z], OS[Mac OS X/10.12/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [aggs-matrix-stats]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [ingest-common]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [lang-expression]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [lang-groovy]
[2016-12-10T20:48:18,326][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [lang-mustache]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [lang-painless]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [percolator]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [reindex]
[2016-12-10T20:48:18,327][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [transport-netty3]
[2016-12-10T20:48:18,328][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] loaded module [transport-netty4]
[2016-12-10T20:48:18,328][INFO ][o.e.p.PluginsService     ] [JCy-1Ry] no plugins loaded
[2016-12-10T20:48:22,371][INFO ][o.e.n.Node               ] [JCy-1Ry] initialized
[2016-12-10T20:48:22,372][INFO ][o.e.n.Node               ] [JCy-1Ry] starting ...
[2016-12-10T20:48:22,701][INFO ][o.e.t.TransportService   ] [JCy-1Ry] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2016-12-10T20:48:25,814][INFO ][o.e.c.s.ClusterService   ] [JCy-1Ry] new_master {JCy-1Ry}{JCy-1RyUTKeObYvr_uHlCw}{93GbHutRQPm3p5EUS

指定name启动

./elasticsearch -Ecluster.name=kobehaha -Enode.name=nodename

5 cluster health

   两种方式

   1  curl 

    curl -XGET 'localhost:9200/_cat/health?v&pretty'

    2 kibana console 

    GET /_cat/health?v

    result

    epoch      timestamp cluster  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1481374854 21:00:54  kobehaha green           1         1      0   0    0    0        0             0                  -                100.0%

6 创建索引

curl -XPUT 'localhost:9200/customer?pretty&pretty'

{
  "acknowledged" : true,
  "shards_acknowledged" : true
}



curl -XGET 'localhost:9200/_cat/indices?v&pretty' 

health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   customer i8YhESMiQgykO7fYxMN4kQ   5   1          0            0       260b           260b

7 索引添加数据

curl -XPUT 'localhost:9200/customer/external/1?pretty&pretty' -d'

kabana

1 down load and unzip kibana

2 open config/kibana.yml and set elasticsearch.url to point at elasticsearch instance

3 bin/kibana

borower at http://localhost:5601

logstash

download and unzip logstash 5.0.2

    https://www.elastic.co/downloads/past-releases/logstash-5-0-2   

watch logstash how to config

    https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

    this is very helpful for log

    this conclude many plugins for logstash 


    因为和下面filebeat所以我直接上最终的配置文件

input {
    beats {
        port => "5043"
    }
}
 filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

    9200是输出 

    filter是处理

    input是入口

install filebeat[作为输入Log的处理]

    我就不做详细的介绍了 这主要是对本地文件的一个filter

    配置文件看一下就知道什么意思了

    我处理的都是最常规的apach日志

    /usr/local/logs/logstash-tutorial.log

    86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.haskell.org/haskellwiki/Xmonad/Frequently_asked_questions" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"

    86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"

    86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"

    启动
    sudo ./filebeat -e -c filebeat.yml -d "publish"

    配置文件
    filebeat.prospectors:
         - input_type: log
   paths:
          - /usr/local/logs/logstash-tutorial.log
   output.logstash:
        hosts: ["localhost:5043"]

   输入为/usr/loal/.....

   输出为logstash的入口5043....

测试

1 查看elasticsearch是否有数据

curl 'localhost:9200/_cat/indices?v'

很明显在所以中看到了logstash今天的索引数据

2 打开 http://localhost:5601/ 就可以看到kibana的数据面板了