ELK集群环境搭建

ELK下载地址: 

https://www.elastic.co/cn/downloads/past-releases

 

ELK版本兼容性:

es6.5.4 不兼容 kibana 7.9.3

kibana配置elasticsearch:

7.X版本以下不支持配置es集群地址,如下:
elasticsearch.url: "http://192.168.1.1:9200"

7.X版本以上支持配置es集群地址,如下:

elasticsearch.hosts: ["http://172.16.4.241:9000","http://172.16.4.123:9000","http://172.16.4.59:9000"]

 

下载ik分词器:
wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.10.1/elasticsearch-analysis-ik-7.10.1.zip

ElasticSearch默认的分词器为Stardard分词器.

 

Elasticsearch集群搭建:

需要三台服务器: node-1, node-2, node-3

#启动

bin/elasticsearch -d

 

#node-1配置:
cluster.name: warf
node.name: node-1
node.master: true  
node.data: true
path.data: /data/elk/elasticsearch-6.5.4/data
path.logs: /data/elk/elasticsearch-6.5.4/logs
network.host: 172.16.4.123
bootstrap.memory_lock: false
http.port: 9000
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.port: 9800
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts: ["172.16.4.241", "172.16.4.59","172.16.4.123"]
discovery.zen.minimum_master_nodes: 2

 

#node-2配置
cluster.name: warf
node.name: node-2
node.master: true
node.data: true
path.data: /data/elk/elasticsearch-6.5.4/data
path.logs: /data/elk/elasticsearch-6.5.4/logs
network.host: 172.16.4.59
bootstrap.memory_lock: false
http.port: 9000
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.port: 9800
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts: ["172.16.4.241", "172.16.4.59","172.16.4.123"]
discovery.zen.minimum_master_nodes: 2

 

#node-3配置
cluster.name: warf
node.name: node-3
node.master: true
node.data: true
path.data: /data/elk/elasticsearch-6.5.4/data
path.logs: /data/elk/elasticsearch-6.5.4/logs
network.host: 172.16.4.241
bootstrap.memory_lock: false
http.port: 9000
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.port: 9800
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts: ["172.16.4.241", "172.16.4.59","172.16.4.123"]
discovery.zen.minimum_master_nodes: 2

 

ElasticSearch Head搭建:

1. 修改Gruntfile.js文件中hostname和port

 connect: {
                        server: {
                                options: {
                                        hostname: '172.16.4.59',
                                        port: 9100,
                                        base: '.',
                                        keepalive: true
                                }
                        }
                }

2.  修改_site/app.js

     配置ip为任意es集群中节点的ip

     this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://172.16.4.59:9000";

3. 启动head插件: 

     /elasticsearch-head/node_modules/grunt/bin/grunt server &

 

Kibana单机搭建

1. 修改config/kibana.yml

server.port: 5601
server.host: "172.16.4.54"
server.name: "warf_kibana"
elasticsearch.hosts: ["http://172.16.4.241:9000","http://172.16.4.123:9000","http://172.16.4.59:9000"]
elasticsearch.requestTimeout: 99999

2. 启动

./kibana &

 

Filebeat:

1. 新建 filebeat-dashboard.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /itcast/logs/*.log
setup.template.settings:
  index.number_of_shards: 3
output.logstash:
  hosts: ["192.168.40.133:5044"]

2. 启动
./filebeat -e -c filebeat-dashboard.yml

 

Logstash:

1. 新建 logstash-dashboard.conf

input {
      beats {
            port => "5044"
    }
}

filter {
    mutate {
        split => {"message"=>"|"}
    }
    mutate {
        add_field => {
            "userId" => "%{message[1]}"
            "visit" => "%{message[2]}"
            "date" => "%{message[3]}"
        }
    }
    mutate {
        convert => {
            "userId" => "integer"
            "visit" => "string"
            "date" => "string"
        }
    }
}

output {
    stdout { codec => rubydebug }
}

output {
    elasticsearch {
        hosts => [ "http://172.16.4.241:9000","http://172.16.4.123:9000","http://172.16.4.59:9000"]
    }
}

 

2. 启动

./bin/logstash -f logstash-dashboard.conf &