ELK(elasticsearch+logstash+kibana)实时日志分析系统搭建

2019独角兽企业重金招聘Python工程师标准>>>

ELK实时日志分析系统搭建

相关文章：searchguard插件安装：http://my.oschina.net/lengchuan/blog/729745

ElasticSearch安装及其配置

安装

1. ELK下载：https://www.elastic.co/downloads/
2. 解压安装：

tar -zxvf elasticsearch-2.3.3.tar.gz

cd elasticsearch-2.3.3

插件安装

1. 安装head插件

./bin/plugin install mobz/elasticsearch-head

ES配置

编辑配置文件elasticsearch.yml

vi config/elasticsearch.yml

修改以下配置项：

cluster.name=es_cluster #集群名称，同一集群名称必须相同

node.name=node0 #集群节点的名称

path.data=/tmp/elasticsearch/data #数据存储路径

path.logs=/tmp/elasticsearch/logs #日志存储路径

network.host=192.168.1.101 #当前hostname或IP

network.port=9200 #端口号，默认9200

其它配置按需配置。

启动ES

ES跟其他的节点的传输端口为9300，接受HTTP请求的端口为9200。

./bin/elasticsearch

或者后台进程启动

nohup ./bin/elasticsearch &

浏览器打开http://ip:9200/就可以获取到ES相关信息 head插件页面：http://ip:9200/_plugin/head,即可查看集群页面信息

Logstash安装配置

安装

1. ELK下载：https://www.elastic.co/downloads/
2. 解压安装：

tar -zxvf logstash-2.3.3.tar.gz

cd logstash-2.3.3

配置

编写配置文件(名字和位置可以随意，这里放在config目录下，取名为log4j_to_es.conf，配置文件可以有多个，可以同时启动多个logstash实例) 更多配置请参考logstash的官方文档:https://www.elastic.co/guide/en/logstash/index.html

关于logstash收集log4j的日志信息，请大家参考另一篇博文：http://my.oschina.net/lengchuan/blog/725358

mkdir config vi config/log4j_to_es.conf 具体的配置请参考logstash的官方文档:https://www.elastic.co/guide/en/logstash/index.html 输入以下内容:

#For detail structure of this file
#Set:https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input {
  #For detail config for log4j as input,
  #See:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
  log4j {
    mode => "client"
    host => "localhost"
    port => 4560
    type => "cms_log"
  }
log4j {
    mode => "client"
    host => "localhost"
    port => 4561
    type => "front_log"
  }
}
filter {
  #Only matched data are send to output.
}
output {
  #For detail config for elasticsearch as output,
  #See:https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
  elasticsearch {
    action => "index"          #The operation on ES
    hosts  => ["ip地址:9200"]   #ElasticSearch host, can be array.
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
    document_type => "%{type}"        #The index to write data to.
  }
}

启动logstash

logstash命令只有2个参数，我们使用agent来启动,使用-f来启动：

nohup ./bin/logstash agent -f config/log4j_to_es.conf &

kibana4安装及配置

安装

1. ELK下载：https://www.elastic.co/downloads/
2. 解压安装 :

tar -zxvf kibana-4.5.0-linux-x86.tar.gz

cd kibana-4.5.0-linux-x86

vi config/kibana.yml

配置

修改以下几项:

server.port: 5601

server.host: "localhost"

elasticsearch.url: http://localohost:9200

kibana.index: ".kibana"

启动

nohup ./bin/kibana &

在生产环境部署

Nginx 代理配置

upstream kibana4 {
    server 127.0.0.1:5601 fail_timeout=0;
}
server {
    listen               *:80;
    server_name          kibana_server;
    access_log           /var/log/nginx/kibana.srv-log-dev.log;
    error_log            /var/log/nginx/kibana.srv-log-dev.error.log;

    ssl                  on;
    ssl_certificate      /etc/nginx/ssl/all.crt;
    ssl_certificate_key  /etc/nginx/ssl/server.key;

    location / {
        root   /var/www/kibana;
        index  index.html  index.htm;
    }

    location ~ ^/kibana4/.* {
        proxy_pass           http://kibana4;
        rewrite              ^/kibana4/(.*)  /$1 break;
        proxy_set_header     X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header     Host            $host;
        auth_basic           "Restricted";
        auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
    }
}

如果用户够多，可以单独跑一个 kibana4 集群，然后在 upstream 配置段中添加多个代理地址做负载均衡。

转载于:https://my.oschina.net/lengchuan/blog/725355