elk安装

安装JDK

首先请在elk-server机器上JDK8；

 

请在ELK官网https://www.elastic.co/downloads下载以下文件：

1. elasticsearch-6.2.3.tar.gz；

2. logstash-6.2.3.tar.gz；

3. kibana-6.2.3-linux-x86_64.tar.gz；

 

上述三个文件，推荐在CentOS7的命令行输入以下四个命令下载：

 

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz

 

下载完毕后，创建目录/usr/local/work，将刚刚下载的三个文件全部在这个目录下解压，得到以下三个文件夹：

1. /usr/local/work/elasticsearch-6.2.3

2. /usr/local/work/logstash-6.2.3

3. kibana-6.2.3-linux-x86_64

 

创建用户

ElasticSerach要求以非root身份启动，所以我们要创建一个用户：

1. 创建用户组：groupadd elasticsearch；

2. 创建用户加入用户组：useradd elasticsearch -g elasticsearch；

3. 设置ElasticSerach文件夹为用户elasticsearch所有：chown -R elasticsearch.elasticsearch /usr/local/work/elasticsearch-6.2.3；

 

系统设置

设置hostname，打开文件/etc/hostname，将内容改为elk-server

关闭防火墙（如果因为其他原因不能关闭防火墙，也请不要禁止80端口）：systemctl stop firewalld.service

禁止防火墙自动启动：systemctl disable firewalld.service

打开文件/etc/security/limits.conf，添加下面四行内容：

* soft nofile 65536

* hard nofile 131072

* soft nproc 2048

* hard nproc 4096

 

打开文件/etc/sysctl.conf，添加下面一行内容：

 

vm.max_map_count=655360

 

加载sysctl配置，执行命令：sysctl -p

重启电脑；

 

 

1.启动ElasticSerach

切换到用户elasticsearch：su elasticsearch；

进入目录/usr/local/work/elasticsearch-6.2.3；

执行启动命令：bin/elasticsearch -d，此时会在后台启动elasticsearch；

查看启动日志可执行命令：tail -f /usr/local/work/elasticsearch-6.2.3/logs/elasticsearch.log，大约五到十分钟后启动成功，提示如下：

[2018-04-07T10:12:27,392][INFO ][o.e.n.Node ] initialized

[2018-04-07T10:12:27,392][INFO ][o.e.n.Node ] [MNb1nGq] starting ...

[2018-04-07T10:12:39,676][INFO ][o.e.t.TransportService ] [MNb1nGq] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}

[2018-04-07T10:12:42,772][INFO ][o.e.c.s.MasterService ] [MNb1nGq] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300}

[2018-04-07T10:12:42,776][INFO ][o.e.c.s.ClusterApplierService] [MNb1nGq] new_master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

[2018-04-07T10:12:42,817][INFO ][o.e.g.GatewayService ] [MNb1nGq] recovered [0] indices into cluster_state

[2018-04-07T10:12:42,821][INFO ][o.e.h.n.Netty4HttpServerTransport] [MNb1nGq] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}

[2018-04-07T10:12:42,821][INFO ][o.e.n.Node ] [MNb1nGq] starte

 

执行curl命令检查服务是否正常响应：curl 127.0.0.1:9200，收到响应如下：

 

[elasticsearch@elk-server work]$ curl 127.0.0.1:9200

{

"name" : "MNb1nGq",

"cluster_name" : "elasticsearch",

"cluster_uuid" : "ZHkI7PCQTnCqMBM6rhyT5g",

"version" : {

"number" : "6.2.3",

"build_hash" : "c59ff00",

"build_date" : "2018-03-13T10:06:29.741383Z",

"build_snapshot" : false,

"lucene_version" : "7.2.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

},

"tagline" : "You Know, for Search"

}

 

至此，ElasticSerach服务启动成功，接下来是Logstash；

 

 

logstash安装

# 下载tar包 wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz

# 解压 tar xf logstash-6.2.3.tar.gz # 进入到目录 cd logstash-6.2.3/

# 准备一个config file

cat > logstash.conf << EOF

input { beats { port => "5044" } }

output { stdout { codec => rubydebug } }

EOF

# 运行 bin/logstash -f logstash.conf

# 使用 -e 在命令行上指定配置内容 bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'

# 上面的命令执行可能需要等待一会

# 使用 -f或--path.config 指定配置文件

# 使用 --config.test_and_exit 解析配置文件并报告可能的错误

bin/logstash -f some.conf --config.test_and_exit

# 如果配置文件通过了检查,执行下面的命令

# 使用 --config.reload.automatic 表示修改了配置文件之后自动重新加载配置,这样你就不用重启

Logstash bin/logstash -f some.conf --config.reload.automatic

#/usr/local/work/logstash-6.2.3/bin/logstash -f /usr/local/work/logstash-6.2.3/config/default.conf --config.reload.automatic > /usr/local/work/logstash-6.2.3/logs/logstash-plain.log 2>&1 &

 

 

 

 

 

 

 

Kibana安装

打开Kibana的配置文件/usr/local/work/kibana-6.2.3-linux-x86_64/config/kibana.yml，找到下面这行：

#server.host: "localhost"

 

改成如下内容：

elasticsearch.url: "http://139.199.124.136:9200"    #ela地址

server.host: 0.0.0.0

server.port: 5601

logging.dest: /elk/logs/kibana.log  #日志存放目录，自定义　

 

这样其他电脑就能用浏览器访问Kibana的服务了；

进入Kibana的目录：/usr/local/work/kibana-6.2.3-linux-x86_64

执行启动命令：nohup bin/kibana &

查看启动日志：tail -f nohup.out

以下信息表示启动成功：

 

{"type":"log","@timestamp":"2018-04-07T04:44:59Z","tags":["status","plugin:elasticsearch@6.2.3","info"],"pid":3206,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2018-04-07T04:44:59Z","tags":["status","plugin:console@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:timelion@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:metrics@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["listening","info"],"pid":3206,"message":"Server running at http://localhost:5601"}

{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:elasticsearch@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

 

 

在浏览器访问http://IP:5601

 

安装运行 FileBeats

FileBeats 也提供了下载包，地址为 https://www.elastic.co/downloads/beats/filebeat 。找到系统对应的包下载后解压即可。

因为没有特负责的业务，用FileBeats收集日志直接输出到elasticsearch ，用kibana展示查询

tar zxvf filebeat-6.2.2-darwin-x86_64.tar.gz

cd filebeat-6.2.2-darwin-x86_64

 

采集多台服务器上的nginx日志时，可以用filebeat来采集，整个过程如下。

 

filebeat1 -->

filebeat2 --> logstash --> elasticsearch

filebeat3 -->

 

安装filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.2-x86_64.rpm

rpm -ivh filebeat-5.6.2-x86_64.rpm

 

主要是/etc/filebeat/filebeat.yml 文件，配置如下

 

filebeat.prospectors:

- type: log

 #enabled: false 改为true才生效

  enabled: true

- input_type: log

paths:

- /opt/weblogs/9885_access.log

tags: ["uwsgi-log"]

document_type: uwsgilog

output.logstash:

hosts: ["172.16.93.237:5044"]

 

 

启动

service filebeat start

chkconfig filebeat on #开机启动

 

另一种启动：/usr/bin/filebeat.sh -e -c /etc/filebeat/filebeat.yml