安装JDK
首先请在elk-server机器上JDK8;
请在ELK官网https://www.elastic.co/downloads下载以下文件:
1. elasticsearch-6.2.3.tar.gz;
2. logstash-6.2.3.tar.gz;
3. kibana-6.2.3-linux-x86_64.tar.gz;
上述三个文件,推荐在CentOS7的命令行输入以下四个命令下载:
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
下载完毕后,创建目录/usr/local/work,将刚刚下载的三个文件全部在这个目录下解压,得到以下三个文件夹:
1. /usr/local/work/elasticsearch-6.2.3
2. /usr/local/work/logstash-6.2.3
3. kibana-6.2.3-linux-x86_64
创建用户
ElasticSerach要求以非root身份启动,所以我们要创建一个用户:
1. 创建用户组:groupadd elasticsearch;
2. 创建用户加入用户组:useradd elasticsearch -g elasticsearch;
3. 设置ElasticSerach文件夹为用户elasticsearch所有:chown -R elasticsearch.elasticsearch /usr/local/work/elasticsearch-6.2.3;
系统设置
设置hostname,打开文件/etc/hostname,将内容改为elk-server
关闭防火墙(如果因为其他原因不能关闭防火墙,也请不要禁止80端口):systemctl stop firewalld.service
禁止防火墙自动启动:systemctl disable firewalld.service
打开文件/etc/security/limits.conf,添加下面四行内容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
打开文件/etc/sysctl.conf,添加下面一行内容:
vm.max_map_count=655360
加载sysctl配置,执行命令:sysctl -p
重启电脑;
1.启动ElasticSerach
切换到用户elasticsearch:su elasticsearch;
进入目录/usr/local/work/elasticsearch-6.2.3;
执行启动命令:bin/elasticsearch -d,此时会在后台启动elasticsearch;
查看启动日志可执行命令:tail -f /usr/local/work/elasticsearch-6.2.3/logs/elasticsearch.log,大约五到十分钟后启动成功,提示如下:
[2018-04-07T10:12:27,392][INFO ][o.e.n.Node ] initialized
[2018-04-07T10:12:27,392][INFO ][o.e.n.Node ] [MNb1nGq] starting ...
[2018-04-07T10:12:39,676][INFO ][o.e.t.TransportService ] [MNb1nGq] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-07T10:12:42,772][INFO ][o.e.c.s.MasterService ] [MNb1nGq] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300}
[2018-04-07T10:12:42,776][INFO ][o.e.c.s.ClusterApplierService] [MNb1nGq] new_master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {MNb1nGq}{MNb1nGq6Tn6VskdKFQckow}{_DglQhgmRsGAF2D7eTfVfg}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-07T10:12:42,817][INFO ][o.e.g.GatewayService ] [MNb1nGq] recovered [0] indices into cluster_state
[2018-04-07T10:12:42,821][INFO ][o.e.h.n.Netty4HttpServerTransport] [MNb1nGq] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-04-07T10:12:42,821][INFO ][o.e.n.Node ] [MNb1nGq] starte
执行curl命令检查服务是否正常响应:curl 127.0.0.1:9200,收到响应如下:
[elasticsearch@elk-server work]$ curl 127.0.0.1:9200
{
"name" : "MNb1nGq",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "ZHkI7PCQTnCqMBM6rhyT5g",
"version" : {
"number" : "6.2.3",
"build_hash" : "c59ff00",
"build_date" : "2018-03-13T10:06:29.741383Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
至此,ElasticSerach服务启动成功,接下来是Logstash;
logstash安装
# 下载tar包 wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz
# 解压 tar xf logstash-6.2.3.tar.gz # 进入到目录 cd logstash-6.2.3/
# 准备一个config file
cat > logstash.conf << EOF
input { beats { port => "5044" } }
output { stdout { codec => rubydebug } }
EOF
# 运行 bin/logstash -f logstash.conf
# 使用 -e 在命令行上指定配置内容 bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'
# 上面的命令执行可能需要等待一会
# 使用 -f或--path.config 指定配置文件
# 使用 --config.test_and_exit 解析配置文件并报告可能的错误
bin/logstash -f some.conf --config.test_and_exit
# 如果配置文件通过了检查,执行下面的命令
# 使用 --config.reload.automatic 表示修改了配置文件之后自动重新加载配置,这样你就不用重启
Logstash bin/logstash -f some.conf --config.reload.automatic
#/usr/local/work/logstash-6.2.3/bin/logstash -f /usr/local/work/logstash-6.2.3/config/default.conf --config.reload.automatic > /usr/local/work/logstash-6.2.3/logs/logstash-plain.log 2>&1 &
Kibana安装
打开Kibana的配置文件/usr/local/work/kibana-6.2.3-linux-x86_64/config/kibana.yml,找到下面这行:
#server.host: "localhost"
改成如下内容:
elasticsearch.url: "http://139.199.124.136:9200" #ela地址
server.host: 0.0.0.0
server.port: 5601
logging.dest: /elk/logs/kibana.log #日志存放目录,自定义
这样其他电脑就能用浏览器访问Kibana的服务了;
进入Kibana的目录:/usr/local/work/kibana-6.2.3-linux-x86_64
执行启动命令:nohup bin/kibana &
查看启动日志:tail -f nohup.out
以下信息表示启动成功:
{"type":"log","@timestamp":"2018-04-07T04:44:59Z","tags":["status","plugin:elasticsearch@6.2.3","info"],"pid":3206,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:44:59Z","tags":["status","plugin:console@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:timelion@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:metrics@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["listening","info"],"pid":3206,"message":"Server running at http://localhost:5601"}
{"type":"log","@timestamp":"2018-04-07T04:45:01Z","tags":["status","plugin:elasticsearch@6.2.3","info"],"pid":3206,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
在浏览器访问http://IP:5601
安装运行 FileBeats
FileBeats 也提供了下载包,地址为 https://www.elastic.co/downloads/beats/filebeat 。找到系统对应的包下载后解压即可。
因为没有特负责的业务,用FileBeats收集日志直接输出到elasticsearch ,用kibana展示查询
tar zxvf filebeat-6.2.2-darwin-x86_64.tar.gz
cd filebeat-6.2.2-darwin-x86_64
采集多台服务器上的nginx日志时,可以用filebeat来采集,整个过程如下。
filebeat1 -->
filebeat2 --> logstash --> elasticsearch
filebeat3 -->
安装filebeat
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.2-x86_64.rpm
rpm -ivh filebeat-5.6.2-x86_64.rpm
主要是/etc/filebeat/filebeat.yml 文件,配置如下
filebeat.prospectors:
- type: log
#enabled: false 改为true才生效
enabled: true
- input_type: log
paths:
- /opt/weblogs/9885_access.log
tags: ["uwsgi-log"]
document_type: uwsgilog
output.logstash:
hosts: ["172.16.93.237:5044"]
启动
service filebeat start
chkconfig filebeat on #开机启动
另一种启动:/usr/bin/filebeat.sh -e -c /etc/filebeat/filebeat.yml