一、ELKStack简介
1、ELK介绍
中文指南:https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details
ELK Stack包含:ElasticSearch、Logstash、Kibana
ElasticSearch是一个搜索引擎,用来搜索、分析、存储日志。它是分布式的,也就是说可以横向扩容,可以自动发现,索引自动分片,总之很强大。文档https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html
Logstash用来采集日志,把日志解析为json格式交给ElasticSearch。
Kibana是一个数据可视化组件,把处理后的结果通过web界面展示
Beats在这里是一个轻量级日志采集器,其实Beats家族有5个成员
早期的ELK架构中使用Logstash收集、解析日志,但是Logstash对内存、cpu、io等资源消耗比较高。相比 Logstash,Beats所占系统的CPU和内存几乎可以忽略不计
x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包,是收费的。
2、ELK架构:
二、Elasticsearch部署
1、安装JDK
[root@linux-node1 ~]# yum install -y java
[root@linux-node1 ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
源码安装JDK:
下载
[root@linux-node1 ~]# wget http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz
配置Java环境
[root@linux-node1 ~]# tar zxf jdk-8u151-linux-x64.tar.gz -C /usr/local/
[root@linux-node1 ~]# ln –s /usr/local/jdk1.8.0_91 /usr/local/jdk
[root@linux-node1 ~]# vim /etc/profile
export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
[root@linux-node1 ~]# source /etc/profile
[root@linux-node1 ~]# java -version
注:linux-node2节点上也安装JDK
2、安装Elasticsearch
linux-node2节点也需要安装elasticsearch
使用yum安装elasticsearch会很慢,建议先下载:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
(1)源码安装elasticsearch:
安装elasticsearch
[root@linux-node1 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm
[root@linux-node1 ~]# yum install -y elasticsearch-6.0.0.rpm
配置elasticsearch,linux-node2配置一个相同的节点,通过组播进行通信,会通过cluster进行查找,如果无法通过组播查询,修改成单播即可。
[root@linux-node1 ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name:elk-cluster #集群名称
node.name:elk-node1 #节点名称,一个集群之内节点的名称不能重复
path.data:/data/elkdata #数据路径
path.logs:/data/logs #日志路径
#bootstrap.memory_lock:true #锁住es内存,保证内存不分配至交换分区,这里暂时不配置。注释掉
network.host:192.168.56.11 #网络监听地址
http.port:9200 #用户访问查看的端口,9300是组件访问使用
discovery.zen,ping.unicast.hosts:["192.168.56.11","192.168.56.12"] #单播(配置一台即可,生产可以使用组播方式)
[root@linux-node1 ~]# mkdir /data/{elkdata,logs}
[root@linux-node1 ~]# chown elasticsearch.elasticsearch /data -R
[root@linux-node1 ~]# systemctl start elasticsearch.service
[root@linux-node1 ~]# netstat -tulnp |grep java
tcp6 0 0 192.168.56.11:9200 :::* LISTEN 26866/java
tcp6 0 0 192.168.56.11:9300 :::* LISTEN 26866/java
将配置文件拷贝到linux-node2
[root@linux-node1 ~]# scp /etc/elasticsearch/elasticsearch.yml 192.168.56.12:/etc/elasticsearch/
[root@linux-node2 ~]# vim /etc/elasticsearch/elasticsearch.yml
修改:
node.name=elk-node2
network.host=192.168.56.12
[root@linux-node2 ~]# mkdir /data/{elkdata,logs}
[root@linux-node2 ~]# chown elasticsearch.elasticsearch /data -R
[root@linux-node2 ~]# systemctl start elasticsearch.service
[root@linux-node2 ~]# netstat -tulnp |grep java
tcp6 0 0 192.168.56.12:9200 :::* LISTEN 16346/java
tcp6 0 0 192.168.56.12:9300 :::* LISTEN 16346/java
(2)yum安装elasticsearch
1.下载并安装GPG key
[root@linux-node1 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
2.添加yum仓库
[root@linux-node1 ~]# vim /etc/yum.repos.d/es.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
3.安装elasticsearch
[root@hadoop-node1 ~]# yum install -y elasticsearch
三、Elasticsearch的集群配置和监控
可以使用命令来查看elasticsearch的状态内容
[root@linux-node1 ~]# curl http://192.168.56.11:9200/_cluster/health?pretty=true
{
"cluster_name" : "elk-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@linux-node2 ~]# curl http://192.168.56.12:9200/_cluster/health?pretty=true
{
"cluster_name" : "elk-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@linux-node1 ~]# curl -i -XGET 'http://192.168.56.11:9200/_count?' #查看es里面有什么内容
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 71
{"count":0,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0}}
解释:
返回头部200,执行成功0个,返回0个
curl http://192.168.56.11:9200/_cluster/health?pretty 健康检查
curl http://192.168.56.11:9200/_cluster/state?pretty 集群详细信息
注:但是我们不可能经常通过命令来查看集群的信息,这里就使用elasticsearch的插件--head
插件是为了完成不同的功能,而官方提供了一些插件但大部分是收费的,另外也有一些开发爱好者提供的插件。可以实现对elasticsearch集群的状态与管理配置等功能。
四、Elasticsearch插件
Head插件
插件作用:主要是做集群管理的插件
Github下载地址:https://github.com/mobz/elasticsearch-head
[root@linux-node1 ~]# git clone git://github.com/mobz/elasticsearch-head.git
[root@linux-node1 ~]# cd elasticsearch-head
[root@linux-node1 elasticsearch-head]# yum install -y npm
[root@linux-node1 elasticsearch-head]# npm install grunt -save
[root@linux-node1 elasticsearch-head]# npm install #执行安装
[root@linux-node1 elasticsearch-head]# npm run start & #后台运行
注:在elasticsearch 2.x以前的版本可以通过/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head来安装head插件,在elasticsearch 5.x以上版本需要通过npm进行安装。
浏览器访问:http://192.168.56.11:9100,可以看到各个节点的状态信息,如图: