ELK日志分析系统

提供了一个分布式多用户能力的全文搜索引擎。Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。
Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是第二流行的企业搜索引擎。设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。

1.2、Elasticsearch的基础核心概念

1、接近实时（NRT）
elasticsearch是一个接近实时的搜索平台，这意味着，从索引一个文档直到这个文档能够被搜索到有一个轻微的延迟（通常是1秒）

2、集群（cluster）一个集群就是由一个或多个节点组织在一起，它们共同持有你整个的数据，并一起提供索引和搜索功能。其中一个节点为主节点，这个主节点是可以通过选举产生的，并提供跨节点的联合索引和搜索的功能。集群有一个唯一性标示的名字，默认是elasticsearch，集群名字很重要，每个节点是基于集群名字加入到其集群中的。因此，确保在不同环境中使用不同的集群名字。

一个集群可以只有一个节点。强烈建议在配置elasticsearch时，配置成集群模式。

3、节点（node）
节点就是一台单一的服务器，是集群的一部分，存储数据并参与集群的索引和搜索功能。像集群一样，节点也是通过名字来标识，默认是在节点启动时随机分配的字符名。当然，你可以自己定义。该名字也很重要，在集群中用于识别服务器对应的节点。
节点可以通过指定集群名字来加入到集群中。默认情况，每个节点被设置成加入到elasticsearch集群。如果启动了多个节点，假设能自动发现对方，他们将会自动组建一个名为elasticsearch的集群。

4、索引（type）

一个索引就是一个拥有几分相似特征的文档的集合。比如说，你可以有一个客户数据的索引，另一个产品目录的索引，还有一个订单数据的索引。
一个索引由一个名字来标识（必须全部是小写字母的），并且当我们要对对应于这个索引中的文档进行索引、搜索、更新和删除的时候，都要使用到这个名字。在一个集群中，如果你想，可以定义任意多的索引。

5、类型（type）

在一个索引中，你可以定义一种或多种类型。一个类型是你的索引的一个逻辑上的分类/分区，其语义完全由你来定。通常，会为具有一组共同字段的文档定义一个类型。比如说，我们假设你运营一个博客平台并且将你所有的数据存储到一个索引中。在这个索引中，你可以为用户数据定义一个类型，为博客数据定义另一个类型，当然，也可以为评论数据定义另一个类型。

6、文档（document）

一个文档是一个可被索引的基础信息单元。比如，你可以拥有某一个客户的文档，某一个产品的一个文档，当然，也可以拥有某个订单的一个文档。文档以JSON（Javascript Object Notation）格式来表示，而JSON(yaml 和json)是一个到处存在的互联网数据交互格式。
在一个index/type里面，只要你想，你可以存储任意多的文档。注意，虽然一个文档在物理上位于一个索引中，实际上一个文档必须在一个索引内被索引和分配一个类型。

ES 索引（库）——》类型（表）——》文档（字段）

1.3、分片和副本

在实际情况下，索引存储的数据可能超过单个节点的硬件限制。如一个10亿文档需1TB空间可能不适合存储在单个节点的磁盘上，或者从单个节点搜索请求太慢了。为了解决这个问题，elasticsearch提供将索引分成多个分片的功能。当在创建索引时，可以定义想要分片的数量。每一个分片就是一个全功能的独立的索引，可以位于集群中任何节点上。

每个索引可以被分成多个分片。一个索引也可以被复制0次（意思是没有复制）或多次。一旦复制了，每个索引就有了主分片（作为复制源的原来的分片）和复制分片（主分片的拷贝）之别。分片和副本的数量可以在索引创建的时候指定。

在索引创建之后，你可以在任何时候动态地改变副本的数量，但是你事后不能改变分片的数量。默认情况下，Elasticsearch中的每个索引被分片5个主分片和1个副本。

2、Logstash介绍

1、一款强大的数据处理工具

2、可实现数据传输、格式处理、格式化输出

3、数据输入（从业务输入）、数据加工（如过滤、改写等）以及数据输出（输出到Elasticsearch群集）

4、Logstash的主要组件

shipper：日志收集者，负责监控本地日志文件的变化，及时把日志文件的最新内容收集起来。通常，远程代理端（agent）只需要运行这个组件即可
indexer：日志存储者，负责接收日志并写入到本地文件
broker：日志hub，负责连接多个shipper和多个indexer
search and storage：允许对事件进行搜索和存储
web interface：基于Web的展示界面

3、Kibana介绍

1、一个针对Elasticsearch的开源分析及可视化平台

2、搜索、查看存储在Elasticsearch索引中的数据

3、通过各种图表进行高级数据分析及展示

4、主要功能

Elasticsearch无缝之集成
整合数据，复杂数据分析
让更多团队成员收益
接口灵活，分享更容易
配置简单，可视化多数据源
简单数据导出

四、ELK日志分析系统部署

相关安装包下载地址：（百度网盘）

链接：https://pan.baidu.com/s/1iCBfDrC4d0hIa3oOIBgxgw?pwd=lnmp
提取码：lnmp

1、实验环境

为了防止实验出问题，每台内存给到8G

主机名	IP地址	主要软件
node1	192.168.159.13	Elasticsearch、kibana
node2	192.168.159.20	Elasticsearch
apache	192.168.159.159	apache、logstash

2、配置node1基础环境

①、修改主机名

[root@zwb ~]# hostnamectl set-hostname node1
[root@zwb ~]# su
[root@node1 ~]#

②、配置映射关系

[root@node1 ~]# vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.159.13 node1
192.168.159.20 node2
192.168.159.159 apache

③、配置JDK

[root@node1 ~]# java -version             ## 检查java版本
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

[root@node1 ~]# cd /opt/                   ## 上传rpm包至opt目录
[root@node1 opt]# ls

jdk-8u201-linux-x64.rpm

[root@node1 opt]# rpm -ivh jdk-8u201-linux-x64.rpm   ## 安装JDK包
警告：jdk-8u201-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:jdk1.8-2000:1.8.0_201-fcs        ################################# [100%]
Unpacking JAR files...
	tools.jar...
	plugin.jar...
	javaws.jar...
	deploy.jar...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar..

[root@node1 opt]# vim /etc/profile            ## 修改/etc/profile配置

......................................................
    fi
done

unset i
unset -f pathmunge


###################   添加  #########################################
unset i
unset -f pathmunge

export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH



[root@node1 opt]# source /etc/profile  ## 重新加载文件

[root@node1 opt]# java -version                  ## 版本已更改
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

3、配置node2基础环境

①、修改主机名

[root@zwb ~]# hostnamectl set-hostname node2
[root@zwb ~]# su
[root@node2 ~]#

②、配置映射关系

[root@node2 ~]# vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.159.20 www.zhu.com www.wen.com
192.168.159.13 node1
192.168.159.20 node2
192.168.159.159 apache

③、配置JDK

[root@node2 ~]# java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
[root@node2 ~]# cd /opt
[root@node2 opt]# rz -E
rz waiting to receive.
[root@node2 opt]# ls
rh  jdk-8u201-linux-x64.rpm 

[root@node2 opt]# rpm -ivh jdk-8u201-linux-x64.rpm 
警告：jdk-8u201-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
准备中...                          ################################# [100%]
	软件包 jdk1.8-2000:1.8.0_201-fcs.x86_64 已经安装
[root@node2 opt]# vim /etc/profile

....................................................
unset i
unset -f pathmunge

#################### 增加 ###################################

export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH

  
[root@node2 opt]# java -version                      ### 版本号
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

4、部署Elasticsearch

4.1、在node1安装ES

[root@node1 opt]# rz -E                 ## 上传es安装包
rz waiting to receive.
[root@node1 opt]# rpm -ivh elasticsearch-5.5.0.rpm                      ##  安装
警告：elasticsearch-5.5.0.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
正在升级/安装...
   1:elasticsearch-0:5.5.0-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
[root@node1 opt]# systemctl daemon-reload                     ### 加载配置文件
[root@node1 opt]# systemctl enable elasticsearch.service    ##设置开机自启
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@node1 opt]# cp /etc/elasticsearch/elasticsearch.yml  
/etc/elasticsearch/elasticsearch.yml.bak                 ## 备份，防止配置失误


################# 注意：该配置文件每个冒号后面都有一个空格，切记不可删除######
[root@node1 opt]# vi /etc/elasticsearch/elasticsearch.yml   ## 修改文件

17/ cluster.name: my-elk-cluster                   ####集群名字
23/ node.name: node1                                  ####节点名字
33/ path.data: /data/elk_data                        ####数据存放路径
37/ path.logs: /var/log/elasticsearch/           ####日志存放路径
43/ bootstrap.memory_lock: false                 ####不在启动的时候锁定内存（前 
                              ## 端缓存。与IOPS-性能测试方式，每秒读写次数相关）
56/ network.host: 0.0.0.0                               ####提供服务绑定的IP地 
                                                ##  址，0.0.0.0代表所有地址
60/ http.port: 9200                                        ####侦听端口为9200
69/ discovery.zen.ping.unicast.hosts: ["node1", "node2"]           ####集群发 
                                                            现通过单播实现



### 查看刚才的配置
[root@node1 opt]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: node1
path.data: /data/elk_data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock:false

network.host:0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"]


### 创建数据存放路径
[root@node1 opt]# mkdir -p /data/elk_data

###给目录授予权限
[root@node1 opt]# chown elasticsearch:elasticsearch /data/elk_data/

## 启动
[root@node1 sbin]# systemctl start elasticsearch.service

## 查看端口
[root@node1 sbin]# netstat -antp |grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      27514/java

4.2、在node2安装ES

[root@node2 opt]# rz -E                 ## 上传es安装包
rz waiting to receive.
[root@node2 opt]# rpm -ivh elasticsearch-5.5.0.rpm                      ##  安装
警告：elasticsearch-5.5.0.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
正在升级/安装...
   1:elasticsearch-0:5.5.0-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
[root@node2 opt]# systemctl daemon-reload                     ### 加载配置文件
[root@node2 opt]# systemctl enable elasticsearch.service   ## 设置开机自启
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@node2 opt]# cp /etc/elasticsearch/elasticsearch.yml  
/etc/elasticsearch/elasticsearch.yml.bak                 ## 备份，防止配置失误


################# 注意：该配置文件每个冒号后面都有一个空格，切记不可删除######
[root@node2 opt]# vi /etc/elasticsearch/elasticsearch.yml   ## 修改文件

17/ cluster.name: my-elk-cluster                   ####集群名字
23/ node.name: node1                                  ####节点名字
33/ path.data: /data/elk_data                        ####数据存放路径
37/ path.logs: /var/log/elasticsearch/           ####日志存放路径
43/ bootstrap.memory_lock: false                 ####不在启动的时候锁定内存（前 
                              ## 端缓存。与IOPS-性能测试方式，每秒读写次数相关）
56/ network.host: 0.0.0.0                               ####提供服务绑定的IP地 
                                                ##  址，0.0.0.0代表所有地址
60/ http.port: 9200                                        ####侦听端口为9200
69/ discovery.zen.ping.unicast.hosts: ["node1", "node2"]           ####集群发 
                                                            现通过单播实现



##  查看刚才的配置
[root@node2 opt]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml  
cluster.name: my-elk-cluster
node.name: node1
path.data: /data/elk_data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock:false

network.host:0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node1", "node2"]


### 创建数据存放路径
[root@node2 opt]# mkdir -p /data/elk_data

###给目录授予权限
[root@node2 opt]# chown elasticsearch:elasticsearch /data/elk_data/

## 启动
[root@node2 sbin]# systemctl start elasticsearch.service

## 查看端口
[root@node2 elasticsearch]# netstat -antp |grep 9200
tcp6       0      0 :::9200                 :::*                    LISTEN      43165/java

查看两个节点信息

集群检查健康和状态

5、部署elasticsearch-head插件

上述查看集群的方式，及其不方便，我们可以通过安装elasticsearch-head插件后，来管理集群

5.1、在node1上安装插件

①在node1上安装node组件依赖包

[root@node1 opt]# ls          ## 上传node-v8.2.1.tar.gz压缩包
 
node-v8.2.1.tar.gz

[root@node1 opt]# yum install gcc gcc-c++ make -y   ## 安装依赖环境


[root@node1 opt]# tar xzvf node-v8.2.1.tar.gz  ## 解压缩

[root@node1 opt]# cd node-v8.2.1/
[root@node1 node-v8.2.1]# ./configure          ##初始化

[root@node1 node-v8.2.1]# make -j10            ##编译

[root@node1 node-v8.2.1]# make install         ##安装

②安装phantomjs前端框架

[root@node1 node-v8.2.1]# cd /usr/local/src/      ## 上传安装包
[root@node1 src]# rz -E 
rz waiting to receive.

## 解压安装包
[root@node1 src]# tar xjvf phantomjs-2.1.1-linux-x86_64.tar.bz2 

## 配置环境变量
[root@node1 src]# cd phantomjs-2.1.1-linux-x86_64/
[root@node1 phantomjs-2.1.1-linux-x86_64]# cd bin/
[root@node1 bin]# cp phantomjs /usr/local/bin/

③安装elasticsearch-head数据可视化工具

## 先上传安装包到/usr/local/src/
[root@node1 opt]# cd /usr/local/src/
[root@node1 src]# tar xzvf elasticsearch-head.tar.gz      ## 解压缩


[root@node1 src]# cd elasticsearch-head/
[root@node1 elasticsearch-head]# npm install            ## 安装
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/karma/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression

up to date in 21.873s

④修改著配置文件elasticsearch.yml

[root@node1 elasticsearch-head]# vim /etc/elasticsearch/elasticsearch.yml

''''''''''''''''''''''''''''''''
### 结尾插入两行####

http.cors.enabled: true        
http.cors.allow-origin: "*"

## 修改了配置文件，重启生效
[root@node1 elasticsearch-head]# systemctl restart elasticsearch.service

⑤启动elasticsearch-head 启动服务器

[root@node1 elasticsearch-head]# cd /usr/local/src/elasticsearch-head/
[root@node1 elasticsearch-head]# np
npm    nproc  npx    
[root@node1 elasticsearch-head]# npm run start &    ## 加载到后台启动

[1] 79757
[root@node1 elasticsearch-head]#           
> elasticsearch-head@0.0.0 start /usr/local/src/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100


[root@node1 elasticsearch-head]# ss -antp | grep 9100
LISTEN     0      128          *:9100                     *:*                   users:(("grunt",pid=79767,fd=12))
[root@node1 elasticsearch-head]# ss -antp | grep 9200
LISTEN     0      128         :::9200                    :::*                   users:(("java",pid=79421,fd=171))

5.2、在node2上安装插件

①在node2上安装node组件依赖包

[root@node2 opt]# ls          ## 上传node-v8.2.1.tar.gz压缩包
 
node-v8.2.1.tar.gz

[root@node2 opt]# yum install gcc gcc-c++ make -y   ## 安装依赖环境


[root@node2 opt]# tar xzvf node-v8.2.1.tar.gz  ## 解压缩

[root@node2 opt]# cd node-v8.2.1/
[root@node2 node-v8.2.1]# ./configure          ##初始化

[root@node2 node-v8.2.1]# make -j10            ##编译

[root@node2 node-v8.2.1]# make install         ##安装

②安装phantomjs前端框架

[root@node2 node-v8.2.1]# cd /usr/local/src/   ## 上传安装包
[root@node2 src]# rz -E
rz waiting to receive.


## 解压安装包
[root@node2 src]# tar xjvf phantomjs-2.1.1-linux-x86_64.tar.bz2 

## 配置环境变量
[root@node2 src]# cd phantomjs-2.1.1-linux-x86_64/
[root@node2 phantomjs-2.1.1-linux-x86_64]# cd bin/
[root@node2 bin]# cp phantomjs /usr/local/bin/

③安装elasticsearch-head数据可视化工具

## 先上传安装包到/usr/local/src/
[root@node2 opt]# cd /usr/local/src/
[root@node2 src]# tar xzvf elasticsearch-head.tar.gz      ## 解压缩


[root@node2 src]# cd elasticsearch-head/
[root@node2 elasticsearch-head]# npm install            ## 安装
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/karma/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression

up to date in 3.454s

④修改著配置文件elasticsearch.yml

[root@node2 elasticsearch-head]# vim /etc/elasticsearch/elasticsearch.yml

''''''''''''''''''''''''''''''''
### 结尾插入两行####

http.cors.enabled: true        
http.cors.allow-origin: "*"

## 修改了配置文件，重启生效
[root@node2 elasticsearch-head]# systemctl restart elasticsearch.service

⑤启动elasticsearch-head 启动服务器（此功能每次关机重启后需要再开启）

[root@node2 elasticsearch-head]# cd /usr/local/src/elasticsearch-head/
[root@node2 elasticsearch-head]# np
npm    nproc  npx    
[root@node2 elasticsearch-head]# npm run start &         ## 加载到后台启动
[1] 95468
[root@node2 elasticsearch-head]# 
> elasticsearch-head@0.0.0 start /usr/local/src/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100


[root@node2 elasticsearch-head]# ss -antp | grep 9100
LISTEN     0      128          *:9100                     *:*                   users:(("grunt",pid=95478,fd=12))
[root@node2 elasticsearch-head]# ss -antp | grep 9200
LISTEN     0      128         :::9200                    :::*                   users:(("java",pid=95134,fd=197))

5.3验证：

5.4、创建索引

方法一、直接在界面上创建

方法二、命令行创建索引

在node1主机，建立索引为index-demo,类型为test

[root@node1 elasticsearch-head]# curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'
{
  "_index" : "index-demo",
  "_type" : "test",
  "_id" : "1",
  "_version" : 2,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "created" : false
}

打开浏览器，查看新建的索引

6、安装logstash

安装logstash做日志搜集输出到elasticsearch中

在apache节点上进行配置

[root@apache ~]# systemctl stop firewalld     ### 关闭防火墙，安全中心

[root@apache ~]# setenforce 0
setenforce: SELinux is disabled

[root@apache ~]# yum -y install httpd   ## 安装apache

[root@apache ~]# systemctl start httpd  ## 启动
 

[root@apache ~]# java -version              ## java版本
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

[root@apache opt]# rz -E                    ## 上传jdk安装包
rz waiting to receive.

[root@apache opt]# rpm -ivh jdk-8u201-linux-x64.rpm 
警告：jdk-8u201-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:jdk1.8-2000:1.8.0_201-fcs        ################################# [100%]
Unpacking JAR files...
	tools.jar...
	plugin.jar...
	javaws.jar...
	deploy.jar...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar...
[root@apache opt]# vim /etc/profile
''''''''''''''''''''''''''''''''''''''''''
###增加#############

export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH

[root@apache opt]# source /etc/profile
[root@apache opt]# java -version            ### 统一版本
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

①安装logstash

[root@apache ~]# cd /opt/          ##上次安装包
[root@apache opt]# rz -E
rz waiting to receive.
[root@apache opt]# ls
logstash-5.5.1.rpm  rh


[root@apache opt]# rpm -ivh logstash-5.5.1.rpm   ## 安装


[root@apache opt]# systemctl enable logstash.service --now  ##设置为开机启动并立 
                                                                        即启动

## 建立软连接
[root@apache opt]#  ln -s /usr/share/logstash/bin/logstash /usr/local/bin/

②测试：logstash（Apache）与elasticsearch（node）功能是否正常，做对接测试

logstash命令的测试

字段描述：

-f 通过这个选项可以指定logstash的配置文件，根据配置文件配置logstash

-e 后面跟着字符串该字符串可以被当做logstash的配置（如果是”"，则默认使用stdin做为输入、stdout作为输出)

-t 测试配置文件是否正确，然后退出

③、logstash中input、filter、output的使用

input：采集

filter：过滤/格式化

output：输出

在apache节点测试：

[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{} }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.  ### 第一行报错，不用管
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
22:37:20.795 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
22:37:20.797 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
22:37:20.810 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"052967a6-0f74-43ec-b629-2920a4259fa9", :path=>"/usr/share/logstash/data/uuid"}
22:37:20.978 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
22:37:21.021 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
22:37:21.095 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com                                  #### 手动输入字符
2022-10-11T14:39:30.520Z apache www.baidu.com  #### 自动输出字符
www.taobao.com                                 #### 手动输入字符
2022-10-11T14:39:38.633Z apache www.taobao.com #### 自动输出字符
www.nicaicai.com                                 #### 手动输入字符
2022-10-11T14:39:56.901Z apache www.nicaicai.com #### 自动输出字符 
^C22:40:10.881 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
22:40:10.888 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

④、利用logstash 使用rubydebug显示详细输出，codec为一种编解码器

[root@apache opt]# logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
22:47:47.383 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
22:47:47.414 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
22:47:47.459 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com                                   ### 输入内容
{                                               ### 显示详细信息
    "@timestamp" => 2022-10-11T14:48:05.439Z,  
      "@version" => "1",
          "host" => "apache",
       "message" => "www.baidu.com"
}
www.taobao.com                                      ### 输入内容
{                                                   ### 显示详细信息
    "@timestamp" => 2022-10-11T14:48:14.346Z, 
      "@version" => "1",
          "host" => "apache",
       "message" => "www.taobao.com"
}

⑤、使用logstash将信息写入elasticsearch中，输入、输出、对接

[root@apache opt]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.159.13:9200"] } }'
[root@apache opt]# logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.159.13:9200"] } }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
22:58:48.970 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.159.13:9200/]}}
22:58:48.974 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.159.13:9200/, :path=>"/"}
22:58:49.080 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x2bb2e94e>}
22:58:49.081 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
22:58:49.198 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
22:58:49.205 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Installing elasticsearch template to _template/logstash
22:58:49.271 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x7a5c57e8>]}
22:58:49.274 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
22:58:49.307 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
22:58:49.354 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com                      ###输入下列内容
www.taobao.com
www.zhihu.com

到浏览器查看

7、 logstash配置文件，收集log

[root@apache opt]# ll /var/log/messages 
-rw-------. 1 root root 1825539 10月 11 23:16 /var/log/messages
[root@apache opt]# chmod o+r /var/log/messages 
[root@apache opt]# ll /var/log/messages 
-rw----r--. 1 root root 1832364 10月 11 23:17 /var/log/messages

①在配置文件/etc/logstash/conf.d/system.conf 中定义的是收集系统日志

采集系统日志放到ES内

[root@apache opt]# vim /etc/logstash/conf.d/system.conf 

input {
       file{
        path => "/var/log/messages"
        type => "system"
        start_position => "beginning"
        }
      }
output {
        elasticsearch {
          hosts => ["192.168.159.13:9200"]
          index => "system-%{+YYYY.MM.dd}"
          }
        }

#### 重启##############
[root@apache opt]# systemctl restart logstash.service

打开浏览器查看索引信息

8、在node1节点上安装kibana

[root@node1 elasticsearch-head]# cd /usr/local/src/  ## 上传安装包至指定位置
[root@node1 src]# rz -E
rz waiting to receive.
[root@node1 src]# ls                               ## 查看是否上传成功
elasticsearch-head         phantomjs-2.1.1-linux-x86_64
elasticsearch-head.tar.gz  phantomjs-2.1.1-linux-x86_64.tar.bz2
kibana-5.5.1-x86_64.rpm
[root@node1 src]# rpm -ivh kibana-5.5.1-x86_64.rpm  ##安装kibana的rpm包
警告：kibana-5.5.1-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:kibana-5.5.1-1                   ################################# [100%]

[root@node1 src]# cd /etc/kibana/
[root@node1 kibana]# ls
kibana.yml
[root@node1 kibana]# cp kibana.yml kibana.yml.bak  ## 备份配置文件
[root@node1 kibana]# vim kibana.yml       ##修改配置文件

2/ server.port: 5601                #### kibana打开的端口
7/ server.host: "0.0.0.0"           ####kibana侦听的地址
21/ elasticsearch.url: "http://192.168.226.128:9200"             
                                      ###和elasticsearch建立联系
30/ kibana.index: ".kibana"          ####在elasticsearch中添加.kibana索引

[root@node1 kibana]# grep -v "^#" kibana.yml |grep -v "^$"  ## 检验配置
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.159.13:9200"
kibana.index: ".kibana"




[root@node1 kibana]# systemctl restart kibana.service  ## 重启
[root@node1 kibana]# systemctl enable kibana.service   ## 设置开机自启
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.

打开浏览器

在上面可自行选择字段进行展示

9、对接Apache主机的Apache 日志文件（访问的、错误的）

在apache节点进行操作

[root@apache opt]# cd /etc/logstash/conf.d/
[root@apache conf.d]# ls
system.conf
[root@apache conf.d]# touch apache_log.conf
[root@apache conf.d]# vim apache_log.conf  ## 编写采集apache日志的配置文件
[root@apache conf.d]# cat apache_log.conf 
input {
       file{
        path => "/etc/httpd/logs/access_log"
        type => "access"
        start_position => "beginning"
        }
       file{
        path => "/etc/httpd/logs/error_log"
        type => "error"
        start_position => "beginning"
        } 
      }
output {
        if [type] == "access" {
        elasticsearch {
          hosts => ["192.168.159.13:9200"]
          index => "apache_access-%{+YYYY.MM.dd}"
          }
        }
        if [type] == "error" {
        elasticsearch {
          hosts => ["192.168.159.13:9200"]
          index => "apache_error-%{+YYYY.MM.dd}"
          }
        }
        }


[root@apache conf.d]# logstash -f apache_log.conf

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
00:06:57.442 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.159.13:9200/]}}
00:06:57.443 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.159.13:9200/, :path=>"/"}
00:06:57.476 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x20a21c4c>}
00:06:57.477 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
00:06:57.554 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
00:06:57.557 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x657d463b>]}
00:06:57.559 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.159.13:9200/]}}
00:06:57.559 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.159.13:9200/, :path=>"/"}
00:06:57.564 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x10dd1d49>}
00:06:57.566 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
00:06:57.571 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
00:06:57.575 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x415afa01>]}
00:06:57.576 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
00:06:57.652 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
00:06:57.683 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9601}
www.baidu.com   
www.huawei.com

打开浏览器，生成了apache的两个索引