ELK日志分析系统（二）

【安装logstash并做一些日志搜集输出到elasticsearch中】

关闭防火墙关闭核心防护

systemctl stop firewalld.service 
setenforce 0

更改主机名

hostnamectl set-hostname apache

安装Apahce服务(httpd)

[root@apache ~]# yum -y install httpd
[root@apache ~]# systemctl start httpd  
[root@apche ~]# cd /var/log/httpd/
[root@apache httpd]# ls    ##开启服务的时候才产生的日志
access_log  error_log

安装Java环境

[root@apache ~]# java -version 	 ####如果没有装安装yum -y install java
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

附加

jdk			 java编译器		 1.5  1.8  1.11
源文件--------->[jdk编译器]---------->执行文件----------->jvm(java虚拟机执行)
xxx.java           		   xxx.class   	       结果

手工编译部署
xxx.tar.gz(源码包)
c/c++源码----------->[gcc/gcc-c++编译器]（借助make工具）---------->执行文件----------->make install

安装logstash

上传logstash-5.5.1-rpm到根目录下
[root@apache ~]# cd /opt
[root@apache ~]# rpm -ivh logstash-5.5.1.rpm   ##安装logstash
[root@apache ~]# systemctl start logstash.service  ##启动logstash
[root@apache ~]# systemctl enable logstash.service
[root@apache ~]# In -s /usr/share/logstash/bin/logstash /usr/local/bin/ ##建立logstash软连接

附加

[root@apache ~]# rpm -qc logstash   #查找配置文件位置
/etc/logstash/logstash.yml

输入采用标准输入输出采用标准输出

[root@apache ~]# logstash -e 'input{ stdin{} } output {  stdout{} }'
....（需要等待一会）
14:37:43.845 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com   #自行输入
2020-10-29T06:37:59.924Z apache www.baidu.com

使用rubydebug显示详细输出,codec为一种编解码器

[root@apache ~]# logstash -e 'input{ stdin{} } output {  stdout{ codec=>rubydebug } }'
....（需要等待一会）
14:47:48.735 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com
{
    "@timestamp" => 2020-10-29T06:47:54.671Z,
      "@version" => "1",
          "host" => "apache",
       "message" => "www.baidu.com"
}

使用logstash将信息写入elasticsearch中

[root@apache ~]# logstash -e 'input{ stdin{} } output {  elasticsearch{ hosts=>["20.0.0.21:9200"] } }'
....（需要等待一会）
14:54:14.277 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
www.baidu.com

logstash配置文件
Logstash配置文件主要由三部分组成: input、output、filter(根据需要)

[root@apache ~]# chmod o+r /var/log/messages
[root@apache ~]# ll /var/log/messages 
-rw----r--. 1 root root 817602 10月 29 15:03 /var/log/messages

[root@apache opt]# vim /etc/logstash/conf.d/system.conf
input {
        file{
          path => "/var/log/messages"
          type => "system"
          start_position => "beginning"
          }
        }
output {
        elasticsearch {
          hosts => ["20.0.0.21:9200"]
          index => "system-%{+YYYY.MM.dd}"
          }
        }
        
[root@apache conf.d]# systemctl restart logstash.service 

在主机安装kibana

上传kibana-5.5.1-x86_64.rpm 到/usr/local/src目录
[root@localhost ~]# cd /usr/local/src/
[root@localhost src]# rpm -ivh kibana-5.5.1-x86_64.rpm 

[root@localhost src]# cd /etc/kibana/
[root@localhost kibana]# cp kibana.yml  kibana.yml.bak

[root@localhost kibana]# vim kibana.yml
2/ server.port:5601  ####kibana打开的端口
7/ server.host:"0.0.0.0" ####kibana侦听的地址
21/ elasticsearch.url: "http://20.0.0.21:9200" ###和elasticsearch建立联系
30/ kibana.index: ".kibana"  ###在elasticsearch中添加.kibana索引

[root@localhost kibana]# systemctl start kibana.service  ###启动kibana服务
[root@localhost kibana]# systemctl enable kibana.service  ###开机启动kibana服务

使用浏览器输入http://20.0.0.12:5601/
首次登录创建一个索引名字:system-* ##这是对接系统日志文件
lndex name or pattern ##下面输入system-*
然后点最下面的出面的create按钮创建
然后点最左上角的Discover按钮会发现system-*信息
然后点下面的host旁边的add 会发现右面的图只有 Time 和 host 选项了这个比较友好

对接Apache主机的Apache日志文件（访问的、错误的)

[root@apache opt]# cd /etc/logstash/conf.d/
[root@apache conf.d]# touch apache_log.conf
[root@apache conf.d]# vim apache_log.conf 
input {
        file{
          path => "/etc/httpd/logs/access_log"  ##access日志存放位置
          type => "access"    ##标签（可自行定义）
          start_position => "beginning"   ##从beginning开始
          }
        file{
          path => "/etc/httpd/logs/error_log"
          type => "error"
          start_position => "beginning"
          }
                  }
output {
        if [type] == "access" {
        elasticsearch {
          hosts => ["20.0.0.21:9200"]
          index => "apache_access-%{+YYYY.MM.dd}"
          }
        }
   if [type] == "error" {
        elasticsearch {
          hosts => ["20.0.0.21:9200"]
          index => "apache_error-%{+YYYY.MM.dd}"
          }
        }
        }
          
 [root@apache conf.d]# /usr/share/logstash/bin/logstash -f apache_log.conf 

报错：

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
15:57:32.317 [LogStash::Runner] ERROR logstash.inputs.file - Unknown setting 'start_posotion' for file
15:57:32.324 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Something is wrong with your configuration."}

原因：
something is wrong with your configuration
您的配置有问题

打开浏览器输入http://20.0.0.21:9100/查看索引信息

创建apache_error- 和 apache_access- 的索引**

创建完成