filebeat

QChestnut

于 2024-08-02 16:38:40 发布

阅读量256

点赞数 8

文章标签： elk

本文链接：https://blog.csdn.net/QChestnut/article/details/140872139

版权

1、作用

1、可以在本机收集日志

2、也可以远程收集日志

3、轻量级的日志收集系统，可以在非java环境运行。

logstash是在jmv环境中运行，资源消耗很大，启动一个logstash要消耗500M左右的内存，filebeat只消耗10M左右的内存。

收集nginx的日志

2、收集nginx日志

systemctl restart nginx
systemctl stop firewalld
setenforce 0

#解压，将filebeat移到/usr/local下
tar -xf filebeat-6.7.2-linux-x86_64.tar.gz
mv filebeat-6.7.2-linux-x86_64 /usr/local/filebeat

#备份
cd /usr/local/filebeat
cp filebeat.yml filebeat.yml.bak

#修改配置文件
vim filebeat.yml
type: log
enabled: true
paths:
  - /usr/local/nginx/logs/access.log
  - /usr/local/nginx/logs/error.log
#开启日志收集，以及确定日志文本的路径，指定标签和发送到目标主机的logstash
  tags: ["nginx"]
  fields: 
    service_name: 192.168.230.21_nginx
    log_type_ nginx
    from: 192.168.230.21

#output.elasticsearch:
  # Array of hosts to connect to.
 #hosts: ["localhost:9200"]

outout elasticsearch
output.logsatsh:
hosts: ["192.168.230.30:5045"]
#5044是logstash默认的端口，只要是logstash主机上没有被占用的端口都可以使用，端口号要大于1024

#修改从21接收的nginx日志文件
vim nginx_21.conf

input {
  beats { port => "5045"}
}

output {
  if "nginx" in [tags] {
    elasticsearch {
       hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
       index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
    }
  }
}

#启动filebeat
nohup ./filebeat -e -c filebeat.yml > filebeat.out &
#-e：输出到标准输出
 -c：指定配置文件
 nohup：在系统的后台运行，不会因为终端的关闭导致程序停机运行；可以把运行的日志保存到指定文件
 
 #后后台运行
logstash -f nginx_61.conf --path.data /opt/test2 &

3、远程收集nginx、http、mysql日志

filebeat远程收集发送到logstash主机

展示的索引：

192.168.230.21_mysql-*

192.168.230.21_nginx-*

192.168.230.21_http-*

systemctl stop firewalld
setenforce 0

#修改配置文件
vim /etc/my.cnf
添加：
general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log

#安装httpd、nginx
yum -y install httpd nginx

#重启mysqld、httpd
systemctl restart mysqld
systemctl restart httpd

#修改nginx的端口号
listen       85;
#listen       [::]:80;

#重启nginx
systemctl restart nginx

#浏览器访问nginx、httpd

#将filebeat的安装包放到/opt目录下，并解压
tar -xf filebeat-6.7.2-linux-x86_64.tar.gz

#将filebeat-6.7.2-linux-x86_64放到/opt命令下，改名为filebeat
mv filebeat-6.7.2-linux-x86_64 filebeat

#修改filebeat的配置文件
vim filebeat.yml
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log
  tags: ["nginx"]
  fields:
    service_name: 192.168.230.21_nginx
    log_type: nginx
    from: 192.168.230.21

- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log
    - /var/log/httpd/error_log
  tags: ["httpd"]
  fields:
    service_name: 192.168.230.21_httpd
    log_type: httpd
    from: 192.168.230.21

- type: log
  enabled: true
  paths:
    - /usr/local/mysql/data/mysql_general.log
  tags: ["mysqld"]
  fields:
    service_name: 192.168.230.21_mysqld
    log_type: mysqld
    from: 192.168.230.21
    
#output.elasticsearch:
  # Array of hosts to connect to.
 #hosts: ["localhost:9200"]
 
 output.logstash:
  # The Logstash hosts
  hosts: ["192.168.230.30:5048"]
 
 
vim nmh_21.conf

input {
  beats { port => "5048"}
}

output {
  if "nginx" in [tags] {
    elasticsearch {
       hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
      index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
    }
  }

if "httpd" in [tags] {
    elasticsearch {
       hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
      index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
    }
  }

if "mysqld" in [tags] {
    elasticsearch {
       hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
      index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
    }
  }
}

#开启日志收集
nohup ./filebeat -e -c filebeat.yml > filebeat.out &

#后台运行
logstash -f nmh_21.conf --path.data /opt/test3 &

#多了httpd、nginx、mysqld的日志

#到kibana可视化界面创建索引模式

在这里插入图片描述

4、收集http日志

#安装httpd
yum -y install httpd

cd /etc/logstash
cd conf.d

#配置文件
vim http.conf

input {
   file {
      path => "/etc/httpd/logs/access_log"
      type => "access"
      start_position => "beginning"
   } 
   file {
      path => "/etc/httpd/logs/error_log"
      type => "error"
      start_position => "beginning"
   }
}

output {
    if [type] == "access" {
      elasticsearch {
      hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
      index => "apache_access-%{+YYYY.MM.dd}"
 } 
}
    if [type] == "error" {
      elasticsearch {
      hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
      index => "apache_error-%{+YYYY.MM.dd}"
 }
}

#启动logstash
logstash -f http.conf --path.data /opt/test1 &
显示：Successful started Logstash API endpoint {:port=>9601}
API接口：软件内部代码之间通信的接口，代码的连接点
端口是对外提供访问程序的内容接口

#到kibana可视化界面创建索引模式