filebeat日志收集实践

最新推荐文章于 2024-07-24 16:54:22 发布

Gun&rose

最新推荐文章于 2024-07-24 16:54:22 发布

阅读量1.4k

点赞数

文章标签： filebeat elk

本文链接：https://blog.csdn.net/weixin_40224560/article/details/96299265

版权

本文介绍了在ELK架构中，Filebeat作为日志收集工具的使用方法。首先，讲述了如何下载与配置Filebeat 6.6.1，强调必须在启动Filebeat前先启动Logstash。接着，详细说明了Logstash的下载、配置过程，并展示了如何启动Logstash。最后，提到了通过Kibana来查看和分析收集的日志。

摘要由CSDN通过智能技术生成

在ELK的架构中，Beats取代了logstash，直接扮演了面向日志源的角色，只专做＂收集＂一项工作，而logstash则负责居中的过滤和分析

filebeat 6.6.1源码包

前提ELK已部署完成，其中elasticsearch，kibana，logstash全部采用6.6.1源码包

下载

从官网上直接下载

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.1-linux-x86_64.tar.gz

配置

解压并复制到/usr/local/filebeat，目录下包括filebeat的配置文件和log文件等


/usr/local/filebeat$ ls -l
total 36476
drwxr-x---  2 root root     4096 Jul 17 04:10 data
-rw-r--r--  1 root root   131266 Jul 17 02:51 fields.yml
-rwxr-xr-x  1 root root 36927942 Jul 17 02:51 filebeat
-rw-r--r--  1 root root    69578 Jul 17 02:51 filebeat.reference.yml
-rw-------  1 root root     7717 Jul 17 03:08 filebeat.yml
drwxr-xr-x  4 root root     4096 Jul 17 02:51 kibana
-rw-r--r--  1 root root    13675 Jul 17 02:51 LICENSE.txt
drwx------  2 root root     4096 Jul 17 03:17 logs
drwxr-xr-x 20 root root     4096 Jul 17 02:51 module
drwxr-xr-x  2 root root     4096 Jul 17 02:51 modules.d
-rw-r--r--  1 root root   163067 Jul 17 02:51 NOTICE.txt
-rw-r--r--  1 root root      802 Jul 17 02:51 README.md

修改filebeat.yml


- type: log

  # Change to true to enable this input configuration.
  enabled: true #此处修改为true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/auth.log  #此处配置需要收集的日志文件路径


#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:　＃注释输出到elasticsearch


#----------------------------- Logstash output -------------------------------
-
output.logstash:　　＃开启输出到logstash
  # The Logstash hosts
  hosts: ["10.x.x.5:5044"]

配置完成后，需先将logstash服务启动后，才能运行filebeat,否则将报错不能正常运行

logstash 6.6.1源码

下载


https://www.elastic.co/downloads/past-releases/logstash-6-6-1

配置

下载完成后，解压将其复制到 /usr/local/logstash/，然后在 /usr/local/logstash/config/下，复制logstash-sample.conf到配置文件logstash.conf

配置/usr/local/logstash/config/logstash.conf

input {
   
  beats {
   
    port => 5044　#默认filebeat输出的端口，可自定义
  }
}

output {
   
  elasticsearch {
   
    hosts => ["http://10.x.x.5:9200"]
    index => "backup-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

配置/usr/local/logstash/bin/logstash.yml
#修改如下内容,ip与elasticsearch配置中的ip的一致

http.host: "10.x.x.5"
http.port: 9600-9700

运行logstash

/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf

Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2019-07-16T23:39:15,460][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-07-16T23:39:15,485][INFO ][logstash.runner          ] Starting Logstash {
   "logstash.version"=>"6.6.1"}
[2019-07-16T23:39:24,435][INFO ][logstash.pipeline        ] Starting pipeline {
   :pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-07-16T23:39:25,277][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {
   :changes=>{
   :removed=>[], :added=>[http://10.x.x.5:9200/]}}
[2019-07-16T23:39:25,556][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {
   :url=>"http://10.26.80.50:9200/"}
[2019-07-16T23:39:25,693][INFO ][logstash.outputs.elasticsearch