日志服务系统搭建之：filebeat入门(下载、安装、入门使用)

本文链接：https://blog.csdn.net/weixin_42346767/article/details/101476614

步骤总结：

1.官网下载filebeat安装包；

2.下载之后移动至自己的软件目录下进行解压缩

3. 修改filebeat提供的配置文件：filebeat.yml文件

4. 测试本地启动，查看日志打印及输出

详细步骤解读：

1.下载filebeat的安装包

可以考虑在网上随便搜一下或者去elastic的官网下载：https://www.elastic.co/cn/

https://www.elastic.co/cn/downloads/past-releases/filebeat-5-6-0

根据系统选择适合自己的版本，主流应该就是我圈起来这几个

2.下载之后移动至自己的软件目录下进行解压缩

刚下载下来这样：

1. 移动
  a. 通过手动移动
  // 备注：/root/user/tools 是目标目录，比如自己的软件统一安装目录等
  b. mv ./filebeat-5.x.x-darwin-x86_64.tar_gz /root/user/tools

2. 解压缩
  tar -xzvf filebeat-5.x.x-darwin-x86_64.tar_gz

3. 重命名(将解压后的文件夹重命名为filebeat，便于查看使用)
  mv filebeat-5.x.x-darwin-x86_64 filebeat

解压缩后的文件夹目录，其中test目录是我自己准备的测试要使用的日志目录

3. 修改filebeat提供的配置文件：filebeat.yml文件

修改前：

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*




#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

vim filebeat.yml

// 需要修改的地方：
// 1. filebeat 的input配置，enable设置为true，表示使input配置生效
// /Users/mingqi/tools/test/test.log该目录是为了测试提前准备的目录及日志文件

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/mingqi/tools/test/test.log
    #- c:\programdata\elasticsearch\logs\*


// 2. filebeat 的output出口，添加如下配置
// 本地测试配置output的出口为打印台，

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- console ----------------------------------
output.console:
  enable: true

4. 测试本地启动，查看日志打印及输出

完成前边的准备工作之后开始测试

// 本地测试用可以考虑前台启动
localhost:filebeat mingqi$ pwd
/Users/mingqi/tools/filebeat

// 确认处于filebeat文件夹下执行./filebeat -c filebeat.yml 
// ./filebeat 可以使用绝对路径，后边的配置文件路径也要匹配
localhost:filebeat mingqi$ ./filebeat -c filebeat.yml 

// 启动成功后，console打印filebeat的输出
localhost:filebeat mingqi$ ./filebeat -c filebeat.yml 
{"@timestamp":"2019-09-26T13:44:00.282Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.3.2"},"host":{"id":"2062150B-C712-53A1-A2FA-1CE22AECF408","name":"localhost","hostname":"localhost","architecture":"x86_64","os":{"family":"darwin","name":"Mac OS X","kernel":"18.6.0","build":"18F132","platform":"darwin","version":"10.14.5"}},"agent":{"type":"filebeat","ephemeral_id":"bbd8db24-2d2f-4e7c-be9a-26db11e38da3","hostname":"localhost","id":"a7796e9f-02d0-4a35-a829-2ef64636ce9c","version":"7.3.2"},"log":{"offset":0,"file":{"path":"/Users/mingqi/tools/test/test.log"}},"message":"hello world ~","input":{"type":"log"},"ecs":{"version":"1.0.1"}}
{"@timestamp":"2019-09-26T13:44:03.259Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.3.2"},"log":{"offset":14,"file":{"path":"/Users/mingqi/tools/test/test.log"}},"message":"hello world~","input":{"type":"log"},"ecs":{"version":"1.0.1"},"host":{"hostname":"localhost","architecture":"x86_64","os":{"build":"18F132","platform":"darwin","version":"10.14.5","family":"darwin","name":"Mac OS X","kernel":"18.6.0"},"name":"localhost","id":"2062150B-C712-53A1-A2FA-1CE22AECF408"},"agent":{"type":"filebeat","ephemeral_id":"bbd8db24-2d2f-4e7c-be9a-26db11e38da3","hostname":"localhost","id":"a7796e9f-02d0-4a35-a829-2ef64636ce9c","version":"7.3.2"}}
{"@timestamp":"2019-09-26T13:44:03.259Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.3.2"},"input":{"type":"log"},"ecs":{"version":"1.0.1"},"host":{"os":{"family":"darwin","name":"Mac OS X","kernel":"18.6.0","build":"18F132","platform":"darwin","version":"10.14.5"},"name":"localhost","id":"2062150B-C712-53A1-A2FA-1CE22AECF408","hostname":"localhost","architecture":"x86_64"},"agent":{"version":"7.3.2","type":"filebeat","ephemeral_id":"bbd8db24-2d2f-4e7c-be9a-26db11e38da3","hostname":"localhost","id":"a7796e9f-02d0-4a35-a829-2ef64636ce9c"},"log":{"offset":27,"file":{"path":"/Users/mingqi/tools/test/test.log"}},"message":"2"}
{"@timestamp":"2019-09-26T13:44:03.259Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.3.2"},"ecs":{"version":"1.0.1"},"host":{"name":"localhost","os":{"version":"10.14.5","family":"darwin","name":"Mac OS X","kernel":"18.6.0","build":"18F132","platform":"darwin"},"id":"2062150B-C712-53A1-A2FA-1CE22AECF408","hostname":"localhost","architecture":"x86_64"},"agent":{"hostname":"localhost","id":"a7796e9f-02d0-4a35-a829-2ef64636ce9c","version":"7.3.2","type":"filebeat","ephemeral_id":"bbd8db24-2d2f-4e7c-be9a-26db11e38da3"},"log":{"offset":29,"file":{"path":"/Users/mingqi/tools/test/test.log"}},"message":"hello world~","input":{"type":"log"}}

// 明显可以看出我的日志内容是几行hello world~ ，其中的message字段就是日志数据

注意：结束后终止进程，再重新执行该命令，发现一直处于等待状态：

原因在于filebeat自动记录了beat对该日志文件的读取位置，上一次读到多少行。具体原因稍后解释。

追写日志：接下来我们再测一下如果这时候往日志文件中追加新的日志效果如何：

// 通过另启动一个终端，往测试的日志文件中追写日志

localhost:test mingqi$ echo "你好 filebeat~ " >> test.log 
localhost:test mingqi$ echo "你好 filebeat~ " >> test.log


// 在另一个终端的监控页面发现有新的日志出现
{"@timestamp":"2019-09-26T13:51:36.967Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.3.2"},"log":{"offset":735,"file":{"path":"/Users/mingqi/tools/test/test.log"}},"message":"你好 filebeat~ ","input":{"type":"log"},"ecs":{"version":"1.0.1"},"host":{"name":"localhost","hostname":"localhost","architecture":"x86_64","os":{"kernel":"18.6.0","build":"18F132","platform":"darwin","version":"10.14.5","family":"darwin","name":"Mac OS X"},"id":"2062150B-C712-53A1-A2FA-1CE22AECF408"},"agent":{"id":"a7796e9f-02d0-4a35-a829-2ef64636ce9c","version":"7.3.2","type":"filebeat","ephemeral_id":"e58002dd-947e-4dc3-a78c-cdb58c84f132","hostname":"localhost"}}
{"@timestamp":"2019-09-26T13:54:02.024Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.3.2"},"message":"你好 filebeat~ ","input":{"type":"log"},"ecs":{"version":"1.0.1"},"host":{"name":"localhost","hostname":"localhost","architecture":"x86_64","os":{"build":"18F132","platform":"darwin","version":"10.14.5","family":"darwin","name":"Mac OS X","kernel":"18.6.0"},"id":"2062150B-C712-53A1-A2FA-1CE22AECF408"},"agent":{"version":"7.3.2","type":"filebeat","ephemeral_id":"e58002dd-947e-4dc3-a78c-cdb58c84f132","hostname":"localhost","id":"a7796e9f-02d0-4a35-a829-2ef64636ce9c"},"log":{"offset":753,"file":{"path":"/Users/mingqi/tools/test/test.log"}}}

如图：

再来说一下刚才重新启动问什么什么数据都没有的问题：

在filebeat目录下有个data文件下，打开查看下边有一个文件，一个文件件

localhost:filebeat mingqi$ cd data/
localhost:data mingqi$ ls
meta.json	registry
localhost:data mingqi$

其中，registry就是filebeat记录日志读取位置的注册表，如需重新读取日志，删除此项即可。

补充：后台启动filebeat的服务

// 将所有标准输出及标准错误输出到/dev/null空设备，即没有任何输出
nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 & 

// 或者写到filebeat的日志文件中，在filebeat目录下
nohup ./filebeat -e -c filebeat.yml > filebeat.log &


localhost:filebeat mingqi$ pwd
/Users/mingqi/tools/filebeat
localhost:filebeat mingqi$ nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &
[1] 2004
localhost:filebeat mingqi$ ps -ef | grep filebeat
  501  2004  1941   0 10:01下午 ttys001    0:00.05 ./filebeat -e -c filebeat.yml
  501  2006  1941   0 10:01下午 ttys001    0:00.00 grep filebeat
localhost:filebeat mingqi$ 

// 通过查看filebeat的相关进程，发现果然已经在后台执行
// 如果是直接打印在filebeat.log的话，还可以监控日志查看输出
tail -150f filebeat.log