学习filebeat入门

最新推荐文章于 2024-08-30 10:16:19 发布

LargeOrangeCoder

最新推荐文章于 2024-08-30 10:16:19 发布

阅读量241

点赞数 1

分类专栏： ELK 文章标签： es

本文链接：https://blog.csdn.net/LargeOrangeCoder/article/details/111061748

版权

ELK 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

filebeat + kafka日志收集

背景

入职新公司，让我用filebeat 收集日志输出到kafka，做简单的日志收集工作

filebeat安装

本机用的是mac os所以直接采用brew 安装filebeat

brew install filebeat

安装完成之后

 ~  filebeat version
filebeat version 7.10.0 (amd64), libbeat 7.10.0 [1428d58cf2ed945441fb2ed03961cafa9e4ad3eb built 2020-11-09 19:57:03 +0000 UTC]

zookeeper和kafka安装

安装zookeeper

brew install zookeeper

验证完成

~  brew services list
Name              Status  User        Plist
tomcat            stopped
zookeeper         stopped

启动zookeeper

brew services start zookeeper

验证启动

~  brew services list
Name              Status  User        Plist
tomcat            stopped
zookeeper         started dacheng.sun /Users/dacheng.sun/Library/LaunchAgents/homebrew.mxcl.zookeeper.plist

安装kafka

brew 安装kafka

brew install kafka

验证安装

~  brew services list
Name              Status  User        Plist
tomcat            stopped
kafka             stopped
zookeeper         started dacheng.sun /Users/dacheng.sun/Library/LaunchAgents/homebrew.mxcl.zookeeper.plist

启动kafka

~  brew services start kafka
~  brew services list
Name              Status  User        Plist
tomcat            stopped
kafka             started dacheng.sun /Users/dacheng.sun/Library/LaunchAgents/homebrew.mxcl.kafka.plist
zookeeper         started dacheng.sun /Users/dacheng.sun/Library/LaunchAgents/homebrew.mxcl.zookeeper.plist

启动一个golang服务定时输出日志

func main() {
	fd, _ := os.OpenFile("a.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)
	defer fd.Close()
	for true {
		fd_content := strings.Join([]string{`time="2020-12-11T16:14:34+08:00" level=info msg="kuaishou SyncVendorAllDataList sync ks all data start, seq:0, length:1274, index:83, vendor account id:3651, 7825762" func="go-makepolo/adserver/cron.(*KuaishouSyncHistoryCron).SyncVendorAllDataList.func1" file="ad_history_cron.go:170"`, "\n"}, "")
		buf := []byte(fd_content)
		fd.Write(buf)
		time.Sleep(2 * time.Second)
	}
}

启动filebeat

filebeat config yml

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /data/deploy/makepolo/adserver/log.log #读取文件位置 完整路径
#output.console: #这个是直接输出到console的
#  pretty: true
#  enable: true
output.kafka:
  hosts: ["127.0.0.1:9092"]
  topic: filebeat_a_log
  version: 2.0.0
  enable: true

启动filebeat

filebeat -e -c {filebeat_path}/filebeat_config_test.yml # 完整路径

启动kafka 消费者

~  kafka-console-consumer --bootstrap-server localhost:9092 --topic filebeat_a_log --from-beginning
{"@timestamp":"2020-12-11T08:04:22.241Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.10.0"},"agent":{"id":"e375f1ee-4183-4248-9f03-dfd7cc5f72be","name":"MacBook-Pro.local","type":"filebeat","version":"7.10.0","hostname":"MacBook-Pro.local","ephemeral_id":"4521e0b0-eb3d-4861-ba7c-f0cff6c4ab45"},"ecs":{"version":"1.6.0"},"log":{"offset":57997,"file":{"path":"/Users/dacheng.sun/go/src/study/a.log"}},"message":"======2020-12-11 16:04:20=====test log","input":{"type":"log"},"host":{"name":"MacBook-Pro.local"}}
{"@timestamp":"2020-12-11T08:04:23.247Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.10.0"},"message":"======2020-12-11 16:04:22=====test log","input":{"type":"log"},"ecs":{"version":"1.6.0"},"host":{"name":"MacBook-Pro.local"},"agent":{"version":"7.10.0","hostname":"MacBook-Pro.local","ephemeral_id":"4521e0b0-eb3d-4861-ba7c-f0cff6c4ab45","id":"e375f1ee-4183-4248-9f03-dfd7cc5f72be","name":"MacBook-Pro.local","type":"filebeat"},"log":{"offset":58036,"file":{"path":"/Users/dacheng.sun/go/src/study/a.log"}}}
{"@timestamp":"2020-12-11T08:04:26.257Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.10.0"},"host":{"name":"MacBook-Pro.local"},"agent":{"name":"MacBook-Pro.local","type":"filebeat","version":"7.10.0","hostname":"MacBook-Pro.local","ephemeral_id":"4521e0b0-eb3d-4861-ba7c-f0cff6c4ab45","id":"e375f1ee-4183-4248-9f03-dfd7cc5f72be"},"log":{"offset":58075,"file":{"path":"/Users/dacheng.sun/go/src/study/a.log"}},"message":"======2020-12-11 16:04:24=====test log","input":{"type":"log"},"ecs":{"version":"1.6.0"}}
{"@timestamp":"2020-12-11T08:04:27.260Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.10.0"},"log":{"offset":58114,"file":{"path":"/Users/dacheng.sun/go/src/study/a.log"}},"message":"======2020-12-11 16:04:26=====test log","input":{"type":"log"},"ecs":{"version":"1.6.0"},"host":{"name":"MacBook-Pro.local"},"agent":{"id":"e375f1ee-4183-4248-9f03-dfd7cc5f72be","name":"MacBook-Pro.local","type":"filebeat","version":"7.10.0","hostname":"MacBook-Pro.local","ephemeral_id":"4521e0b0-eb3d-4861-ba7c-f0cff6c4ab45"}}