starocks 实时日志分析
-
日志实时同步
- 技术路径
filebeat+kafka+starrocks
-
filebeat 部署
# 下载安装包 https://www.elastic.co/cn/downloads/beats/filebeat # tar解压 tar -zvxf filebeat-8.0.0-linux-x86_64.tar.gz
-
配置文件
## 新建配置文件 cp filebeat.yml filebeat-kafka.yml vim filebeat-kafka.yml ## 配置文件设置如下
filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. # filestream is an input for collecting log messages from files. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /opt/StarRocks-1.19.6/fe/log/fe.audit.log* output.kafka: # initial brokers for reading cluster metadata hosts: ["kafka1:9092", "kafka1:9092", "kafka1:9092"] topic: starocks_log_prd partition.round_robin: reachable_only: true keep_alive: 10s
-
启动脚本,抽取日志
nohup ./filebeat -c filebeat-kafka.yml &
-
查看kafka数据
{ "@timestamp": "2022-03-02T06:44:40.814Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "8.0.0" }, "input": { "type": "log" }, "agent": { "version": "8.0.0", "ephemeral_id": "a2187d49-ce13-4d83-8a50-49a86fc183b9", "id": "dfa6ed24-0493-4150-9af6-d849533b2561", "name": "p7bdsrdb104", "type": "filebeat" }, "ecs": { "version": "8.0.0" }, "host": { "name": "p7bdsrdb104", "architecture": "x86_64", "os": { "codename": "Maipo", "type": "linux",