starocks集群 实时日志分析

starocks 实时日志分析

---

1. 日志实时同步

   • 技术路径

filebeat+kafka+starrocks

• filebeat 部署

  # 下载安装包
  https://www.elastic.co/cn/downloads/beats/filebeat
  # tar解压
  tar -zvxf filebeat-8.0.0-linux-x86_64.tar.gz
  

• 配置文件

  ## 新建配置文件
  cp filebeat.yml filebeat-kafka.yml
  vim filebeat-kafka.yml	
  ## 配置文件设置如下
  

  filebeat.inputs:
  # Each - is an input. Most options can be set at the input level, so
  # you can use different inputs for various configurations.
  # Below are the input specific configurations.
  
  # filestream is an input for collecting log messages from files.
  - type: log
  
    # Change to true to enable this input configuration.
    enabled: true
  
    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      - /opt/StarRocks-1.19.6/fe/log/fe.audit.log*
      
      
  output.kafka:
    # initial brokers for reading cluster metadata
     hosts: ["kafka1:9092", "kafka1:9092", "kafka1:9092"]
     topic: starocks_log_prd
     partition.round_robin: 
       reachable_only: true
       keep_alive: 10s
  

• 启动脚本，抽取日志

   nohup ./filebeat -c filebeat-kafka.yml &
  

• 查看kafka数据

  {
       
    "@timestamp": "2022-03-02T06:44:40.814Z",
    "@metadata": {
       
      "beat": "filebeat",
      "type": "_doc",
      "version": "8.0.0"
    },
    "input": {
       
      "type": "log"
    },
    "agent": {
       
      "version": "8.0.0",
      "ephemeral_id": "a2187d49-ce13-4d83-8a50-49a86fc183b9",
      "id": "dfa6ed24-0493-4150-9af6-d849533b2561",
      "name": "p7bdsrdb104",
      "type": "filebeat"
    },
    "ecs": {
       
      "version": "8.0.0"
    },
    "host": {
       
      "name": "p7bdsrdb104",
      "architecture": "x86_64",
      "os": {
       
        "codename": "Maipo",
        "type": "linux",