filebeat配置问题收集

**下列操作基于filebeat版本为7.14.1和7.17.2和7.10.1中，其他的可以参考这个做调整

1、filebeat配置用日志字段替换@timestamp

下列为日志例子：
{"message":测试文件","context":{},"level":100,"level_name":"DEBUG","channel":"uat","datetime":"2024-03-13T15:30:48.310629+08:00","extra":{"uid":"111111111111111","cid":140379,"hostname":"XXXXXXXXXXXXX","process_id":6000,"memory_usage":"38 MB","file":"/var/www/html/test.php","line":93,"class":"test","function":"basicReplace"}}

目标：将json日志中的datetime的时间替换@timestamp的时间
方法：filebeat.yaml配置processors，具体如下：

filebeat.inputs:

- type: log
  enabled: true
  paths:
     - /app/logs/test-server/*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  fields:
     index: "log-test-%{+yyyy.MM.dd}"
  tags: ["log-test"]
  processors:     #这里的processors是这个log为这个log的
     - timestamp:
         source: "datetime"
         target: "@timestamp"
         field: datetime
         layouts:
            - "2006-01-02T15:04:05.999999999Z07:00" # 这里的格式需要与你的 datetime 字段格式相匹配

• 注意：processors可以为某个log的，也可以定义为全局的

2、切割某字段提取其中数据生成新字段

原来收集的字段：
{
  "_index": "log-test-2024.03.13",
  "_type": "_doc",
  "_id": "wN7UNo4Bycz3FWFFhTxx",
  "_version": 1,
  "_score": 1,
  "_source": {
    "@timestamp": "2024-03-13T08:01:10.768Z",
    ... ...
    "message": "test数据test数据test数据test数据test数据test数据test数据test数据test数据",
    "context": {},
    "level": 200,
    "log": {
      "offset": 109655,
      "file": {
        "path": "/app/logs/test-server/laravel-2024-03-13.log"
      }
    }
}

目标：将source的log.file.path的值中的test-server切出来，赋值给server_name
方法：filebeat.yaml配置processors，具体如下：

filebeat.inputs:

- type: log
  enabled: true
  paths:
     - /app/logs/test-server/*.log
  json.keys_under_root: true
  json.overwrite_keys: true
  fields:
     index: "log-test-%{+yyyy.MM.dd}"
  tags: ["log-test"]

processors:
  - script:
        lang: javascript
        id: my_filter
        tag: enable
        source: >
          function process(event) {
              var fields_index= event.Get("fields")["index"];
              if(fields_index.indexOf('test') != "-1"){      //根据上面的fields.index做一个判断
                  var path= event.Get("log")["file"]["path"];
                  var app=path.split('/')[3];                       //获取test-server
                  var logfile=path.split('/')[4];                    //获取
                  event.Put("log_file",logfile);
                  event.Put("server_name",app);
              }
              return event;
          }