日志收集

elk 解决日志收集问题记录

整体架构
依赖服务安装&部分实现
一、安装Kafka

下载地址

https://mirrors.bfsu.edu.cn/apache/kafka/2.7.0/kafka_2.13-2.7.0.tgz

https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.7.0/kafka_2.13-2.7.0.tgz

参考wiki：https://blog.csdn.net/qq_39314099/article/details/99710706?utm_term=linux%E4%B8%8B%E5%AE%89%E8%A3%85kafka&utm_medium=distribute.pc_aggpage_search_result.none-task-blog-2_allsobaiduweb~default-0-99710706&spm=3001.4430

查看所有topic： ./bin/kafka-topics.sh --list --zookeeper localhost:2181

nohup sh zookeeper-server-start.sh /kafka/config/zookeeper.properties >/dev/null 2>&1 &

查看消费组消息：./kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic TEST-TOPIC --from-beginning

查看创建的消费组：./kafka-topics.sh --list --zookeeper localhost:2181

创建topic：sh kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 3 --topic TEST-TOPIC

二、安装filebeat

filebeat.yml 文件

filebeat.inputs:
 
- type: log
 
  enabled: true
 
  paths:
     - /home/xiaoju/logs/manhattan/manhattan-oppenheimer-wolverine/manhattan-oppenheimer-wolverine-biz.log
  include_lines: ['_com_request_out']
name: "10.190.6.206"
output.kafka:
  enabled: true
  hosts: ["10.190.34.86:9092"]
  topic: 'TEST-TOPIC'
  partition.round_robin:
    reachable_only: false
 
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
logging.level: debug

启动 nohup ./filebeat -e -c filebeat.yml > filebeat.log &

三、Java代码

添加依赖

<dependency>
   <groupId>org.springframework.kafka</groupId>
   <artifactId>spring-kafka</artifactId>
   <version>2.4.0.RELEASE</version>
</dependency>

添加kafka消费代码

@Component
@Slf4j
public class KafkaMyConsumer {
 
    @KafkaListener(topics = {"TEST-TOPIC"})
    public void listener(ConsumerRecord record){
        Optional msg = Optional.ofNullable(record.value());
        JSONObject jsonObject = JSONObject.parseObject((String) msg.get());
        String source = (String) jsonObject.get("source");
        String message = (String) jsonObject.get("message");
        // TODO 现在写的比较定制化 只能匹配Manhattan项目 后面改进
        String serviceName = source.split("/")[5];
        String[] splitMessage = message.split("\\|\\|");
        String args= splitMessage[1];
        String response = splitMessage[6];
        response = response.replace("response=","");
        args= args.replace("args=","");
        JSONObject jsonObjectArgs = JSONObject.parseObject(args);
        JSONObject header = (JSONObject) jsonObjectArgs.get("header");
        String host= (String) header.get("host");
        String contentType= String.valueOf(header.get("content-type"));
        String headerStr = String.valueOf(jsonObjectArgs.get("header"));
        String parameters= String.valueOf(jsonObjectArgs.get("parameters"));
        String body= String.valueOf(jsonObjectArgs.get("body"));
        String uri= String.valueOf(jsonObjectArgs.get("uri"));
        String requestMethod = uri.split("-")[1];
        insertMysql(serviceName,host,contentType,requestMethod,headerStr,parameters,body,response);
    }
 
 
    public void insertMysql(String serviceName,String host,String contentType,String requestMethod,String headerStr,String parameters,String body,String response ){
 
        log.info("--serviceName---"+serviceName+"----host----"+host+"--" +
                "-contentType-----"+contentType+"---requestMethod-------"+requestMethod+"-" +
                "-headerStr---"+headerStr+"---parameters--"+parameters+"--body--"+body+"-" +
                "-response---"+response);
    }
 
}

剩下的就是日志的存储和展示啦