Flink cep监控 Flink任务日志

最新推荐文章于 2023-06-06 20:03:57 发布

黄瓜炖啤酒鸭

最新推荐文章于 2023-06-06 20:03:57 发布

阅读量814

点赞数 1

分类专栏： Flink 文章标签： flink flink日志 flink cep监控日志

本文链接：https://blog.csdn.net/qq_31866793/article/details/118079945

版权

本文介绍了如何配置Flink的日志4j，包括在IDEA本地测试和集群环境下的设置，以便将日志数据发送到Kafka topic。数据以'全域名'进行分割，便于后续分析。此外，通过CEP对Kafka中的错误日志进行分析，当检测到ERROR级别时触发报警。这是一个简单的日志监控和分析示例。

摘要由CSDN通过智能技术生成

1, 配置flink的log4j（idea本地测试）

#log4j.rootLogger=info,console
#log4j.appender.console=org.apache.log4j.ConsoleAppender
#log4j.appender.console.layout=org.apache.log4j.PatternLayout
#log4j.appender.console.layout.ConversionPattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n



log4j.rootLogger=WARN,kafka
# for package com.demo.kafka, log would be sent to kafka appender.
log4j.logger.kafka=WARN
log4j.logger.org.apache.kafka=WARN
log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.I0Itec.zkclient=WARN
## appender console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n

## appender kafka
log4j.appender.kafka=org.apache.kafka.log4jappender.KafkaLog4jAppender

#指定topic
log4j.appender.kafka.topic=aaa


log4j.appender.kafka.brokerList=192.168.6.34:9092,192.168.6.35:9092,192.168.6.36:9092


log4j.appender.kafka.compressionType=none
log4j.appender.kafka.syncSend=false
log4j.appender.kafka.layout=org.apache.log4j.PatternLayout
log4j.appender.kafka.ThresholdFilter.level=INFO
log4j.appender.kafka.ThresholdFilter.onMatch=ACCEPT
log4j.appender.kafka.ThresholdFilter.onMismatch=DENY

#log4j.appender.kafka.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n
log4j.appender.kafka.layout.ConversionPattern=%p||%c{1}||%L||%d{yyyy-MM-dd HH:mm:ss}||%m%n

2, 集群配置log4j.properties

rootLogger.appenderRef.rolling.ref = RollingFile
rootLogger.appenderRef.kafka.ref = Kafka
rootLogger.level = WARN


# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeep