目录
背景
公司提供比较粗糙的ELK 监控的stack ,目前项目的监控是 collected + grafana 这种。 APP 内日志通过java 的日志门面 SLF4j 滚动输出到日志文件中。
集成KAFKA appender
maven
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<version>0.10.1.0</version>
</dependency>
配置log4j.properties
log4j.rootLogger=INFO,console,KAFKA
## appender KAFKA
log4j.appender.KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.KAFKA.topic=xx
log4j.appender.KAFKA.brokerList=xx:9092
log4j.appender.KAFKA.compressionType=gzip
log4j.appender.KAFKA.syncSend=false
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n
## appender console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n
查看网上一些比较老的版本,配置的appender类是 kafka.producer.KafkaLog4jAppender,开始直接黏贴过来发现找不到这个类,仔细看一下依赖的包 ,用org.apache.kafka.log4jappender.KafkaLog4jAppender 这个类
kafka-log4j-appender 就这一个类~
客户端LOG 的代码
import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class ElkLogTest { private static final Logger LOG = LoggerFactory.getLogger(ElkLogTest.class); public static void main(String[] args){ int count = 0; while (count <= 1000){ LOG.info("test kafka appender"); try { Thread.sleep(200); } catch (InterruptedException e) { e.printStackTrace(); } } } }
kafka 开启consumer 监控输出
/home/centos/kafka_2.10-0.10.2.2/bin/kafka-console-consumer.sh --bootstrap-server xxxx:9092 --topic xxx --from-beginning
结果->
集成Logstash ES Kibana
上边的简单配置使用,只是达到了通达的目的,确实日志通过Log4j的 KAFKA Appender 发送出去了,但是kafka 发出的日志记录要经过 logstash 解析,进入到 Elastic Search 被存储和索引,最终查询监控。
这里logstash 做的就是从kafka topic 里边读取,做转换,写入到ES 里边 。
logstash 配置文件demp.conf
input {
kafka {
bootstrap_servers => "kafka:9092"
codec => json
group_id => group
client_id => client_id
auto_commit_interval_ms => "30000"
auto_offset_reset => latest
topics => 'log1'
}
}
output {
elasticsearch {
hosts => ["es-host:9200"]
index => "log1-%{+YYYY.MM.dd}"
user => "u"
password => "p"
}
}
这里设置从kafka 中读取数据利用json 解码
###启动logstash
/bin/logstash -f demo.conf --path.data=data1 --path.logs=log1
logstash日志会输出类似如下信息
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2021-03-08 02:54:28.982 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-03-08 02:54:28.995 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.3"}
[INFO ] 2021-03-08 02:54:33.725 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2021-03-08 02:54:34.076 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash:xxxxxx@elkes.dev.aws.fwmrm.net:9200/]}}
[WARN ] 2021-03-08 02:55:33.515 [[main]-pipeline-manager] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash:xxxxxx@kafka-host:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash:xxxxxx@kafka-host:9200/][Manticore::ClientProtocolException] kafka-host:9200 failed to respond"}
[INFO ] 2021-03-08 02:55:33.533 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//kafka-host:9200"]}
[INFO ] 2021-03-08 02:55:33.572 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x52c79b9d run>"}
[INFO ] 2021-03-08 02:55:33.680 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-03-08 02:55:33.734 [[main]<kafka] ConsumerConfig - ConsumerConfig values:
auto.commit.interval.ms = 30000
auto.offset.reset = latest
bootstrap.servers = [kafka-host:9092]
应用程序运行后,有log 输出到kafka, 秒级ES 建立索引就可以查询了
###至此, 真个ELK日志搜集,转换,传输,存储,索引,可视化查询等简单版本完成
应用分组规划
因为实际情况的原因,我们不会一个小应用就创建一个kafka 的topic,或者有时候我们只是会检索一个应用中一个小组件的日志,这个时候的情况是 kafka topic 只有一个, logstash 实例只有一个,ELK索引只有一个,但是为了更加方便的查询,尤其是在Flink on yarn 的环境下,需要使用一种灵活的方式在 logger 的json 输出中增加标记字段。
### 在配置文件中添加
appender.kafka.layout.type = JsonLayout
appender.kafka.layout.extra.type = KeyValuePair
appender.kafka.layout.extra.key = App-Name
appender.kafka.layout.extra.value = ${sys:AppName}
启动java 程序时候 java 系统属性添加 -DAppName=Flin