在CDH中运行flume用于nginx日志采集
source:kafka 日志数据存储在kafka中
channel: memory channel 基于内存
sink: elasticsearch (es)
偶尔会发生,数据无法写入Elasticsearch的问题,channel中堆积的数据达到预设值,但是sink到es却停止了,一条数据也不进行写入。查询日志发现:
ERROR org.apache.flume.source.kafka.KafkaSource: KafkaSource EXCEPTION, {}
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at org.apache.flume.source.kafka.KafkaSource.doProcess(KafkaSource.java:295)
at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:60)
at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133)
at java.lang.Thread.run(Thread.java:748)
经过查询发现可能的解决方案:
https://help.aliyun.com/document_detail/158838.html
意思是比如:sinks的batchSize是 2000 ,但是可用的内存 为 20M ,2000个的数据量大于20M,则会出现无法写入的问题。
方案1:调小batchSize 需要根据现有实际的写入速度来定,如果写入速度太快,后端ES支持的写入足够大,但是Sink batchSize太小的话,可能会产生,积压到channel满的问题。
方案2:增大flume最大堆内存
JAVA_OPTS ="-Xmx20m" 改成 JAVA_OPTS = "-Xmx1024m" 根据实际服务器内存大小以及业务量 调整