kafka配置中的坑之: replica.fetch.max.bytes

首先我使用的版本是0.8.2

最近在使用kafka的时候发现服务器的网卡经常吃满，使用iftop和dstat看网络流量，发现kafka broker之间有很大流量。如果topic设置了多个replica，那么经常会有replica因为不能同步成功而掉线，最终topic的可用isr只剩下一个。

这个问题非常头疼，首先会导致机器网络io不正常，影响其他服务，其次会导致topic的备份失效。

在log4j.properties的配置中把 log4j.logger.kafka改为TRACE后，看到日志如下:

[2015-09-10 20:11:52,649] TRACE Processor id 0 selection time = 10121190 ns (kafka.network.Processor)
[2015-09-10 20:11:52,649] TRACE [KafkaApi-4] 1 bytes written to log test_queue-38 beginning at offset 679 and ending at offset 679 (kafka.server.KafkaApis)
[2015-09-10 20:11:52,649] TRACE Socket server received response to send, registering for write: Response(0,Request(0,sun.nio.ch.SelectionKeyImpl@14d964af,null,1441887112600,/192.168.201.238:46389),kafka.api.FetchResponseSend@7640d611,SendAction) (kafka.network.Processor)
[2015-09-10 20:11:52,649] DEBUG [KafkaApi-4] Produce to local log in 10 ms (kafka.server.KafkaApis)
[2015-09-10 20:11:52,649] TRACE Processor id 0 selection time = 36105 ns (kafka.network.Processor)
[2015-09-10 20:11:52,649] TRACE Bytes written as part of multisend call : 162Total bytes written so far : 162Expected bytes to write : 162 (kafka.api.TopicDataSend$$anon$1)
[2015-09-10 20:11:52,650] TRACE File