librdkafka 打印连接断开错误

1. librdkafka日志

2016-01-24 08:56:58.773506|0x000022a4|error|gg::lqs::kafka::KafkaProducer::event_cb|KafkaProducer.cpp:363 FAIL: localhost:9092/bootstrap: Connection closed
2016-01-24 08:56:58.787507|0x00002a60|error|gg::lqs::kafka::KafkaProducer::event_cb|KafkaProducer.cpp:363 Error event: -195: Local: Broker transport failure - localhost:9092/bootstrap: Connection closed

2. kafka 日志

less ../log/kafka.log | grep "Removed 0 expired offsets in 0 milliseconds."
[2016-07-08 05:11:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:21:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:31:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:41:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:51:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:01:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:11:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:21:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:31:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:41:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:51:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:01:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:11:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:21:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:31:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:41:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)

3. 现象说明

经过测试，kafka中并不会少数据。

kafka 0.9.0+ 版本会每隔一段时间主动关闭空闲连接，默认是10分钟。上面kafka也是每隔十分钟才会出现特定的日志信息。

在librdkafka中，有一个决定是否打印断连信息的开关。如果对接的是0.9.0+版本的kafka，建议关闭。在CONFIGURATION.md中有如下一段说明，该开关默认是开启的。

log.connection.close                     |  *  |          true | Log broker disconnects. It might be useful to turn this off when interacting with 0.9 brokers with an aggressive `connection.max.idle.ms` value.

关闭该开关的办法：在调用librdkafka创建kafka实例时，增加配置。

if (rd_kafka_conf_set(rk_conf, "log.connection.close", "false", errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {
    writeLog("set log.connection.close failed.\n");
}

rk = rd_kafka_new(RD_KAFKA_PRODUCER, rk_conf, errstr, sizeof(errstr));

4. github issue

https://github.com/edenhill/librdkafka/issues/516