kafka手动提交offset异常 org.apache.kafka.clients.consumer.CommitFailedException

最近做项目，需要手动提交kafka的offset，结果出现了下面的异常

1、异常

Exception in thread "Thread-2" org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

粗略看出，poll()时长超过了session.timeout.ms。解决办法可以增加session.timeout.ms或者减少max.poll.records

2、手动提交offset源码

props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
consumer.commitSync();

public class StorageController {
    private String GROUP_NAME = "MonitorDataGroup";
    private String TOPIC = "basp_monitor";
    private String SERVER = "";
    private String OFFSET = "earliest";

    public StorageController(String topic, String server) {
        this.TOPIC = topic;
        this.SERVER = server;
    }

    private Properties getProperties() {
        Properties props = new Properties();
        props.setProperty(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "100");
        props.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, SERVER);
        props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, GROUP_NAME);
        props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, OFFSET);
        props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

        String kerberos = YamlUtils.get(YamlUtils.CONFIG_KERBEROS_OPEN);
        if ("1".equals(kerberos)) {
            props.put("sasl.kerberos.service.name", "kafka");
            props.put("sasl.mechanism", "GSSAPI");
            props.put("security.protocol", "SASL_PLAINTEXT");
        }
        return props;
    }

    public void consume() {
        try {
            KafkaConsumer<String, String> consumer = new KafkaConsumer(getProperties());
            consumer.subscribe(Collections.singletonList(TOPIC));
            while (true) {
                try {
                    ConsumerRecords<String, String> records = consumer.poll(100);
                    if (!records.isEmpty()) {
                        boolean flag = true;
                        for (ConsumerRecord<String, String> record : records) {
                            //重复数据返回true
                            flag = flag && StorageService.dealMonitorData(record.value());
                            if (!flag) {
                                break;
                            }
                        }
                        if (flag) {
                            consumer.commitSync();
                        }
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

    }
}

3、解决方法解释

·heartbeat.interval.ms

心跳间隔。心跳是在consumer与coordinator之间进行的。心跳是确定consumer存活，加入或者退出group的有效手段。

    这个值必须设置的小于session.timeout.ms，因为：当Consumer由于某种原因不能发Heartbeat到coordinator时，并且时间超过session.timeout.ms时，就会认为该consumer已退出，它所订阅的partition会分配到同一group 内的其它的consumer上。（就是上面的异常）

通常设置的值要低于session.timeout.ms的1/3。    默认值是：3000 （3s）  记为H

·session.timeout.ms

Consumer session 过期时间。这个值必须设置在broker configuration中的group.min.session.timeout.ms 与 group.max.session.timeout.ms之间。  其默认值是：10000 （10 s） 记为S

·max.poll.records

Consumer每次调用poll()时取到的records的最大数  默认500。 记为 R

设每条记录操作时间 记为t

在H时长内不断发送H，只要有一次正常就认为不过期，则有R*t<S；

所以有两种解决办法可以增加session.timeout.ms或者减少max.poll.records，由于session.timeout.ms关联的参数限制影响比较大（比如·heartbeat.interval.ms），所以我将max.poll.records调为100。