消息队列kafka丢数据解决方案

 

ACK=0：发出消息不进行确认；

ACK=1：发出消息确认Leader接受成功；

ACK=-1：发出消息确认Leader和所有follower都接受成功。

 

MQ丢数据场景主要分为以下几种情况：

一、客户端未发出

1、Client端宕机或程序问题导致消息为发出，就报错了；

2、异步情况下，消息并非实时发出，而是积累到一定程度后发出，此时Client端出现问题导致积累的所有消息丢失。

二、网络异常

1、Client端发出了消息，由于网络原因，MQ没有收到；

        ACK=1可解决问题。

三、MQ的Leader未保存上

1、成功发送消息给MQ，但是MQ的Leader未保存上就宕机了，导致消息丢失；

        ACK=1可解决问题。

四、MQ的Leader保存上了，follower未保存上，然后Leader出现问题，导致消息丢失；

ACK=-1可解决问题。

此时如果follower上接收了部分消息，leader宕机了，对生产者来说也是没有ack确认，认为失败了，重发消息，可能导致消息重复，同样的消息可使用同一个key发送消息，这样消费者可以处理幂等。

 

总结：

高性能：异步+ack=0；

高可用：同步+ack=-1。

 

kafka官网文档介绍：http://kafka.apache.org/documentation/

acks

The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed:

• acks=0 If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1.
• acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.
• acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.

min.insync.replicas：最小同步副本数，如果总共3个副本，设置此值为2，则所有的副本都需要写入

When a producer sets acks to "all" (or "-1"), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).
When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.