0.8 is a huge step forward in functionality from 0.7.x. There are still some rough edges and likely some bugs, but we are feature complete and are beginning the process of rolling it out to various test clusters to see what happens. We wanted to make an early pre-release version available for the brave to try out even before all the documentation is up to date and production hardening is complete. Please let us know any problems you find.
This release includes the following major features:
- Partitions are now replicated. Previously the topic would remain available in the case of server failure, but individual partitions within that topic could disappear when the server hosting them stopped. If a broker failed permanently any unconsumed data it hosted would be lost. Starting with 0.8 all partitions have a replication factor and we get the prior behavior as the special case where replication factor = 1. Replicas have a notion of committed messages and guarantee that committed messages won't be lost as long as at least one replica survives. Replica logs are byte-for-byte identical across replicas.
- Producer and consumer are replication aware. When running in sync mode, by default, the producer send() request blocks until the messages sent is committed to the active replicas. As a result the sender can depend on the guarantee that a message sent will not be lost. Latency sensitive producers have the option to tune this to block only on the write to the leader broker or to run completely async if they are willing to forsake this guarantee. The consumer will only see messages that have been committed.
- The consumer has been moved to a "long poll" model where fetch requests block until there is data available. This enables low latency without frequent polling. In general end-to-end message latency from producer to broker to consumer of only a few milliseconds is now possible.
- We now retain the key used in the producer for partitioning with each message, so the consumer knows the partitioning key.
- We have moved from directly addressing messages with a byte offset to using a logical offset (i.e. 0, 1, 2, 3...). The offset still works exactly the same - it is a monotonically increasing number that represents a point-in-time in the log - but now it is no longer tied to byte layout. This has several advantages: (1) it is asthetically nice, (2) it makes it trivial to calculate the next offset or to traverse messages in reverse order, (3) it fixes a corner case interaction between consumer commit() and compressed message batches. Data is still transferred using the same efficient zero-copy mechanism as before.
- We have removed the zookeeper dependency from the producer and replaced it with a simple cluster metadata api.
- We now support multiple data directories (i.e. a JBOD setup).
- We now expose both the partition and the offset for each message in the high-level consumer.
- We have substantially improved our integration testing, adding a new integration test framework and over 100 distributed regression and performance test scenarios that we run on every checkin.
A complete list of changes is available here.
This is also our first backwards incompatible release--maintaining wire compatibility with the older code was simply too difficult given the volume of changes. We don't intend to make a habit of this. To ease the migration we have provided a migration tool to allow replicating from a 0.7.x cluster to a 0.8 cluster. This allows a no downtime migration: first create a 0.8 cluster and use the migration tool to replicate the 0.7 cluster's data to the 0.8 cluster, then move 0.7 consumers to 0.8, and finally move 0.7 producers to the new cluster.
1666
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
