Kafka Study : Understand the Partitions and replicates

原创 2016年08月31日 09:56:35
test configurations:
  1. start zookeeper at port 2181
  2. start kafka instance 0 at port 9092 (using default server.properties).
    • kafka_folder] bin/kafka-server-start.sh
  3. start kafka instance 1
    • create new server1.properteis, change the id=1, port=9093 and log directory;
    • bin/kafka-server-start.sh config/server1.properties
  4. start kafka instance 2
    • create new server2.properteis, change the id=2, port=9094 and log directory;
    • bin/kafka-server-start.sh config/server2.properties
  5. create topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic my-replicated-test-topic
  6. check the topic partion and replicates: 
          kafka-topics.sh --describe --topic my-replicated-test-topic --zookeeper localhost:2181
    • Topic:my-replicated-test-topic  PartitionCount:3        ReplicationFactor:2     Configs:
    •        Topic: my-replicated-test-topic Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
    •         Topic: my-replicated-test-topic Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
    •        Topic: my-replicated-test-topic Partition: 2    Leader: 2       Replicas: 2,0   Isr: 2,0

  1. publish message through different kafka instance:
    • kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-test-topic
      • 1st message
      • 2nd message
      • 3rd message
    • kafka-console-producer.sh --broker-list localhost:9093 --topic my-replicated-test-topic
      • 4th message
      • 5th message
      • 6th message
    • kafka-console-producer.sh --broker-list localhost:9094 --topic my-replicated-test-topic
      • 7th message
      • 8th message
      • 9th message
  2. check the storage:                   
    kafka instance 0 kafka instance 1 kafka instance 2
    my-replicated-test-topic-replicate-0:
    2nd message, 6th message, 8th message
    my-replicated-test-topic-replicate-0:
    2nd message, 6th message, 8th message
     
      my-replicated-test-topic-replicate-1:
    1st message, 5th message, 7th message
    my-replicated-test-topic-replicate-1:
    1st message, 5th message, 7th message
    my-replicated-test-topic-replicate-2:
    3rd message, 4th message, 9th message
      my-replicated-test-topic-replicate-2:
    3rd message, 4th message, 9th message
     
  3. consumer from beginning from any broker:
  4. kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-test-topic --from-beginning
  5. 1st message
  6. 2nd message
  7. 5th message
  8. 7th message
  9. 6th message
  10. 8th message
  11. 3rd message
  12. 4th message
  13. 9th message
  14. we can see that: the order from one partition was assured, but from different partitions is not assured

Apache Kafka笔记(二):Topics,Partitions and Brokers

Topics,Partitions and Replication Factor在消息订阅发布过程中的分析。

Kafka分区分配计算(分区器Partitions)

KafkaProducer在调用send方法发送消息至broker的过程中,首先是经过拦截器Inteceptors处理,然后是经过序列化Serializer处理,之后就到了Partitions阶段,即...

如何为Kafka集群选择合适的Topics/Partitions数量

这是许多kafka使用者经常会问到的一个问题。本文的目的是介绍与本问题相关的一些重要决策因素,并提供一些简单的计算公式。 文章目录 [hide] 1 越多的分区可以提供更高的吞吐量...

11G Concept 第四章翻译 Partitions, Views, and Other Schema Objects(分区,视图,以及其他Schema Objects)

虽然表和索引是最重要、最常用的schema Objects,但是数据库还是很多其他schema objects。其中最常用的都在本章介绍。 本章包含下面几小节: ·Overview of Part...

Item23 Understand std::move and std::forward

这个系列的文章来自于Effective Modern C++的读书笔记,我抽取了其中比较重要的,不容易理解的,平常我们开发过程中也不太在意的一些Item进行分析。 ​ std::move和std:...

Understand Wave Format, and Implement a Wave Reader

I found there is no detail discussion for extra data of wave format. Here I post a explicit (for mys...

Understand Linux Shell and Basic Shell Scripting Language Tips-Part I

Picture speak more than words and the below picture says all about the working of Linux.   Und...
  • Viidiot
  • Viidiot
  • 2013年09月21日 08:49
  • 1141

Use gitk to understand git – merge and rebase

This is the second part of my Use gitk to understand git post. In my initial overview, I demons...

How to read and understand a scientific paper: a guide for non-scientists

How to read and understand a scientific paper: a guide for non-scientists        网站链接:http://viole...

how to understand TSs – S1 handover with MME and SGW relocation and Indirect Tunneling

http://www.imacandi.net/windancer/2010/04/08/how-to-understand-tss-s1-handover-with-mme-and-sgw-relo...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:Kafka Study : Understand the Partitions and replicates
举报原因:
原因补充:

(最多只允许输入30个字)