KAFKA

install apache kafka

 

1 - download zookeeper of version 3.4.10

http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/

 

 

2 - download kafka of version kafka_2.12-1.0.0

http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.12-1.0.0.tgz

 

3 - follow the instruction 

http://programming-tips.in/kafka-set-up-apache-kafka-on-windows/

 

4 - add sasl for kafka:

 

4.1 add the following configuration to file C:\kafka\config\server.properties

listeners=SASL_PLAINTEXT://me:9092

 

security.inter.broker.protocol=SASL_PLAINTEXT 

 

sasl.enabled.mechanisms=PLAIN

 

sasl.mechanism.inter.broker.protocol=PLAIN

 

4.2 create a new file nanmed C:\kafka\config\kafka_server_jaas.conf and add the following configuration to it.

KafkaServer {

   org.apache.kafka.common.security.plain.PlainLoginModule required

   username="admin"

   password="adminsecret"

   user_admin="adminsecret"

   user_kafka="kafkasecret";

};

 

4.3 create a new file named C:\kafka\config\kafka_client_jaas.conf and add the following configration to it.

KafkaClient {

org.apache.kafka.common.security.plain.PlainLoginModule required

username="kafka"

password="kafkasecret";

};

 

4.4 add this configuration to file producer.properties and consumer.properties seprately(for command line use later)

security.protocol=SASL_PLAINTEXT

 

sasl.mechanism=PLAIN

 

 

4.5 add this command to file C:\kafka\bin\windows\kafka-server-start.bat

IF ["%KAFKA_OPTS%"] EQU [""] (

 

    set KAFKA_OPTS=-Djava.security.auth.login.config=c:/kafka/config/kafka_server_jaas.conf

 

)

 

4.6 add the following line to file C:\kafka\bin\windows\kafka-console-producer.bat

set KAFKA_OPTS=-Djava.security.auth.login.config=c:/kafka/config/kafka_client_jaas.conf

 

4.7 add the following line to file C:\kafka\bin\windows\kafka-console-consumer.bat

set KAFKA_OPTS=-Djava.security.auth.login.config=c:/kafka/config/kafka_client_jaas.conf

 

 

The following command are all run in kafka directory:

***start zookeeper

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

 

***start kafka

.\bin\windows\kafka-server-start.bat .\config\server.properties

 

***create topic of kafka

.\bin\windows\kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic java_in_use_topic

 

***list topic

.\bin\windows\kafka-topics.bat –list –zookeeper localhost:2181

 

*** send message to topic custom_topic

.\bin\windows\kafka-console-producer.bat –broker-list localhost:9092 –topic java_in_use_topic

sasl: .\bin\windows\kafka-console-producer.bat -broker-list localhost:9092 -topic java_in_use_topic --producer.config  c:\kafka\config\producer.properties

 

*** consume message from kafka

.\bin\windows\kafka-console-consumer.bat –zookeeper localhost:2181 –topic java_in_use_topic –from-beginning

 

sasl: .\bin\windows\kafka-console-producer.bat -broker-list localhost:9092 -topic java_in_use_topic --producer.config  c:\kafka\config\producer.properties

 

 

5. connect kafka from java

 

 

bootstrapServers = localhost:9092 (default port)

 

PRODUCER:

 

    Map<String, Object> props = new HashMap<>();

    // list of host:port pairs used for establishing the initial connections to the Kakfa cluster

    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);

    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

 

    String jaasTemplate = "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";";

    props.put("sasl.jaas.config", String.format(jaasTemplate, "kafka","kafkasecret"));

    

    props.put("security.protocol","SASL_PLAINTEXT");

    props.put("sasl.mechanism", "PLAIN");

 

 

CONSUMER:

 

    Map<String, Object> props = new HashMap<>();

    // list of host:port pairs used for establishing the initial connections to the Kafka cluster

    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);

    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);

    // allows a pool of processes to divide the work of consuming and processing records

    props.put(ConsumerConfig.GROUP_ID_CONFIG, "helloworld");

    // automatically reset the offset to the earliest offset

    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

 

String jaasTemplate = "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";";

    props.put("sasl.jaas.config", String.format(jaasTemplate, "kafka","kafkasecret"));

    

    props.put("security.protocol","SASL_PLAINTEXT");

    props.put("sasl.mechanism", "PLAIN");

 

 

REF:

https://www.cnblogs.com/zongzong/p/6566656.html

https://www.youtube.com/watch?v=twvdT6A1eeE&list=PLkz1SCf5iB4enAR00Z46JwY9GGkaS2NON&index=7

 

https://kafka.apache.org/quickstart