install apache kafka
1 - download zookeeper of version 3.4.10
http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/
2 - download kafka of version kafka_2.12-1.0.0
http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.12-1.0.0.tgz
3 - follow the instruction
http://programming-tips.in/kafka-set-up-apache-kafka-on-windows/
4 - add sasl for kafka:
4.1 add the following configuration to file C:\kafka\config\server.properties
listeners=SASL_PLAINTEXT://me:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
4.2 create a new file nanmed C:\kafka\config\kafka_server_jaas.conf and add the following configuration to it.
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="adminsecret"
user_admin="adminsecret"
user_kafka="kafkasecret";
};
4.3 create a new file named C:\kafka\config\kafka_client_jaas.conf and add the following configration to it.
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka"
password="kafkasecret";
};
4.4 add this configuration to file producer.properties and consumer.properties seprately(for command line use later)
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
4.5 add this command to file C:\kafka\bin\windows\kafka-server-start.bat
IF ["%KAFKA_OPTS%"] EQU [""] (
set KAFKA_OPTS=-Djava.security.auth.login.config=c:/kafka/config/kafka_server_jaas.conf
)
4.6 add the following line to file C:\kafka\bin\windows\kafka-console-producer.bat
set KAFKA_OPTS=-Djava.security.auth.login.config=c:/kafka/config/kafka_client_jaas.conf
4.7 add the following line to file C:\kafka\bin\windows\kafka-console-consumer.bat
set KAFKA_OPTS=-Djava.security.auth.login.config=c:/kafka/config/kafka_client_jaas.conf
The following command are all run in kafka directory:
***start zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
***start kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
***create topic of kafka
.\bin\windows\kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic java_in_use_topic
***list topic
.\bin\windows\kafka-topics.bat –list –zookeeper localhost:2181
*** send message to topic custom_topic
.\bin\windows\kafka-console-producer.bat –broker-list localhost:9092 –topic java_in_use_topic
sasl: .\bin\windows\kafka-console-producer.bat -broker-list localhost:9092 -topic java_in_use_topic --producer.config c:\kafka\config\producer.properties
*** consume message from kafka
.\bin\windows\kafka-console-consumer.bat –zookeeper localhost:2181 –topic java_in_use_topic –from-beginning
sasl: .\bin\windows\kafka-console-producer.bat -broker-list localhost:9092 -topic java_in_use_topic --producer.config c:\kafka\config\producer.properties
5. connect kafka from java
bootstrapServers = localhost:9092 (default port)
PRODUCER:
Map<String, Object> props = new HashMap<>();
// list of host:port pairs used for establishing the initial connections to the Kakfa cluster
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
String jaasTemplate = "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";";
props.put("sasl.jaas.config", String.format(jaasTemplate, "kafka","kafkasecret"));
props.put("security.protocol","SASL_PLAINTEXT");
props.put("sasl.mechanism", "PLAIN");
CONSUMER:
Map<String, Object> props = new HashMap<>();
// list of host:port pairs used for establishing the initial connections to the Kafka cluster
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
// allows a pool of processes to divide the work of consuming and processing records
props.put(ConsumerConfig.GROUP_ID_CONFIG, "helloworld");
// automatically reset the offset to the earliest offset
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
String jaasTemplate = "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";";
props.put("sasl.jaas.config", String.format(jaasTemplate, "kafka","kafkasecret"));
props.put("security.protocol","SASL_PLAINTEXT");
props.put("sasl.mechanism", "PLAIN");
REF:
https://www.cnblogs.com/zongzong/p/6566656.html
https://www.youtube.com/watch?v=twvdT6A1eeE&list=PLkz1SCf5iB4enAR00Z46JwY9GGkaS2NON&index=7
https://kafka.apache.org/quickstart