kafka笔记一
(本人是kafka小白,想把自己的笔记记录下来,慢慢学习,逐渐成长)
一、首先安装kafka集群,最近最新版本是1.0,下载kafka1.0安装包,解压缩,修改配置文件,安装,测试,详见另一篇博客笔记:kafka集群安装部署
二、完成kafka消息订阅三剑客:producer生产者、broker中间件、consumer消费者
三、利用kafkaAPI编写生产消费demo
(1)建立maven项目,pom文件中添加kafka依赖包
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.0.0</version>
</dependency>
(2)写生产者程序
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.Producer;
import java.util.Properties;
public class kafka_producer {
public static void main(String[] args){
//配置信息
Properties props = new Properties();
props.put("bootstrap.servers", "写上自己的集群IP:port")
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//实例化生产者对象
Producer<String, String> producer = new KafkaProducer<String,String>(props);
for(int i=0;i<=50;i++){
String s = "send message : "+i;
producer.send(new ProducerRecord<String,String>("test1",Integer.toString(i), s));
}
producer.close();
}
}
(3)现在在后端切到kafka安装目录的bin目录下,执行kafka的消费者命令
./kafka-console-consumer.sh --bootstrap-server 你的ip:port --topic 你的topic --from-beginning
执行生产者程序,观察到后端kafka服务中consumer在消费消息。
注:保证topic一致,保证kafka集群正常启动。
(4)消费者demo
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.util.Arrays;
import java.util.Properties;
public class kafka_consumer {
public static void main(String[] args){
Properties props = new Properties();
props.put("bootstrap.servers", "your ip:port")
props.put("group.id", "my_group");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList("test1"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
}
}
依据kafkaAPI, 简单的实现了生产消费进程,后续会继续学习,共勉!