kafka初探
环境
必须有java环境
下载安装kafka
1.从kafka官网中下载最新版,http://kafka.apache.org,
wget https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz
tar -xzvf kafka_2.11-1.1.0.tgz
cd kafka_2.11-1.1.0/
启动
kafka默认端口是9092,需要保证9092端口已经开启
1.启动自带的zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
2.启动kafka server:
bin/kafka-server-start.sh config/server.properties
测试
代码:
public class KafkaUtils {
private static final Logger LOGGER = LoggerFactory.getLogger(KafkaUtils.class);
private static Producer<String, String> producer;
private static Consumer<String, String> consumer;
private KafkaUtils(){
}
/**
* 生产者config
*/
static {
Properties prop = new Properties();
prop.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaProperties.KAFKA_SERVER_URL+":"+ KafkaProperties.KAFKA_SERVER_PORT);
prop.put(ProducerConfig.CLIENT_ID_CONFIG, KafkaProperties.CLIENT_ID);
prop.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
prop.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
producer = new KafkaProducer<>(prop);
}
static {
Properties prop = new Properties();
prop.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaProperties.KAFKA_SERVER_URL+":"+ KafkaProperties.KAFKA_SERVER_PORT);
prop.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
prop.put(ConsumerConfig.GROUP_ID_CONFIG, KafkaProperties.GROUP_ID);
prop.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
prop.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
// prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
prop.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
prop.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
consumer = new KafkaConsumer<>(prop);
}
/**
* 发送消息
* @param msg
* @param async 是否异步
*/
public static void sendMsg(String msg, boolean async){
if (!async){
try {
producer.send(new ProducerRecord<String, String>(KafkaProperties.TOPIC,
String.valueOf(new Date().getTime()), msg)).get();
LOGGER.info("发送消息成功,msg: {}", msg);
} catch (InterruptedException e) {
LOGGER.error("发送消息失败-{}", e);
} catch (ExecutionException e) {
LOGGER.error("发送消息失败-{}", e);
}
}else {
producer.send(new ProducerRecord<String, String>(KafkaProperties.TOPIC,
String.valueOf(new Date().getTime()), msg));
}
}
public static void getMessage(){
consumer.subscribe(Collections.singletonList(KafkaProperties.TOPIC));
while (true){
ConsumerRecords<String, String> records = consumer.poll(1000);
if (records.count() > 0){
for (ConsumerRecord<String, String> record: records){
LOGGER.info("接收消息: key:{} - value: {} - offset: {}",record.key(),record.value(), record.offset());
}
break;
}
}
}
}
测试用例:
public class KafkaUtilsTest {
@Test
public void sendMsgTest(){
String msg = "hello, 你好吗?";
KafkaUtils.sendMsg(msg, false);
}
@Test
public void getMessageTest(){
KafkaUtils.getMessage();
}
}
1.运行sendMsgTest
调试,控制台报错:
Connection to node 0 could not be established. Broker may not be available
解决方案:
在kafka解压目录中,打开config/server.properties文件,找到listeners,并取消注解,在PLAINTEXT
后面添加broker的ip(kafka服务器的ip):
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1xx.xxx:9092
重启kafka服务器,再次运行sendMsgTest
:
[main] INFO com.kafka.util.KafkaUtils: 发送消息成功,msg: hello, 你好吗?
2.消费消息,运行getMessageTest
,控制台好像一直在poll,并不能消费获取刚才发送的消息
解决方案
打开config/server.properties文件,找到advertised.listeners
,并取消注释,在PLAINTEXT
后面添加broker的ip(kafka服务器ip):
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1xx.xxx:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://192.168.1xx.xxx:9092
重启kafka服务器,再次运行getMessageTest
,从控制台可以看到:
接收消息: key:1523765927237 - value: hello, 你好吗? - offset: 9