kafka初探

kafka初探

环境

必须有java环境

下载安装kafka

1.从kafka官网中下载最新版，http://kafka.apache.org,

wget https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz

tar -xzvf kafka_2.11-1.1.0.tgz 
cd kafka_2.11-1.1.0/

启动

kafka默认端口是9092，需要保证9092端口已经开启

1.启动自带的zookeeper：

bin/zookeeper-server-start.sh config/zookeeper.properties

2.启动kafka server:

bin/kafka-server-start.sh config/server.properties

测试

代码:

public class KafkaUtils {

    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaUtils.class);

    private static Producer<String, String> producer;
    private static Consumer<String, String> consumer;

    private KafkaUtils(){

    }

    /**
     * 生产者config
     */
    static {
        Properties prop = new Properties();
        prop.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaProperties.KAFKA_SERVER_URL+":"+ KafkaProperties.KAFKA_SERVER_PORT);
        prop.put(ProducerConfig.CLIENT_ID_CONFIG, KafkaProperties.CLIENT_ID);
        prop.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        prop.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        producer = new KafkaProducer<>(prop);
    }


    static {
        Properties prop = new Properties();
        prop.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaProperties.KAFKA_SERVER_URL+":"+ KafkaProperties.KAFKA_SERVER_PORT);
        prop.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        prop.put(ConsumerConfig.GROUP_ID_CONFIG, KafkaProperties.GROUP_ID);
        prop.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
        prop.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
//        prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
        prop.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        prop.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        consumer = new KafkaConsumer<>(prop);
    }

    /**
     * 发送消息
     * @param msg
     * @param async 是否异步
     */
    public static void sendMsg(String msg, boolean async){
        if (!async){
            try {
                producer.send(new ProducerRecord<String, String>(KafkaProperties.TOPIC,
                        String.valueOf(new Date().getTime()), msg)).get();
                LOGGER.info("发送消息成功,msg: {}", msg);
            } catch (InterruptedException e) {
                LOGGER.error("发送消息失败-{}", e);
            } catch (ExecutionException e) {
                LOGGER.error("发送消息失败-{}", e);
            }
        }else {
            producer.send(new ProducerRecord<String, String>(KafkaProperties.TOPIC,
                    String.valueOf(new Date().getTime()), msg));
        }
    }


    public static void getMessage(){
        consumer.subscribe(Collections.singletonList(KafkaProperties.TOPIC));
        while (true){
            ConsumerRecords<String, String> records = consumer.poll(1000);
            if (records.count() > 0){
                for (ConsumerRecord<String, String> record: records){
                    LOGGER.info("接收消息: key:{} - value: {} - offset: {}",record.key(),record.value(), record.offset());
                }
                break;
            }
        }
    }
}

测试用例:

public class KafkaUtilsTest {

    @Test
    public void sendMsgTest(){
        String msg = "hello, 你好吗?";
        KafkaUtils.sendMsg(msg, false);
    }

    @Test
    public void getMessageTest(){
        KafkaUtils.getMessage();
    }
}

1.运行sendMsgTest调试，控制台报错:

Connection to node 0 could not be established. Broker may not be available

解决方案:
在kafka解压目录中，打开config/server.properties文件，找到listeners,并取消注解,在PLAINTEXT后面添加broker的ip(kafka服务器的ip):

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1xx.xxx:9092

重启kafka服务器，再次运行sendMsgTest:

[main] INFO  com.kafka.util.KafkaUtils:  发送消息成功,msg: hello, 你好吗?

2.消费消息，运行getMessageTest,控制台好像一直在poll，并不能消费获取刚才发送的消息

解决方案

打开config/server.properties文件,找到advertised.listeners,并取消注释，在PLAINTEXT后面添加broker的ip(kafka服务器ip):

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1xx.xxx:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://192.168.1xx.xxx:9092

重启kafka服务器，再次运行getMessageTest,从控制台可以看到:

接收消息: key:1523765927237 - value: hello, 你好吗? - offset: 9