apache-pulsar集群搭建，以及demo

最新推荐文章于 2024-09-08 20:44:44 发布

露落梨花

最新推荐文章于 2024-09-08 20:44:44 发布

阅读量2.2k

点赞数 1

分类专栏： MySQL-System 文章标签： pulsar pulsar集群搭建 canal和pulsar

本文链接：https://blog.csdn.net/weixin_40126236/article/details/102604309

版权

MySQL-System 专栏收录该内容

19 篇文章 0 订阅

订阅专栏

4、下载安装包

一、介绍

1、apache-pulsar介绍

参考：https://blog.csdn.net/u010869257/article/details/83211152

2、pulsar的优缺点，以及适用场景

3、服务器环境

JDK：1.8+
zookeeper：hadoop001:2181，hadoop002:2181，hadoop003:2181
Linux（centos7+）：hadoop001，hadoop002，hadoop003

4、下载安装包

apache-pulsar-2.4.1-bin.tar.gz 点击下载
连接器下载：http://pulsar.apache.org/en/download/ （需要就下载）

#解压(三台机器)
tar zxvf apache-pulsar-2.4.1-bin.tar.gz

#得到apache-pulsar-2.4.1文件，所有操作都在该文件下操作

二、集群安装

单机版解压即用即可，没有什么配置的东西

1、初始化元数据到zookeeper

在任意一个机器上操作就可以，只执行一次（zookeeper会自动同步的）。如果执行失败可能是端口占用问题，需要修改被占用的端口（查看所有端口：ps aux）

# 初始化元数据到zk
./bin/pulsar initialize-cluster-metadata \
  --cluster pulsar-cluster \
  --zookeeper hadoop001:2181 \
  --configuration-store hadoop001:2181 \
  --web-service-url http://hadoop001:8080 \
  --web-service-url-tls https://hadoop001:8443 \
  --broker-service-url pulsar://hadoop001:6650 \
  --broker-service-url-tls pulsar+ssl://hadoop001:6651

2、部署bookKeeper集群

1）配置文件conf/bookkeeper.conf

#三台机器
vim conf/bookkeeper.conf
  advertisedAddress=hadoop001   # 配置本机ip
  zkServers=hadoop001:2181,hadoop002:2181,hadoop003:2181

2）启动bookKeeper

#三台机器上都启动 
./bin/pulsar-daemon start bookie

#  检验 ：
./bin/bookkeeper shell bookiesanity

3、部署brokers集群

1）配置conf/broker.conf

#三台机器都配置
vim conf/broker.conf
    zookeeperServers=hadoop001:2181,hadoop002:2181,hadoop003:2181
    configurationStoreServers=hadoop001:2181,hadoop002:2181,hadoop003:2181
    advertisedAddress=hadoop001  # 服务器本机ip
    clusterName=pulsar-cluster   #集群名字

2）启动brokers集群（三台机器）

开启：./bin/pulsar-daemon start broker
关闭：./bin/pulsar-daemon stop broker
查看：./bin/pulsar-admin brokers list pulsar-cluster（如果端口不一样修改：conf/client.conf）

三、IDEA开发demo

1、消费者

<dependency>
     <groupId>org.apache.pulsar</groupId>
     <artifactId>pulsar-client</artifactId>
     <version>2.4.1</version>
</dependency>


 
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.SubscriptionType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.concurrent.TimeUnit;

/**
 * @ClassName: MyConsumer
 * @Description: TODO
 * @Author: zikuang
 * @Data: 2019/10/17 14:32
 * @Version: 1.0
 **/
 
public class MyConsumer {
    private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class);
    //hadoop001 ,hadoop002,hadoop003
    private static final String SERVER_URL = "pulsar://hadoop001:6650";

    public static void main(String[] args) throws Exception {
        // 构造Pulsar Client
        PulsarClient client = PulsarClient.builder()
                .serviceUrl(SERVER_URL)
                .enableTcpNoDelay(true)
                .build();
        Consumer consumer = client.newConsumer()
                .consumerName("my-consumer")
                .topic("my-topic")
                .subscriptionName("my-subscription")
                .ackTimeout(10, TimeUnit.SECONDS)
                .maxTotalReceiverQueueSizeAcrossPartitions(10)
                .subscriptionType(SubscriptionType.Exclusive)
                .subscribe();
        while (true) {
            // Wait for a message
            Message msg = consumer.receive();

            try {
                // Do something with the message
                System.out.printf("Message received: %s\n", new String(msg.getData()));

                // Acknowledge the message so that it can be deleted by the message broker
                consumer.acknowledge(msg);
            } catch (Exception e) {
                // Message failed to process, redeliver later
                consumer.negativeAcknowledge(msg);
            }
        }
    }
}

2、生产者

 

import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;

/**
 * @ClassName: ProducerDemo
 * @Description: TODO
 * @Author: zikuang
 * @Data: 2019/10/17 14:29
 * @Version: 1.0
 **/
public class ProducerDemo {
    private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class);
    //hadoop001,hadoop002,hadoop003
    private static final String SERVER_URL = "pulsar://hadoop001:6650";
    public static void main(String[] args) throws Exception {
        // 构造Pulsar Client
        PulsarClient client = PulsarClient.builder()
                .serviceUrl(SERVER_URL)
                .enableTcpNoDelay(true)
                .build();
        // 构造生产者
        Producer<String> producer = client.newProducer(Schema.STRING)
                .producerName("my-producer")
                .topic("my-topic")
                .batchingMaxMessages(1024)
                .batchingMaxPublishDelay(10, TimeUnit.MILLISECONDS)
                .enableBatching(true)
                .blockIfQueueFull(true)
                .maxPendingMessages(512)
                .sendTimeout(10, TimeUnit.SECONDS)
                .blockIfQueueFull(true)
                .create();
        // 同步发送消息
        MessageId messageId = producer.send("Hello World");
        log.info("message id is {}",messageId);
        System.out.println(messageId.toString());
        // 异步发送消息
        CompletableFuture<MessageId> asyncMessageId = producer.sendAsync("This is a async message");
        // 阻塞线程，直到返回结果
        log.info("async message id is {}",asyncMessageId.get());

        
        producer.close();

        // 关闭licent的方式有两种，同步和异步
        // client.close();
        client.closeAsync();

    }
}