Apache Pulsar的produce和consume的Java/Scala API

1. produce

1.1 同步produce

示例如下

import org.apache.pulsar.client.api.{Producer, PulsarClient, Schema}
import org.apache.pulsar.client.impl.schema.AvroSchema

case class User(name:String, age:Int)

object PulsarProducerSyncTest {

  def main(args: Array[String]): Unit = {

    // =======================创建pulsar的客户端对象=======================
    val serviceUrl = "pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650"
    val pulsarClient:PulsarClient = PulsarClient.builder().serviceUrl(serviceUrl).build()

    // =======================基于客户端对象进行构建生产者对象=====================
    val partitionedTopicName = "persistent://my-tenant/my-ns/my-partitioned-topic"
    // 也可以发送字符串，Schema的类型为：Schema.STRING
    val producer:Producer[User] = pulsarClient.newProducer(AvroSchema.of(classOf[User]))
      .topic(partitionedTopicName)
      .create()

    // ======================进行数据生产======================
    val user = User("张三", 18)
    producer.send(user)

    // =======================释放资源======================
    producer.close()
    pulsarClient.close()

  }

}

先运行consume客户端，再运行程序，consume的输出如下：

[root@pulsar1 ~]# apache-pulsar-2.9.1/bin/pulsar-client consume persistent://my-tenant/my-ns/my-partitioned-topic -n 1 -s "syncTest"
......省略部分......
2022-03-21T17:17:56,032+0800 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - [persistent://my-tenant/my-ns/my-partitioned-topic] [syncTest] Success subscribe new topic persistent://my-tenant/my-ns/my-partitioned-topic in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
2022-03-21T17:27:57,819+0800 [pulsar-client-io-1-1] INFO  com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized
----- got message -----
key:[null], properties:[], content:$ 
                                    张三
......省略部分......
2022-03-21T17:27:57,931+0800 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0x06fc3ba9, L:/192.168.23.111:50556 ! R:pulsar1/192.168.23.111:6650] Disconnected
2022-03-21T17:27:59,962+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully consumed
[root@pulsar1 ~]#

1.2 异步produce

示例如下

import org.apache.pulsar.client.api.{Producer, PulsarClient, Schema}

object PulsarProducerAsyncTest {

  def main(args: Array[String]): Unit = {

    // ==========================创建Pulsar的客户端对象============================
    val serviceUrl = "pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650"
    val pulsarClient:PulsarClient = PulsarClient.builder().serviceUrl(serviceUrl).build()

    // ===========================通过客户端构建生产者的对象========================
    val partitionedTopicName = "persistent://my-tenant/my-ns/my-partitioned-topic"
    val producer:Producer[String] = pulsarClient.newProducer(Schema.STRING)
      .topic(partitionedTopicName)
      .create()

    // ===========================进行数据发送操作================================
    // 异步发送，消息会先写入缓存中，等达到一定的数量/大小/时间后，再一起发送
    val response:java.util.concurrent.CompletableFuture[MessageId] = 
    producer.sendAsync("hello async pulsar")
    // 所以我们等待一定时间，等消息发送成功了，再关闭客户端
    Thread.sleep(1000)    // 1秒


    // ==========================释放资源=========================
    producer.close()
    pulsarClient.close()

  }

}

先运行consume客户端，再运行程序，consume的输出如下：

[root@pulsar1 ~]# apache-pulsar-2.9.1/bin/pulsar-client consume persistent://my-tenant/my-ns/my-partitioned-topic -n 1 -s "asyncTest"
......省略部分......
2022-03-21T18:02:20,267+0800 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - [persistent://my-tenant/my-ns/my-partitioned-topic] [asyncTest] Success subscribe new topic persistent://my-tenant/my-ns/my-partitioned-topic in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
2022-03-21T18:02:39,029+0800 [pulsar-client-io-1-1] INFO  com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized
----- got message -----
key:[null], properties:[], content:hello async pulsar
......省略部分......
2022-03-21T18:02:39,121+0800 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xd07c8578, L:/192.168.23.111:47222 ! R:pulsar3/192.168.23.113:6650] Disconnected
2022-03-21T18:02:41,139+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully consumed
[root@pulsar1 ~]#

2. consume

2.1 single consume

示例如下

import org.apache.pulsar.client.api.{Consumer, Message, PulsarClient, Schema, SubscriptionType}
import org.apache.pulsar.client.impl.schema.AvroSchema

case class User(name:String, age:Int)

object PulsarConsumerTest {

  def main(args: Array[String]): Unit = {

    // ===================创建pulsar的客户端的对象===========================
    val serviceUrl = "pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650"
    val pulsarClient:PulsarClient = PulsarClient.builder().serviceUrl(serviceUrl).build()

    // ==========================基于客户端构建消费者对象====================
    val partitionedTopicName = "persistent://my-tenant/my-ns/my-partitioned-topic"
    // 如果传递的是：AvroSchema.of(classOf[User])，会报错误：java.lang.NoSuchMethodException: User.<init>()
    val consumer:Consumer[String]  = pulsarClient.newConsumer(Schema.STRING)
      // 可以传入多个topic
      .topic(partitionedTopicName)
      .subscriptionName("consumeTest")
      .subscriptionType(SubscriptionType.Exclusive)
      .subscribe()

    // =================循环从消费者读取数据====================
    while(true) {
      // 接收消息
      val message:Message[String] = consumer.receive()

      try {
        // 获取消息
        val msg: String = message.getValue()

        // 处理数据
        println("获取的数据为: " + msg)

        // ack确认操作，下次重启从ack的position开始消费数据
        consumer.acknowledge(message)
      } catch {
        case e:Exception => {
          e.printStackTrace()
          consumer.negativeAcknowledge(message)
        }
      }
      
    }

  }

}

消费生产者发送过来的消息，结果如下：

获取的数据为: hello async pulsar

2.2 batch consume

示例如下

import org.apache.pulsar.client.api.{BatchReceivePolicy, Consumer, Messages, PulsarClient, Schema}

import java.util.concurrent.TimeUnit
import scala.collection.convert.ImplicitConversions.`iterable AsScalaIterable`

object PulsarConsumerBatchTest {

  def main(args: Array[String]): Unit = {

    // =============================构建Pulsar的客户端对象==================
    val serviceUrl = "pulsar://192.168.23.111:6650,192.168.23.112:6650,192.168.23.113:6650"
    val pulsarClient:PulsarClient = PulsarClient.builder().serviceUrl(serviceUrl).build()

    // =========================通过客户端构建消费者对象========================
    val partitionedTopicName = "persistent://my-tenant/my-ns/my-partitioned-topic"
    val consumer:Consumer[String] = pulsarClient.newConsumer(Schema.STRING)
      .topic(partitionedTopicName)
      .subscriptionName("batchConsumeTest")
      // 设置支持批量读取参数配置
      .batchReceivePolicy(
        BatchReceivePolicy.builder()
          .maxNumBytes(1024 * 1024)   // 1M
          .maxNumMessages(100)
          .timeout(2000, TimeUnit.MILLISECONDS)
          .build()
      )
      .subscribe()

    // ==========================循环读取数据=======================
    while (true) {

      // 读取消息
      val messages:Messages[String]  = consumer.batchReceive()

      // 获取消息
      messages.foreach(message => {

        try {
          val msg: String = message.getValue()
          println("获取的数据为: " + msg)

          // ack确认，下次重启从ack的position开始消费数据
          consumer.acknowledge(message)
        } catch {
          case e:Exception => {
            e.printStackTrace()
            consumer.negativeAcknowledge(message)
          }
        }

      })

    }
  }

}

消费生产者发送过来的消息，结果如下：

获取的数据为: hello async pulsar1
获取的数据为: hello async pulsar2
获取的数据为: hello async pulsar3