Pulsar 集群搭建以及如何使用 debezium mysql connector

序言

hostname	ip	zookeeper	bookie	broker
node1	192.168.2.170	√	√	√
node2	192.168.2.171	√	√	√
node3	192.168.2.172	√	√	√

前提要求都装有 jdk1.8 及以上

一、部署 pulsar

zookeeper 最好使用独立的，使用pulsar自带的zookeeper可能会出问题

zookeeper 3.6.3 下载地址
pulsar 2.8.0 下载地址

分别解压这两个安装包

tar -zxvf 安装包

1、配置 zookeeper

所有节点都需要部署

vim conf/zoo.cfg 

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/software/zookeeper/stored/zookeeper-3.6.3
clientPort=2181
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

mkdir -pv /usr/software/zookeeper/stored/zookeeper-3.6.3

> node1
echo 1 > /usr/software/zookeeper/stored/zookeeper-3.6.3/myid

> node2
echo 2 > /usr/software/zookeeper/stored/zookeeper-3.6.3/myid

> node3
echo 3 > /usr/software/zookeeper/stored/zookeeper-3.6.3/myid

2、配置 bookkeeper

同样是需要部署到所有节点上，bookkeeper 与 broker 都是在 pulsar 中配置

cd ${PULSAR_HOME}

vim conf/bookkeeper.conf

journalDirectory=data/bookkeeper/journal
ledgerDirectories=data/bookkeeper/ledgers
prometheusStatsHttpPort=8100   
zkServers=node1:2181,node2:2181,node3:2181

# 保存、退出
mkdir -pv data/bookkeeper/journal
mkdir -pv data/bookkeeper/ledgers

Tips: prometheusStatsHttpPort 默认为 8000，但在 bookkeeper.conf 中，httpServerPort 默认也是 8000，会导致端口被占用。

3、配置 broker

与 bookkeeper 一样

vim conf/broker.conf

zookeeperServers=node1:2181,node2:2181,node3:2181
configurationStoreServers=node1:2181,node2:2181,node3:2181
webServicePort=8081	# 默认为 8080 ，可以选择不做更改
clusterName=pulsar-cluster   # 与后面初始化指定的 cluster 需一致

如果更改了 broker 的默认端口（webServicePort=8080），那么需要配置下面内容

vim conf/client.conf

webServiceUrl=http://node1:8081,node2:8081,node3:8081/
brokerServiceUrl=pulsar://node1:6650,node2:6650,node3:6650/

如果要使用 pulsra io 中 cdc connector，还需要进行以下配置

vim conf/broker.conf

functionsWorkerEnabled=true # 设置为 true

# 保存 退出

# 配置 Run Functions-worker with brokers 模式

vim conf/functions_worker.yml

numFunctionPackageReplicas: 3 # 需大于 2，必须修改
# 下面三项可改可不改，会从 broker.conf 中继承
pulsarFunctionsCluster: pulsar-cluster
pulsarServiceUrl: pulsar://node1:6650,node2:6650,node3:6650
pulsarWebServiceUrl: http://node1:8081,node2:8081,node3:8081

# 保存 退出即可，修改该文件需重启 broker

二、初始化集群

Tips:

1. 初始化需先启动 zookeeper 集群，初始化集群时会将元数据存在 zookeeper 中
2. 初始化 pulsar 集群以及 bookkeeper，在任一节点上执行一次即可
3. cluster 与 conf/broker.conf 中的 clusterName 一致
4. web-service-url 端口为 conf/broker.conf 中的 webServicePort

# 初始化 pulsar 集群

bin/pulsar initialize-cluster-metadata \
    --cluster pulsar-cluster \
    --zookeeper node1:2181 \
    --configuration-store node1:2181 \
    --web-service-url http://node1:8081,node2:8081,node3:8081 \
    --web-service-url-tls https://node1:8443,node2:8443,node3:8443 \
    --broker-service-url pulsar://node1:6650,node2:6650,node3:6650 \
    --broker-service-url-tls pulsar+ssl://node1:6651,node2:6651,node3:6651

# 初始化 bookkeeper
bin/bookkeeper shell metaformat

# 出现选项，选 y

三、启动集群

三个节点分别按顺序启动 zookeeper、bookie、broker

# 启动 zookeeper

cd ${ZOOKEEPER_HOME}
bin/zkServer.sh start

# 启动 bookie

cd ${PULSAR_HOME}
bin/pulsar-daemon start bookie

# 检查 bookie 状态，出现 Bookie sanity test succeeded 表明启动成功
bin/bookkeeper shell bookiesanity

# 启动 broker

cd ${PULSAR_HOME}
bin/pulsar-daemon start broker

# 检查 broker 状态，出现一下情况说明正常
bin/pulsar-admin brokers list pulsar-cluster
"node1:8081"
"node3:8081"
"node2:8081"

四、使用 Source

Tips:

1. 要使用 pulsar-io-debezium-mysql，需先开启 mysql binlog

# 开启mysql binlog
vim /etc/my.cnf

[mysqld]
log-bin=/var/lib/mysql/mysql-bin
binlog-format=ROW
server_id=174
# 保存 退出

# 创建 source
bin/pulsar-admin source create \
--archive connectors/pulsar-io-debezium-mysql-2.8.0.nar \
--classname org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource \
--name debezium-mysql-source \
--destination-topic-name debezium-mysql-topic \
--tenant public \
--namespace default \
--parallelism 1 \
--source-config-file sourceConfig/monitor-mysql.yaml

# monitor-mysql.yaml
configs:
  database.hostname: "192.168.2.174" # 要获取的数据库的地址
  database.port: "3306"
  database.serverTimezone: "GMT+8" # 设置时区
  database.user: "root"  # 数据库账号
  database.password: "root123" # 数据库访问密码
  database.server.id: "174"  # 对应数据库的 service_id
  database.server.name: "dbserver1" # 任意内容，对应查询的 topic
  database.whitelist: "test"  # 需要访问的数据库
  table.whitelist: "test.t_user" # 需要访问的数据表
  
  database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory"
  database.history.pulsar.topic: "my-history-topic"
  database.history.pulsar.service.url: "pulsar://node1:6650,node2:6650,node3:6650"

  key.converter: "org.apache.kafka.connect.json.JsonConverter"
  value.converter: "org.apache.kafka.connect.json.JsonConverter"

  pulsar.service.url: "pulsar://node1:6650,node2:6650,node3:6650"

  offset.storage.topic: "offset-topic"

# 查看 source 状态，看到 running : true 说明创建并启动成功
bin/pulsar-admin source status --tenant public --namespace default --name debezium-mysql-source
{
  "numInstances" : 1,
  "numRunning" : 1,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : true,
      "error" : "",
      "numRestarts" : 0,
      "numReceivedFromSource" : 10,
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [],
      "numSourceExceptions" : 0,
      "latestSourceExceptions" : [],
      "numWritten" : 10,
      "lastReceivedTime" : 1625476594950,
      "workerId" : "c-pulsar-cluster-fw-node1-8081"
    }
  } ]
}
# 停止 source 命令
bin/pulsar-admin source stop --tenant public --namespace default --name debezium-mysql-source --instance-id 0

# 启动 source 命令
bin/pulsar-admin source start --tenant public --namespace default --name debezium-mysql-source --instance-id 0

# 查看 pulsar topic
bin/pulsar-admin topics list public/default
"persistent://public/default/my-history-topic"
"persistent://public/default/offset-topic"
"persistent://public/default/dbserver1"
"persistent://public/default/dbserver1.test.t_user"
"persistent://public/default/debezium-mysql-topic"

至此，pulsar 集群以及 pulsar io 的环境搭建完成。

如有问题，欢迎一起交流讨论。