k8s 部署 canal 集群,RocketMQ 模式
k8s 部署 canal 集群,RocketMQ 模式
前提
MySQL
开启 binlog 的 MySQL 服务器:192.168.59.1:3306
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复
具有 slave 权限的账号:用户名 canal,密码:canal
CREATE USER canal IDENTIFIED BY 'canal';
-- GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
RocketMQ
RocketMQ 地址:192.168.56.1:9876
创建 topic:canal_test
RocketMQ-Dashboard 地址:192.168.56.1:8080,便于查看消息
制作 canal-admin、canal-server 镜像
由于 canal:v1.1.7 有一个 bug com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded isn’t start , please check · Issue #4930 ,而官方已经在 Release 1.1.8-alpha 中修复了这个问题,但没有发布官方的 docker 镜像,所以需要自己制作镜像。以下是我制作的镜像,已发布到 dockerhub,可直接使用
zhaobingshuang/canal-admin:v1.1.8-alpha
zhaobingshuang/canal-server:v1.1.8-alpha
部署 zookeeper
zk-configmap.yaml
数据目录配置为 /data
apiVersion: v1
kind: ConfigMap
metadata:
name: zk-configmap
data:
"zoo.cfg": |
tickTime=2000
dataDir=/data
clientPort=2181
zk-pvc.yaml
数据目录持久化
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: zk-data-pvc
spec:
resources:
requests:
storage: 2Gi
accessModes:
- ReadWriteOnce
zk-deployment.yaml
无状态服务
将 zk-data-pvc 挂载到数据目录 /data,将 zk-configmap 的 zoo.cfg 挂载到 /conf/zoo.cfg 文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: zk
spec:
replicas: 1
selector:
matchLabels:
app: zk
template:
metadata:
labels:
app: zk
spec:
containers:
- name: zk
image: zookeeper
ports:
- containerPort: 2181
name: serve
volumeMounts:
- mountPath: /data
name: zk-persistent-storage
- mountPath: /conf
name: zk-config
readOnly: true
volumes:
- name: zk-persistent-storage
persistentVolumeClaim:
claimName: zk-data-pvc
- name: zk-config
configMap:
name: zk-configmap
items:
- key: zoo.cfg
path: zoo.cfg
zk-service.yaml
apiVersion: v1
kind: Service
metadata:
name: zk
spec:
selector:
app: zk
ports:
- protocol: TCP
port: 2181
targetPort: 2181
type: ClusterIP
clusterIP: None
执行以下命令
kubectl apply -f zk-configmap.yaml
kubectl apply -f zk-pvc.yaml
kubectl apply -f zk-deployment.yaml
kubectl apply -f zk-service.yaml
打开 kubernetes-dashboard,进入 zk pod 验证。
执行 create /cluster_1
创建 znode,为之后 canal-admin 创建集群使用。
部署 canal-admin
执行 manager.sql 初始化数据库。
canal-admin-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: canal-admin-configmap
data:
"server.port": "8089"
"spring.datasource.address": "192.168.59.1:3306"
"spring.datasource.database": "canal_manager"
"spring.datasource.username": "canal"
"spring.datasource.password": "canal"
"canal.adminPasswd": "admin"
canal-admin-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: canal-admin
spec:
selector:
matchLabels:
app: canal-admin
template:
metadata:
labels:
app: canal-admin
spec:
containers:
- name: canal-admin
image: zhaobingshuang/canal-admin:v1.1.8-alpha
ports:
- containerPort: 8089
envFrom:
- configMapRef:
name: canal-admin-configmap
canal-admin-service.yaml
apiVersion: v1
kind: Service
metadata:
name: canal-admin
spec:
selector:
app: canal-admin
ports:
- protocol: TCP
port: 8089
targetPort: 8089
type: ClusterIP
canal-admin-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: canal-admin
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: canal-admin.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: canal-admin
port:
number: 8089
执行以下命令
kubectl apply -f canal-admin-configmap.yaml
kubectl apply -f canal-admin-deployment.yaml
kubectl apply -f canal-admin-service.yaml
kubectl apply -f canal-admin-ingress.yaml
zhaobingshuang/canal-admin:v1.1.8-alpha 镜像比较大,需要等待一段时间等待,等 pod canal-admin 启动后,浏览器访问 http://canal-admin.com/。
用户名:admin,密码:123456
创建集群,集群名称:cluster_1,ZK 地址:zk:2181/cluster_1
点击主配置 -> 载入模板
修改以下配置:
canal.zkServers =zk:2181/cluster_1
canal.serverMode = rocketMQ
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
rocketmq.namesrv.addr = 192.168.56.1:9876
rocketmq.tag =canal
部署 canal-server
canal-server-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: canal-server-configmap
data:
canal.admin.manager: "canal-admin:8089"
canal.admin.port: "11110"
canal.admin.user: "admin"
canal.admin.passwd: "4ACFE3202A5FF5CF467898FC58AAB1D615029441"
canal.admin.register.auto: "true"
canal.admin.register.cluster: "cluster_1"
canal-server-service.yaml
apiVersion: v1
kind: Service
metadata:
name: canal-server-headless
spec:
selector:
app: canal-server
ports:
- port: 11110
name: admin
- port: 11111
name: tcp
- port: 11112
name: metric
type: ClusterIP
clusterIP: None
canal-server-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: canal-server
spec:
replicas: 1
selector:
matchLabels:
app: canal-server
serviceName: canal-server-headless
template:
metadata:
labels:
app: canal-server
spec:
containers:
- name: canal-server
image: zhaobingshuang/canal-server:v1.1.8-alpha
envFrom:
- configMapRef:
name: canal-server-configmap
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: "metadata.namespace"
- name: POD_HOST
valueFrom:
fieldRef:
fieldPath: "metadata.name"
- name: RAW_NAME
valueFrom:
fieldRef:
fieldPath: "metadata.labels['app']"
- name: canal.register.ip
value: $(POD_HOST).$(RAW_NAME)-headless.$(POD_NAMESPACE)
- name: canal.admin.register.name
value: $(POD_HOST)
ports:
- containerPort: 11110
name: admin
- containerPort: 11111
name: tcp
- containerPort: 11112
name: metric
执行以下命令:
kubectl apply -f canal-server-configmap.yaml
kubectl apply -f canal-server-service.yaml
kubectl apply -f canal-server-statefulset.yaml
等 pod canal-server启动后,查看 Server 管理
测试
载入模板,修改以下配置之后保存:
canal.instance.master.address=192.168.59.1:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.filter.regex=test\\..*
canal.mq.topic=canal_test
启动实例
执行以下 SQL,创建数据库、创建表、插入数据:
CREATE DATABASE IF NOT EXISTS test;
USE test;
CREATE TABLE `table_1` (
`id` int NOT NULL,
`name` varchar(50) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
INSERT INTO table_1 VALUES(1, 'tom');
INSERT INTO table_1 VALUES(2, 'jack')
浏览器访问 http://192.168.56.1:8080/ 打开 RocketMQ-Dashboard 查看 canal_test 的消息: