OpenShift 4 - 使用Debezium实现MySQL的CDC变化数据捕获

《OpenShift 4.x HOL教程汇总》
说明：本文已经在OpenShift 4.6环境中验证

部署应用和MySQL数据库

部署MySQL

1. 创建项目

$ oc new-project debezium-demo

2. 创建内容如下的debezium.cnf的文件

!include /etc/my.cnf
 
[mysqld]
server-id = 223344
server_id = 223344
log_bin = ON
binlog_format = ROW
binlog_row_image = full
binlog_rows_query_log_events = ON
expire_logs_days = 10
gtid_mode = ON
enforce_gtid_consistency = ON

3. 执行命令，创建configmap

$ oc create configmap db-config --from-file=debezium.cnf

4. 创建内容如下的initdb.sql文件

CREATE USER IF NOT EXISTS 'debezium'@'%' IDENTIFIED WITH mysql_native_password BY 'debezium';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium'@'%';
FLUSH PRIVILEGES;

5. 执行命令，创建configmap

$ oc create configmap db-init --from-file=initdb.sql

6. 执行命令，创建configmap

$ oc create secret generic db-creds --from-literal=database-name=music --from-literal=database-password=music --from-literal=database-user=music --from-literal=database-admin-password=music

7. 创建内容如下的mysql.yml文件

kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
  name: spring-music-db
  labels:
    application: spring-music
    app: spring-music
    app.kubernetes.io/part-of: spring-music
    app.openshift.io/runtime: mysql-database
spec:
  replicas: 1
  strategy:
    type: Recreate
    recreateParams:
      post:
        failurePolicy: Abort
        execNewPod:
          command:
            - /bin/sh
            - '-c'
            - sleep 10 && MYSQL_PWD="$MYSQL_ROOT_PASSWORD" $MYSQL_PREFIX/bin/mysql -h $SPRING_MUSIC_DB_SERVICE_HOST -u root < /config/initdb.d/initdb.sql
          containerName: spring-music-db
          volumes:
            - db-init
  selector:
    name: spring-music-db
  template:
    metadata:
      name: spring-music-db
      labels:
        name: spring-music-db
    spec:
      volumes:
        - name: db-data
          emptyDir: {}
        - name: db-init
          configMap:
            name: db-init
        - name: db-config
          configMap:
            name: db-config
      containers:
        - env:
            - name: MYSQL_DEFAULTS_FILE
              value: /config/configdb.d/my-debezium.cnf
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-user
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-password
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-name
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-creds
                  key: database-admin-password
          name: spring-music-db
          image: ' '
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: db-data
              mountPath: /var/lib/mysql/data
            - name: db-init
              mountPath: /config/initdb.d
            - name: db-config
              mountPath: /config/configdb.d
          ports:
            - containerPort: 3306
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3306
            timeoutSeconds: 1
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -i
                - -c
                - MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'
            failureThreshold: 3
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              memory: 512Mi
          securityContext:
            privileged: false
  triggers:
    - type: ConfigChange
    - type: ImageChange
      imageChangeParams:
        automatic: true
        containerNames:
          - spring-music-db
        from:
          kind: ImageStreamTag
          name: mysql:8.0
          namespace: openshift
---
kind: Service
apiVersion: v1
metadata:
  name: spring-music-db
  labels:
    application: spring-music
    app: spring-music
  annotations:
    template.openshift.io/expose-uri: mysql://{.spec.clusterIP}:{.spec.ports[?(.name=="mysql")].port}
spec:
  ports:
    - name: mysql
      port: 3306
      protocol: TCP
      targetPort: 3306
  selector:
    name: spring-music-db

8. 执行命令，创建应用。

$ oc create -f mysql.yml

9. 切换到Developer视图查看Topology，已经部署好MqSQL数据库。
   

部署应用

1. 在Developer视图的“+Add”中进入“container Image”.。
   
2. 在“Image”区域中填入“quay.io/edeandrea/spring-music:latest”，然后选择“Deployment Config”。
   
3. 添加“app.openshift.io/runtime=spring”标签，然后按照下图为Deployment添加3个“Environment Variables”。
   Config Map or Secret
   NAME: SPRING_DATASOURCE_USERNAME
   VALUE: db-creds 的 database-user
   Config Map or Secret
   NAME: SPRING_DATASOURCE_PASSWORD
   VALUE: Secret db-creds 的database-password
   Value
   NAME: SPRING_DATASOURCE_URL
   VALUE: jdbc:mysql://spring-music-db/music
   
4. 在Topology查看应用部署状态。
   
5. 可以访问“spring-music”应用的route访问应用。
   

部署AMQ Streams（Strimzi）和Dezebium

部署 AMQ Streams（Strimzi）Operator

AMQ Streams Operator是RedHat官方提供的，Strimzi Operator是社区提供的，使用两者任何一个都可以。

1. 使用缺省配置安装AMQ Streams Operator或Strimzi Operator。
   
   

部署Kafka

2. 在Administrator视图安装好的AMQ Streams Operator中找到Kafka。
   
3. 在“Create Kafka”的界面中，在YAML视图中填入以下内容，然后创建Kafka。

kind: Kafka
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  name: db-events
  namespace: debezium-demo
  labels:
    app: spring-music-cdc
    template: spring-music-cdc
    app.kubernetes.io/part-of: spring-music-cdc
spec:
  kafka:
    replicas: 3
    listeners:
      plain: {}
    jvmOptions:
      gcLoggingEnabled: false
    config:
      auto.create.topics.enable: "true"
      num.partitions: 1
      offsets.topic.replication.factor: 3
      default.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: true
    template:
      statefulset:
        metadata:
          labels:
            app.kubernetes.io/part-of: spring-music-cdc
            app: spring-music-cdc
            template: spring-music-cdc
          annotations:
            app.openshift.io/connects-to: db-events-zookeeper
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: true
    template:
      statefulset:
        metadata:
          labels:
            app.kubernetes.io/part-of: spring-music-cdc
            app: spring-music-cdc
            template: spring-music-cdc
  entityOperator:
    topicOperator: {}
    userOperator: {}

4. 在Topology中查看部署好的Kafka。
   

通过MySQL Connector获取变化数据，并发到AMQ Streams的Kafka

生成Debezium的MySQL Connector容器镜像

可以直接使用“quay.io/edeandrea/kafka-connect-debezium-mysql:amq-streams-1.4.0-dbz-1.1.0.Final”镜像，也可以根据以下步骤生成该容器镜像。

1. 访问“https://debezium.io/releases/”，通过下图下载Debezium的MySQL Connector Plugin。
   
2. 创建“debezium-connect-image”，然后在下面创建“plugins”目录，再将上一步下载的文件解压缩的plugins目录中。
   
3. 在debezium-connect-image目录中创建如下内容的Dockerfile 文件

FROM registry.redhat.io/amq7/amq-streams-kafka-24-rhel7:1.4.0
USER root:root
COPY ./plugins/ /opt/kafka/plugins
USER jboss:jboss

4. 然后根据Dockerfile 生成容器镜像，然后传到quay.io。

配置Kafka Connect

1. 在Administrator视图的AMQ Streams Operator进入“Kafka Connect”
   
2. 在YAML视图中使用以下yaml创建资源。

kind: KafkaConnect
apiVersion: kafka.strimzi.io/v1beta1
metadata:
  name: db-events
  namespace: debezium-demo
  labels:
    app: spring-music-cdc
    template: spring-music-cdc
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  image: "quay.io/edeandrea/kafka-connect-debezium-mysql:amq-streams-1.4.0-dbz-1.1.0.Final"
  bootstrapServers: "db-events-kafka-bootstrap:9092"
  jvmOptions:
    gcLoggingEnabled: false
  config:
    group.id: spring-music-db
    offset.storage.topic: spring-music-db-offsets
    config.storage.topic: spring-music-db-configs
    status.storage.topic: spring-music-db-status
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
  externalConfiguration:
    volumes:
      - name: connector-config
        secret:
          secretName: db-connector-creds
  template:
    deployment:
      metadata:
        labels:
          app: spring-music-cdc
          app.kubernetes.io/part-of: spring-music-cdc
          template: spring-music-cdc
        annotations:
          app.openshift.io/connects-to: db-events-kafka,spring-music-db

3. 在Developer视图的Topology查看Kafka Connect部署情况。
   
4. 在Administrator视图的AMQ Streams Operator进入“Kafka Connector”
   
5. 在YAML视图中使用以下yaml创建资源。

kind: KafkaConnector
apiVersion: kafka.strimzi.io/v1alpha1
metadata:
  name: db-events
  namespace: debezium-demo
  labels:
    app: spring-music-cdc
    strimzi.io/cluster: db-events
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    database.hostname: spring-music-db
    database.port: 3306
    database.user: "${file:/opt/kafka/external-configuration/connector-config/connector.properties:dbUsername}"
    database.password: "${file:/opt/kafka/external-configuration/connector-config/connector.properties:dbPassword}"
    database.dbname: music
    database.server.name: spring-music
    database.server.id: 223344
    database.whitelist: music
    database.allowPublicKeyRetrieval: true
    database.history.kafka.bootstrap.servers: db-events-kafka-bootstrap:9092
    database.history.kafka.topic: dbhistory.music
    table.whitelist: music.outbox_events
    tombstones.on.delete : false
    transforms: outbox
    transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
    transforms.outbox.route.topic.replacement: "outbox.${routedByValue}.events"
    transforms.outbox.table.field.event.id: event_id
    transforms.outbox.table.field.event.key: aggregate_id
    transforms.outbox.table.field.event.timestamp: event_timestamp
    transforms.outbox.table.field.event.type: event_type
    transforms.outbox.table.field.event.payload.id: aggregate_id
    transforms.outbox.route.by.field: aggregate_type
    transforms.outbox.table.fields.additional.placement: "event_id:envelope:eventId,event_timestamp:envelope:eventTimestamp,aggregate_id:envelope:aggregateId,aggregate_type:envelope:aggregateType"

验证

1. 在Topology中选中“db-events-kafka”，然后在右侧进入3个pod的任意一个。
   
2. 在Terminal中输入以下命令，然后能从下图看到返回的所有捕获到的数据。

$ bin/kafka-console-consumer.sh --bootstrap-server db-events-kafka-bootstrap:9092 --topic outbox.Album.events --from-beginning

3. 在新的界面中打开部署的应用，点击“add an album”链接添加一个新的album数据，然后ok提交数据。

4. 界面中提示保存成功

5. 返回Terminal，确认出现一行新的数据。格式化后为以下内容，其中在“payload”部分是我们提交的新数据内容。

{
	"schema": {
		"type": "struct",
		"fields": [{
			"type": "string",
			"optional": true,
			"name": "io.debezium.data.Json",
			"version": 1,
			"field": "payload"
		}, {
			"type": "string",
			"optional": false,
			"field": "eventType"
		}, {
			"type": "int64",
			"optional": false,
			"field": "eventId"
		}, {
			"type": "int64",
			"optional": false,
			"name": "io.debezium.time.MicroTimestamp",
			"version": 1,
			"field": "eventTimestamp"
		}, {
			"type": "string",
			"optional": false,
			"field": "aggregateId"
		}, {
			"type": "string",
			"optional": false,
			"field": "aggregateType"
		}],
		"optional": false
	},
	"payload": {
		"payload": "{\"album\":{\"id\":\"f8f1ad92-f724-45bc-b0ca-e968c369a9c0\",\"genre\":\"Rock\",\"title\":\"Test\",\"artist\":\"Test\",\"albumId\":null,\"trackCount\":0,\"releaseYear\":\"2020\"},\"eventType\":\"ALBUM_CREATED\"}",
		"eventType": "ALBUM_CREATED",
		"eventId": 30,
		"eventTimestamp": 1617626600716385,
		"aggregateId": "f8f1ad92-f724-45bc-b0ca-e968c369a9c0",
		"aggregateType": "Album"
	}
}

参考

https://dzone.com/articles/change-data-capture-with-debezium-a-simple-how-to