kafka+debezium 监听mysql日志

安腾斯科技

已于 2024-06-27 15:13:34 修改

阅读量288

点赞数 2

文章标签： mysql kafka

于 2024-06-27 15:12:18 首次发布

本文链接：https://blog.csdn.net/2301_81431930/article/details/140014235

版权

简介：使用kafka+Debezium 进行监听mysql的binlog日志来进行同步，然后发送到kafka，再用java进行消费kafka拿到消息，然后对总部进行操作

1、安装docker

使用docker运行相应组件

<span style="color:#313c52"><span style="background-color:#f5f5f5"><strong><span style="color:#880000"># step 1: 安装必要的一些系统工具</span><span style="color:#000000">
sudo yum install </span><span style="color:#666600">-</span><span style="color:#000000">y yum</span><span style="color:#666600">-</span><span style="color:#000000">utils device</span><span style="color:#666600">-</span><span style="color:#000000">mapper</span><span style="color:#666600">-</span><span style="color:#000000">persistent</span><span style="color:#666600">-</span><span style="color:#000000">data lvm2
</span><span style="color:#880000"># Step 2: 添加软件源信息</span><span style="color:#000000">
sudo yum</span><span style="color:#666600">-</span><span style="color:#000000">config</span><span style="color:#666600">-</span><span style="color:#000000">manager </span><span style="color:#666600">--</span><span style="color:#000000">add</span><span style="color:#666600">-</span><span style="color:#000000">repo http</span><span style="color:#666600">://</span><span style="color:#000000">mirrors</span><span style="color:#666600">.</span><span style="color:#000000">aliyun</span><span style="color:#666600">.</span><span style="color:#000000">com</span><span style="color:#666600">/</span><span style="color:#000000">docker</span><span style="color:#666600">-</span><span style="color:#000000">ce</span><span style="color:#666600">/</span><span style="color:#000000">linux</span><span style="color:#666600">/</span><span style="color:#000000">centos</span><span style="color:#666600">/</span><span style="color:#000000">docker</span><span style="color:#666600">-</span><span style="color:#000000">ce</span><span style="color:#666600">.</span><span style="color:#000000">repo
</span><span style="color:#880000"># Step 3: 更新并安装 Docker-CE</span><span style="color:#000000">
sudo yum makecache fast
sudo yum </span><span style="color:#666600">-</span><span style="color:#000000">y install docker</span><span style="color:#666600">-</span><span style="color:#000000">ce
</span><span style="color:#880000"># Step 4: 开启Docker服务</span><span style="color:#000000">
sudo service docker start</span></strong></span></span>

2、启动zookeeper

<span style="color:#313c52"><span style="background-color:#f5f5f5"><span style="color:#000000">docker run </span><span style="color:#666600">-</span><span style="color:#000000">d </span><span style="color:#666600">--</span><span style="color:#000000">name zookeeper </span><span style="color:#666600">-</span><span style="color:#000000">p </span><span style="color:#006666">2181</span><span style="color:#666600">:</span><span style="color:#006666">2181</span><span style="color:#000000"> zookeeper</span></span></span>

要启动kafka，需要有一个注册中心
3、启动kafka

<span style="color:#313c52"><span style="background-color:#f5f5f5"><span style="color:#000000">docker run </span><span style="color:#666600">-</span><span style="color:#000000">d </span><span style="color:#666600">--</span><span style="color:#000000">name kafka </span><span style="color:#666600">-</span><span style="color:#000000">p </span><span style="color:#006666">9092</span><span style="color:#666600">:</span><span style="color:#006666">9092</span><span style="color:#000000"> \
  </span><span style="color:#666600">--</span><span style="color:#000000">env KAFKA_ZOOKEEPER_CONNECT</span><span style="color:#666600">=</span><span style="color:#000000">zookeeper</span><span style="color:#666600">:</span><span style="color:#006666">2181</span><span style="color:#000000"> \
  </span><span style="color:#666600">--</span><span style="color:#000000">env KAFKA_ADVERTISED_LISTENERS</span><span style="color:#666600">=</span><span style="color:#000000">PLAINTEXT</span><span style="color:#666600">:</span><span style="color:#880000">//localhost:9092 \</span>
  <span style="color:#666600">--</span><span style="color:#000000">env KAFKA_LISTENER_SECURITY_PROTOCOL_MAP</span><span style="color:#666600">=</span><span style="color:#000000">PLAINTEXT</span><span style="color:#666600">:</span><span style="color:#000000">PLAINTEXT \
  </span><span style="color:#666600">--</span><span style="color:#000000">env KAFKA_INTER_BROKER_LISTENER_NAME</span><span style="color:#666600">=</span><span style="color:#000000">PLAINTEXT \
  </span><span style="color:#666600">--</span><span style="color:#000000">env KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR</span><span style="color:#666600">=</span><span style="color:#006666">1</span><span style="color:#000000"> \
  kafka</span></span></span>

拉去kafka，连接zookeeper

4、Debezium 配置

<span style="color:#313c52"><span style="background-color:#f5f5f5"><span style="color:#000000">docker run </span><span style="color:#666600">-</span><span style="color:#000000">d </span><span style="color:#666600">--</span><span style="color:#000000">name connect \
  </span><span style="color:#666600">-</span><span style="color:#000000">p </span><span style="color:#006666">8083</span><span style="color:#666600">:</span><span style="color:#006666">8083</span><span style="color:#000000"> \
  </span><span style="color:#666600">-</span><span style="color:#000000">e GROUP_ID</span><span style="color:#666600">=</span><span style="color:#006666">1</span><span style="color:#000000"> \
  </span><span style="color:#666600">-</span><span style="color:#000000">e CONFIG_STORAGE_TOPIC</span><span style="color:#666600">=</span><span style="color:#000000">my_connect_configs \
  </span><span style="color:#666600">-</span><span style="color:#000000">e OFFSET_STORAGE_TOPIC</span><span style="color:#666600">=</span><span style="color:#000000">my_connect_offsets \
  </span><span style="color:#666600">-</span><span style="color:#000000">e ADVERTISED_HOST_NAME</span><span style="color:#666600">=</span><span style="color:#000000">connect \
  </span><span style="color:#666600">--</span><span style="color:#000000">link zookeeper</span><span style="color:#666600">:</span><span style="color:#000000">zookeeper \
  </span><span style="color:#666600">--</span><span style="color:#000000">link kafka</span><span style="color:#666600">:</span><span style="color:#000000">kafka \
  debezium</span><span style="color:#666600">/</span><span style="color:#000000">connect</span></span></span>

拉取Debezium，里面包含 kafka connect来连接kafka并通信

5、写配置文件

<span style="color:#313c52"><span style="background-color:#f5f5f5"><span style="color:#666600">{</span>

  <span style="color:#008800">"name"</span><span style="color:#666600">:</span> <span style="color:#008800">"mysql-connector"</span><span style="color:#666600">,</span>

  <span style="color:#008800">"config"</span><span style="color:#666600">:</span> <span style="color:#666600">{</span>

    <span style="color:#008800">"connector.class"</span><span style="color:#666600">:</span> <span style="color:#008800">"io.debezium.connector.mysql.MySqlConnector"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.hostname"</span><span style="color:#666600">:</span> <span style="color:#008800">"host"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.port"</span><span style="color:#666600">:</span> <span style="color:#008800">"3306"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.user"</span><span style="color:#666600">:</span> <span style="color:#008800">"user"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.password"</span><span style="color:#666600">:</span> <span style="color:#008800">"password"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.server.id"</span><span style="color:#666600">:</span> <span style="color:#008800">"184054"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.server.name"</span><span style="color:#666600">:</span> <span style="color:#008800">"fullfillment"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.include.list"</span><span style="color:#666600">:</span> <span style="color:#008800">"inventory"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.history.kafka.bootstrap.servers"</span><span style="color:#666600">:</span> <span style="color:#008800">"localhost:9092"</span><span style="color:#666600">,</span>

    <span style="color:#008800">"database.history.kafka.topic"</span><span style="color:#666600">:</span> <span style="color:#008800">"schema-changes.inventory"</span>

  <span style="color:#666600">}</span>

<span style="color:#666600">}</span></span></span>

Debezium注册的mysql示例文件

每个分厂都有一份，分别注册进Debezium 进行监听（配置的账户必须得有binlog的权限）
6、java消费解析消息体，然后解析出sql并在总部数据库执行