我们公司的业务流程是通过canal模拟MySQL slave的交互协议,获取MySQL的binary log,从而解析MySQL的日志。将解析后的数据通过kafka发送至其他系统从而达到数据同步的目的。
所以canal与kafka的集成也就是我的下一步工作。老规矩,先看官方给的example。
canal与kafka的安装集成配置请看另一个文章:https://blog.csdn.net/h_big_tiger/article/details/100017885
AbstractKafkaTest.java
public abstract class AbstractKafkaTest extends BaseCanalClientTest {
public static String topic = "example";
public static Integer partition = null;
public static String groupId = "g4";
public static String servers = "slave1:6667,slave2:6667,slave3:6667";
public static String zkServers = "slave1:2181,slave2:2181,slave3:2181";
public void sleep(long time) {
try {
Thread.sleep(time);
} catch (InterruptedException e) {
}
}
}
这个类就是指明canal通过什么主题来进行kafka信息传递,partition是之topic的分区,我们只需要采用默认的null就可以,因为配置里有默认值为1。后面的server是canal集群地址。zkServers是zookeeper集群地址。
KafkaClientRunningTest.java
public class KafkaClientRunningTest extends AbstractKafkaTest {
private Logger logger = LoggerFactory.getLogger(KafkaClientRunningTest.class);
private boolean running = true;
public void testKafkaConsumer() {
final ExecutorService executor = Executors.newFixedThreadPool(1);
final KafkaCanalConnector connector = new KafkaCanalConnector(servers, topic, partition, groupId, null, false);
executor.submit(new Runnable() {
@Override
public void run() {
connector.connect();
connector.subscribe();
while (running) {
List<Message> messages = connector.getList(3L, TimeUnit.SECONDS);
if (messages != null) {
System.out.println(messages);
}
connector.ack();
}
connector.unsubscribe();
connector.disconnect();
}
});
sleep(60000);
running = false;
executor.shutdown();
logger.info("shutdown completed");
}
}
这个类是Kafka consumer获取Message的测试例子。可以看到new KafkaCanalConnector(servers, topic, partition, groupId, null, false);来获取监听到的消息。
connector.connect();//开始连接
connector.subscribe();//设置拦截器,即当什么数据库什么表发生变动时向kafka发送消息
connector.getList(3L, TimeUnit.SECONDS);//获取指定数量的消息
CanalKafkaClientExample.java
kafkaCanalClientExample.start();
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
try {
kafkaCanalClientExample.stop();
} catch (Throwable e) {
} finally {
}
}
});
//执行的具体代码
connector.connect();
connector.subscribe();
while (running) {
try {
List<Message> messages = connector.getListWithoutAck(100L, TimeUnit.MILLISECONDS); // 获取message
if (messages == null) {
continue;
}
for (Message message : messages) {
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
}catch (InterruptedException e) {
} else {
// printSummary(message, batchId, size);
// printEntry(message.getEntries());
logger.info(message.toString());
}
}
connector.ack(); // 提交确认
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
这个代码大同小异,就是一直监听canal传递的消息,当监听到之后就会将消息打印出来。
CanalKafkaOffsetClientExample.java
- KafkaOffsetCanalConnector 与 KafkaCanalConnector 的区别是 auto.offset.reset默认值不同;
- KafkaOffsetCanalConnector 默认为earliest;
- canal-kafka-client重启后从未被消费的记录开始拉取消息,同时提供了修改 auto.offset.reset 的方法setAutoOffsetReset