参考地址
碰到的bug
第一次按照启动canal.adapter的时候报错,经典空指针异常,代码如下:
2024-08-03 22:29:11.937 [pool-3-thread-1] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:112)
at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:60)
at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.sync(ESAdapter.java:104)
at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.sync(ESAdapter.java:83)
at com.alibaba.otter.canal.client.adapter.ProxyOuterAdapter.sync(ProxyOuterAdapter.java:42)
at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.batchSync(AdapterProcessor.java:139)
at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.lambda$null$1(AdapterProcessor.java:97)
at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890)
at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.lambda$null$2(AdapterProcessor.java:94)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
at com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate.insert(ES7xTemplate.java:70)
at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.singleTableSimpleFiledInsert(ESSyncService.java:448)
at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.insert(ESSyncService.java:135)
at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:95)
看了官网和网上各种文档都没找到原因,于是就取找canal报错位置的源码,打印内容重新打包,使用这个jar
@Override
public Object getValFromData(ESMapping mapping, Map<String, Object> dmlData, String fieldName, String columnName) {
String esType = getEsType(mapping, fieldName);
logger.info("gwb1dmlData==="+dmlData);
Object value = dmlData.get(columnName);
logger.info("gwb1===");
logger.info("value1:"+columnName+value);
if (value instanceof Byte) {
if ("boolean".equals(esType)) {
value = ((Byte) value).intValue() != 0;
}
}
日志中发现从mysql数据库中查到的内容和mysql的字段没有对应,数据库中的主键是id,但是canal查出来的竟然是article_id
CREATE TABLE `test_es_articles` (
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`user_id` INT UNSIGNED,
`article_title` VARCHAR(255) NOT NULL,
`article_content` TEXT,
`publish_date` DATETIME DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='文章表';
2024-08-03 21:55:31.247 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - gwb1dmlData==={article_id=1, user_id=1, article_title=小黑子1, article_content=这是文章1的内容, publish_date=2024-07-01 10:00:00.0}
2024-08-03 21:55:31.247 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - gwb1===
2024-08-03 21:55:31.247 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - value1:id
所以导致在查询id的时候找不到,导致报空指针异常,就很蒙比,我在想会不会是从mysql里面查的不是最新的表结构,重启mysql,重启canal
2024-08-03 22:59:54.787 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5,"canal_content":"这是一条示例数据,包含一些文本内容。"},{"id":6,"canal_content":"这是第二条示例数据,同样包含文本信息。"},{"id":7,"canal_content":"示例数据三,继续包含文本,用于测试同步和索引。"},{"id":8,"canal_content":"第四条数据,展示了不同长度的文本内容。"}],"database":"ry-vue","destination":"example","es":1722697194000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test_canal","ts":1722697194494,"type":"INSERT"}
2024-08-03 22:59:54.787 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - 字段名:[com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem$FieldItem@d1b, com.alibaba.otter.canal.client.adapter.es.core.config.SchemaItem$FieldItem@3398835]
2024-08-03 22:59:54.919 [pool-3-thread-1] WARN org.elasticsearch.client.RestClient - request [GET http://127.0.0.1:9200/test_canal/_mapping?master_timeout=30s&ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true] returned 2 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."],[299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
2024-08-03 22:59:55.093 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - gwb1dmlData==={id=5, canal_content=这是一条示例数据,包含一些文本内容。}
2024-08-03 22:59:55.093 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - gwb1===
2024-08-03 22:59:55.093 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - value1:id5
2024-08-03 22:59:55.093 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - gwb2===
2024-08-03 22:59:55.093 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - value2:5
2024-08-03 22:59:55.093 [pool-3-thread-1] INFO c.a.o.canal.client.adapter.es.core.support.ESTemplate - gwb1dmlData==={id=5, canal_content=这是一条示例数据,包含一些文本内容。}
ok啦铁子,没毛病,但是我不能笃定问题在这,于是我有重新创建了es索引和mysql表,然后同步mysql数据到es中
CREATE TABLE `test_canal` (
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`canal_content` TEXT,
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='canal测试表';
PUT /test_canal
{
"settings": {
"analysis": {
"analyzer": {
"ik_analyzer": {
"type": "custom",
"tokenizer": "ik_smart"
}
}
}
},
"mappings": {
"properties": {
"id": { "type": "integer" },
"canal_content": {
"type": "text"
}
}
}
}
再往mysql中同步数据,发现数据已经同步到es所以,上面参考文档没问题,但是我们最好在使用canal前重启一下mysql