canal学习-运行canal-adapter源码并记录解决报错问题(包含缺失的jar文件)（一）

省流：最后面有能直接跑的源码，可以直接下载运行

1. 下载canal源码

1.1 下载源码并安装好环境

官网地址：https://github.com/alibaba/canal/releases/tag/canal-1.1.5

服务端的部署参考我之前写过的文档： canal实时同步mysql数据到elasticsearch，注意只需要去部署一个服务端
保证以下组件已经安装运行

组件	版本
mysql	8.0.23
elasticsearch	7.9.0

1.2 查看目录结构

下载解压后的目录结构如下，我们当前只关注客户端的代码

2.项目运行

使用idea打开项目，由于需求是往es同步数据，因此需要关注以下三个模块

2.1 项目打包

打包顺序如下,务必不要打乱顺序，模块之间存在依赖

1. escore->mvn clean install
2. es7x->mvn clean install
3. launcher->mvn clean install

2.2 项目打包可能遇到的问题：

1.Failure to find com.alibaba.otter:connector.tcp:jar:jar-with-dependencies:1.1.5

[ERROR] Failed to execute goal on project client-adapter.launcher: Could not resolve dependencies for project com.alibaba.otter:client-adapter.launcher:jar:1.1.5: Failure to find com.alibaba.otter:connector.tcp:jar:jar-with-dependencies:1.1.5 in https://maven.aliyun.com/repository/public was cached in the local repository, resolution will not be reattempted until the update interval of aliyunmaven has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 

原因：connector.tcp maven下载不到导致的，tmd吊jar包找了半天才找到，还要劳资扫码关注才能下载

链接：https://pan.baidu.com/s/1Biw-Gwum-vPaqfWhqc_gGQ?pwd=of2e
提取码：of2e

解决方案：下载jar后放到报错信息中maven仓库的位置

2.com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource

ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
	at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
	at co 
Caused by: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
	at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:83) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
	at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:52) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
	... 42 common frames omitted
Caused by: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
	at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.addSyncConfigToCache(ESAdapter.java:146) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
	at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:75) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
	... 43 common frames omitted
2023-05-15 21:29:59.969 [main] INFO  c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: D:\IdeaProjects\canal\canal-canal-1.1.5\client-adapter\launcher\target\canal-adapter\plugin
2023-05-15 21:29:59.985 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: cqu-g1 succeed
2023-05-15 21:29:59.985 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2023-05-15 21:29:59.985 [Thread-9] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: cqu <=============
2023-05-15 21:29:59.989 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2023-05-15 21:29:59.989 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2023-05-15 21:30:00.023 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2023-05-15 21:30:00.026 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.208 seconds (JVM running for 4.171)
2023-05-15 21:30:00.100 [Thread-9] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: cqu succeed <=============

原因：common和escore的druid包冲突
解决方案：
限制escore的pom文件中的druid的作用域，定位到 client-adapter.escore 模块的 pom.xml

上图红框出修改为

<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>druid</artifactId>
    <scope>provided</scope>
</dependency>

3.配置application.yml

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 10000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 192.168.0.10:11111 #canal服务端地址
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 5000
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://192.168.0.10:3306/dev_db?useUnicode=true #测试数据库连接
      username: canal #数据库账号
      password: canal #数据库密码
  canalAdapters:
    - instance: cqu # canal instance Name or mq topic name
      groups:
        - groupId: g1
          outerAdapters:
            - name: es7 # ES同步适配器
              hosts: http://192.168.0.10:11700 # ES连接地址
              properties:
                mode: rest # 模式可选transport(9300) 或者 rest(9200)
                cluster.name: es-single # ES集群名称, 与es目录下 elasticsearch.yml文件cluster.name对应

4.运行项目

4.1 运行启动类：CanalAdapterApplication.java

4.2 Config dir not found 报错解决

运行时可能会有的报错：

ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: Config dir not found.
	at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
	at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.loadAdapter(CanalAdapterLoader.java:225) [classes/:na]
	at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:56) [classes/:na]
	at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) [classes/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_221]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_221]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_221]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_221]

原因：launcher模块下缺少es7文件夹
解决：将es7x下的es7文件夹复制到launcher的resources目录下


target中生成的文件

重新运行，即可运行成功

5.可直接运行的项目源码

以下代码在运行前，需要在根目录执行mvn clean install 命令，然后修改application.yml的配置后就可以直接运行。

链接：https://pan.baidu.com/s/1sKOdIdAMGkgHP1XddWfGLA?pwd=yw33
提取码：yw33