Canal 1.1.5 从MySQL8.0导入数据到ES7.0

教程

坑1：

client-adapter 默认的mysql连接器是 5.x版本,需要在其lib目录下添加8.x版本的mysql连接驱动器

坑2: client-adapter1.1.5版本的 plugin目录下的es7.x 的jar包是有错的,需要换为

这样的：

坑三: ES需要先插入一条数据才保证时间戳类型能转换

https://github.com/alibaba/canal/issues/3017

全量同步命令:

curl http://127.0.0.1:8081/etl/es7/exampleKey/mytest_user.yml -X POST

注意exanpleKey是 application.yaml文件中的 outerAdapters 下的 key

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: -1
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 172.18.66.28:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
  srcDataSources:
    defaultDS: # 自定义名称
      url: jdbc:mysql://rm-wz97bj1yo3quz5b28co.mysql.rds.aliyuncs.com:3306/carbon?useUnicode=true   # jdbc url 
      username: carbon_user                                            # jdbc 账号
      password: 71&0925!jh#kd                                          # jdbc 密码
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger
      - name: es7
        key: exampleKey
        hosts: 122.112.151.24:9200  # 9200的话要加 for rest mode,9300 for transport
        properties:
          mode: rest # or transport
          # security.auth: test:123456 #  only used for rest mode
          cluster.name: elasticsearch