mysql8与elasticsearch7.3.2数据同步，使用docker容器

一、需求软件及相关说明

需求软件
来源容器镜像
是否改造
改造后镜像
支持特别说明
备注
Mysql8.0
mysql:8.0
否
 
 
 
elasticsearch-head
mobz/elasticsearch-head:5-alpine
是
registry.cn-shanghai.aliyuncs.com/qlongit/elasticsearch-head:v1.0
 
 
拉取最近代码修改而成，原有的代码无法显示数据
Elasticsearch7.3.2
Elasticsearch:7.3.2
否
 
 
安全性考虑升级到较高版本7.3.2
但是此版本不支持一个索引下多个类型，支持一个索引下单个类型
go-mysql-elasticsearch
v2.1
mysql:8.0
是
registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v2.1
支持mysql8.0版本
基于mysql8.0做mysqldump的客户端，使用mariadb最新的客户端版本是不兼容mysql8.0服务端的
go-mysql-elasticsearch
v1
https://github.com/siddontang/go-mysql-elasticsearch
是
registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v1
支持mysql5.7以下版本
因为Dockerfile文件构建镜像时go的模块文件下载一直报错，所以改用国内的下载源
kibana
 
 
 
 
待有时间再上线这个
Java服务端调用
 
 
 
 
使用jest方式调用
 

 

二、自己制作的容器镜像上传地址

目前都在阿里云上

 

三、docker-compose的编排脚本

3.1、编排脚本

version: '2.2'

services:

  es01:

    image: elasticsearch:7.3.2

    container_name: es01

    environment:

      - node.name=es01

      - discovery.seed_hosts=es02

      - cluster.initial_master_nodes=es01,es02

      - cluster.name=docker-cluster

      - bootstrap.memory_lock=true

      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

    ulimits:

      memlock:

        soft: -1

        hard: -1

    volumes:

      - esdata01:/usr/share/elasticsearch/data

      - ~/elastic/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

    ports:

      - 9200:9200

    networks:

      - esnet

  es02:

    image: elasticsearch:7.3.2

    container_name: es02

    environment:

      - node.name=es02

      - discovery.seed_hosts=es01

      - cluster.initial_master_nodes=es01,es02

      - cluster.name=docker-cluster

      - bootstrap.memory_lock=true

      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

    ulimits:

      memlock:

        soft: -1

        hard: -1

    volumes:

      - esdata02:/usr/share/elasticsearch/data

      - ~/elastic/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

   networks:

      - esnet

  eshead:

    image: elasticsearch-head:v1.0

    container_name: elasticsearch-head

    ports:

      - 9100:9100

    links:

      - es01:es

    command:

      - sh

      - -c

      - npm run start

    networks:

      - esnet

  mysql8:

    image: mysql:8.0

    container_name: mysql8

    ports:

      - 13306:3306

    environment:

      - MYSQL_ROOT_PASSWORD=*****

      - MYSQL_DATABASE=****

      - MYSQL_USER=*****

      - MYSQL_PASSWORD=******

    volumes:

      - ~/elastic/mysql8/data:/var/lib/mysql

    networks:

      - esnet

  gomysqles:

    image: go-mysql-es:v2.1

    container_name: go-mysql-es

    volumes:

      - ~/elastic/go-mysql-elasticsearch/var:/goworkerspace/bin/var

      - ~/elastic/go-mysql-elasticsearch/etc/river.toml:/goworkerspace/bin/etc/river.toml

    networks:

      - esnet

    links:

      - es01:es

      - mysql8:mysql

    depends_on:

      - es01

      - es02

      - mysql8

volumes:

  esdata01:

    driver: local

  esdata02:

    driver: local

 

networks:

  esnet:

 

 

其中一些敏感的数据库账号与密码需要自行调整，还要镜像调整为上面阿里云镜像

 

关于一些外挂的一些配置文件看下面几章节

3.2、elasticsearch配置文件如下：

~/elastic/elastic/elasticsearch.yml

配置文件位置视系统环境调整

 

cluster.name: "docker-cluster"

network.host: 0.0.0.0

http.cors.enabled: true

http.cors.allow-origin: "*"

 

3.3、go-mysql-elasticsearch的配置文件

~/elastic/go-mysql-elasticsearch/etc/river.toml

 

 

# MySQL address, user and password

# user must have replication privilege in MySQL.

my_addr = "mysql:3306"

my_user = "***"

my_pass = "*****"

my_charset = "utf8"

 

# Set true when elasticsearch use https

#es_https = false

# Elasticsearch address

es_addr = "es:9200"

# Elasticsearch user and password, maybe set by shield, nginx, or x-pack

es_user = ""

es_pass = ""

 

# Path to store data, like master.info, if not set or empty,

# we must use this to support breakpoint resume syncing.

# TODO: support other storage, like etcd.

data_dir = "./var"

 

# Inner Http status address

stat_addr = "127.0.0.1:12800"

 

# pseudo server id like a slave

server_id = 1001

 

# mysql or mariadb

flavor = "mysql"

#flavor = "mariadb"

 

# mysqldump execution path

# if not set or empty, ignore mysqldump.

mysqldump = "mysqldump"

 

# if we have no privilege to use mysqldump with --master-data,

# we must skip it.

#skip_master_data = false

 

# minimal items to be inserted in one bulk

bulk_size = 128

 

# force flush the pending requests if we don't have enough items >= bulk_size

flush_bulk_time = "200ms"

 

# Ignore table without primary key

skip_no_pk_table = false

 

# MySQL data source

[[source]]

schema = "sakila"

 

# Only below tables will be synced into Elasticsearch.

# "t_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023

# I don't think it is necessary to sync all tables in a database.

tables = ["sougoulog","film"]

 

# Below is for special rule mapping

 

# Very simple example

#

# desc t;

# +-------+--------------+------+-----+---------+-------+

# | Field | Type         | Null | Key | Default | Extra |

# +-------+--------------+------+-----+---------+-------+

# | id    | int(11)      | NO   | PRI | NULL    |       |

# | name  | varchar(256) | YES  |     | NULL    |       |

# +-------+--------------+------+-----+---------+-------+

#

# The table `t` will be synced to ES index `test` and type `t`.

[[rule]]

schema = "sakila"

table = "sougoulog"

index = "sakila_sougoulog"

type = "sougoulog"

 

[[rule]]

schema = "sakila"

table = "film"

index = "sakila_film"

type = "film"

 

 

 

如果是es6以下版本支持一个index下多个type，

在es7.3.2版本是不支持一个index多个type的，只能是一个index一个type。

这个视个人情况来配置

 

四、关于go-mysql-elasticsearch不支持DDL操作同步解决方案

4.1、删除做了ddl操作的index

4.2、新开一个docker镜像

新开的只有原来的表，重新mysqldump操作，需要调整配置文件与删除日志存储点文件实现

4.3、销毁新开的，原来的就会重新按binlog来同步

五、关于go-mysql-elasticsearch运行过久不同步的问题

这个大家都遇到过，建立一个定时任务重启docker即可。

真正解决，目前要看作者有无时间研究处理。本人对于go也才刚刚接触