使用事项:
1. mysql binlog必须是ROW模式
2. 要同步的mysql数据表必须包含主键,否则直接忽略,这是因为如果数据表没有主键,UPDATE和DELETE操作就会因为在ES中找不到对应的document而无法进行同步
3. 不支持程序运行过程中修改表结构
4. 要赋予用于连接mysql的账户RELOAD权限以及REPLICATION权限, SUPER权限:
GRANT REPLICATION SLAVE ON *.* TO 'elastic'@'172.16.32.44';
GRANT RELOAD ON *.* TO 'elastic'@'172.16.32.44';
UPDATE mysql.user SET Super_Priv='Y' WHERE user='elastic' AND host='172.16.32.44';
使用方法
git clone https://github.com/siddontang/go-mysql-elasticsearch
cd go-mysql-elasticsearch/src/github.com/siddontang/go-mysql-elasticsearch
vi etc/river.toml, 修改配置文件,同步172.16.0.101:3306数据库中的webservice.building表到ES集群172.16.32.64:9200的building index(更详细的配置文件说明可以参考项目文档)
# MySQL address, user and password
# user must have replication privilege in MySQL.
my_addr ="172.16.0.101:3306"
my_user ="bellen"
my_pass ="Elastic_123"
my_charset ="utf8"
# Set true when elasticsearch use https
#es_https =false
# Elasticsearch address
es_addr ="172.16.32.64:9200"
# Elasticsearch user and password, maybe set by shield, nginx, or x-pack
es_user =""
es_pass =""
# Path to store data, like master.info,if not set or empty,
# we must use this to support breakpoint resume syncing.
# TODO: support other storage, like etcd.
data_dir ="./var"
# Inner Http status address
stat_addr ="127.0.0.1:12800"
# pseudo server id like a slave
server_id =1001
# mysql or mariadb
flavor ="mariadb"
# mysqldump execution path
# if not set or empty, ignore mysqldump.
mysqldump ="mysqldump"
# if we have no privilege to use mysqldump with --master-data,
# we must skip it.
#skip_master_data =false
# minimal items to be inserted in one bulk
bulk_size =128
# force flush the pending requests if we don't have enough items >= bulk_size
flush_bulk_time ="200ms"
# Ignore table without primary key
skip_no_pk_table =false
# MySQL data source
[[source]]
schema ="webservice"
tables =["building"][[rule]]
schema ="webservice"
table ="building"
index ="building"
type ="buildingtype"