Datax实现离线Tidb(Mysql)到Elasticsearch

1、下载Datax

cd /data
wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
tar -zxvf datax.tar.gz
# 需要删除隐藏文件 (重要)
rm -rf /data/datax/plugin/*/._* 

2、解压后，查看自己需要转换的源数据源和目标数据源插件是否支持，如果都有的话则会有如下文件夹。

/data/datax/plugin/reader/mysqlreader
/data/datax/plugin/writer/elasticsearchwriter

3、通过解压所得我没有 elasticsearchwriter 插件，需自行用源码打包

4、下载Datax源码

https://gitee.com/mirrors/DataX.git

5、去掉自己不需要的 module ，我只留下了自己需要的 elasticsearchwriter

    <modules>
        <module>common</module>
        <module>core</module>
        <module>transformer</module>

        <!-- reader -->

        <!-- writer -->
        <module>elasticsearchwriter</module>

        <!-- common support module -->
        <module>plugin-rdbms-util</module>
        <module>plugin-unstructured-storage-util</module>

    </modules>

6、编译 elasticsearchwriter，在Datax根目录执行

mvn clean install '-Dmaven.test.skip=true'

7、生成的插件包在 DataX\elasticsearchwriter\target\datax\plugin\writer，复制到 /data/datax/plugin/writer 下

8、编写tidb-es.json，其中read中也可以使用querySql来替代，但是 read 中的 column 与 writer 中 column 需对应，最好顺序也一致

{
    "job": {
        "setting": {
            "speed": {
                "channel": 8
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "root",
                        "password": "root",
                        "column": [
                            "id as pk",
                            "id",
                            "indicators_name",
                            "indicators_code",
                            "indicators_region_name",
                            "create_time"
                        ],
                        "where": "",
                        "splitPk": "id",
                        "connection": [
                            {
                                "jdbcUrl": [
                                    "jdbc:mysql://IP:PORT/schema?characterEncoding=utf-8"
                                ],
                                "table": [
                                    "table_name"
                                ]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "elasticsearchwriter",
                    "parameter": {
                        "endpoint": "http://IP:PORT",
                        "index": "datax_table_name",
                        "type": "_doc",
                        "settings": {
                            "index": {
                                "number_of_shards": 5,
                                "number_of_replicas": 1
                            }
                        },
                        "writeMode": "insert",
                        "cleanup": false,
                        "discovery": false,
                        "batchSize": 10000,
                        "splitter": ",",
                        "column": [
                            {
                                "name": "pk",
                                "type": "id"
                            },
                            {
                                "name": "id",
                                "type": "keyword"
                            },
                            {
                                "name": "indicators_name",
                                "type": "text"
                            },
                            {
                                "name": "indicators_code",
                                "type": "text"
                            },
                            {
                                "name": "indicators_region_name",
                                "type": "text"
                            },
                            {
                                "name": "create_time",
                                "type": "date",
                                "format": "yyyy-MM-dd HH:mm:ss"
                            }
                        ]
                    }
                }
            }
        ]
    }
}

9、执行datax

Datax依赖python环境，如果没有python环境，需自行安装。我centos自带的 Python 2.7.5

python /data/datax/bin/datax.py /data/datax/job/tidb-es.json

10、过程再无问题，在Elasticsearch中查到迁移的数据

## Datax作为离线数据迁移工具，对实时迁移并不友好。如果需要定时，可搭配crontab使用。