将本地mysql数据库表格导入到odps大数据计算服务项目中
将本地mysql数据库表格导入到odps大数据计算服务项目中
(还有一种方式是通过odps tunnel的方式:https://help.aliyun.com/document_detail/27833.html)
一、工具下载以及部署
-
方法一、直接下载DataX工具包(如果仅是使用,推荐直接下载):
wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
tar zxvf datax.tar.gz
chmod -R 755 datax
二、配置示例:从MySQL读取数据 写入ODPS
cd datax/job
vim job.json
-
第一步、创建作业的配置文件(json格式)
{ "job": { "content": [ { "reader": { "name": "mysqlreader", "parameter": { "column": [], "connection": [ { "jdbcUrl": [], "table": [] } ], "password": "", "username": "", "where": "" } }, "writer": { "name": "odpswriter", "parameter": { "accessId": "", "accessKey": "", "column": [], "odpsServer": "", "partition": "", "project": "", "table": "", "truncate": true } } } ], "setting": { "speed": { "channel": "" } } } }
-
示例
{ "job": { "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "****", "password": "****", "column": ["id","age","name"], "connection": [ { "table": [ "test_table" ], "jdbcUrl": [ "jdbc:mysql://127.0.0.1:3306/test" ] } ] } }, "writer": { "name": "odpswriter", "parameter": { "accessId": "****", "accessKey": "****", "column": ["id","age","name"], "odpsServer": "http://service.odps.aliyun.com/api", "partition": "pt='datax_test'", "project": "datax_opensource", "table": "datax_opensource_test", "truncate": true } } } ], "setting": { "speed": { "channel": 1 } } } }
-
第三步:启动DataX
cd ../bin
python datax.py ../job/job.json
完成