1.安装
1、上传flinkx-1.10.zip 文件,解压
项目地址:https://github.com/DTStack/flinkx
FlinkX压缩包为zip格式,需要安装unzip
安装unzip:yum install unzip
解压
2、 配置环境变量
#flinkX
export FLINKX_HOME=/usr/local/soft/flinkx-1.10
export PATH=$PATH:$FLINKX_HOME/bin
3、给bin目录下的flinkx赋予权限
chmod a+x flinkx
4、在flink-conf.yaml配置文件里配置端口:
#指定web服务端口,不指定的话会随机生成一个
rest.bind-port: 8888
2.使用
2.1 MySQL向HDFS导入数据
1.创建任务目录jsonConf,用于存放任务配置文件
2.MySQL
3.在任务目录下创建任务配置文件 mysqlToHDFS.json
参数说明详见:https://github.com/DTStack/flinkx/blob/master/README_CH.md
{
"job": {
"content": [
{
"reader": {
"parameter": {
"username": "root",
"password": "123456",
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://master:3306/student?characterEncoding=utf8"
],
"table": [
"student"
]
}
],
"column": [
"*"
],
"customSql": "",
"where": "clazz = '理科二班'",
"splitPk": "",
"queryTimeOut": 1000,
"requestAccumulatorInterval": 2
},
"name": "mysqlreader"
},
"writer": {
"name": "hdfswriter",
"parameter": {
"path": "hdfs://master:9000/data/flinkx/student",
"defaultFS": "hdfs://master:9000",
"column": [
{
"name": "col1",
"index": 0,
"type": "int"
},
{
"name": "col2",
"index": 1,
"type": "char"
},
{
"name": "col3",
"index": 2,
"type": "int"
},
{
"name": "col4",
"index": 3,
"type": "char"
},
{
"name": "col5",
"index": 4,
"type": "char"
},
{
"name": "col6",
"index": 5,
"type": "timestamp"
}
],
"fieldDelimiter": ",",
"fileType": "text",
"writeMode": "overwrite"
}
}
}
],
"setting": {
"restore": {
"isRestore": false,
"isStream": false
},
"errorLimit": {},
"speed": {
"channel": 1
}
}
}
}
4.启动任务
flinkx 在bin下
-mode 为本地模式
-job 在jsonConf任务目录下
-pluginRoot 在syncplugins目录下
-flinkconf 在flinkconf目录下
flinkx -mode local -job /usr/local/soft/flinkx-1.10/jsonConf/mysqlToHDFS.json -
pluginRoot /usr/local/soft/flinkx-1.10/syncplugins/ -flinkconf /usr/local/soft/flinkx-
1.10/flinkconf/
命令模板:
bin/flinkx \
-mode local \
-jobType sync \
-job flinkx-local-test/src/main/demo/json/stream/stream.json \
-flinkxDistDir flinkx-dist
5.监听日志
flinkx 任务启动后,会在执行命令的目录下生成一个nohup.out文件
使用tail监听文件
tail -f nohup.out
任务运行后也可以通过8888端口访问flink界面查看任务运行情况
6. 文件导入成功
2.2 更多示例见
https://github.com/DTStack/flinkx/tree/master/flinkx-examples/json