DataX 实现从MySQL导入数据到HDFS

最新推荐文章于 2024-09-03 23:05:14 发布

落花雨时

最新推荐文章于 2024-09-03 23:05:14 发布

阅读量2.5k

点赞数

分类专栏：大数据文章标签： hdfs mysql hadoop

本文链接：https://blog.csdn.net/mengxianglong123/article/details/121007249

版权

大数据专栏收录该内容

107 篇文章 52 订阅

订阅专栏

本文详细介绍了在Linux环境下安装和配置DataX的过程，包括从MySQL读取数据并将其写入HDFS的步骤。首先，通过yum安装Python和DataX，然后编写配置文件进行数据迁移。在案例中，创建了student表并插入数据，最后执行DataX任务将数据从MySQL迁移到HDFS。

摘要由CSDN通过智能技术生成

一、环境准备

Linux
JDK(1.8以上，推荐1.8)
Python(推荐Python2.6.X)

yum安装Python:https://www.cnblogs.com/kaishirenshi/p/11858655.html

# centos7
# 换成阿里云的yum源
yum -y install epel-release
yum repolist
yum -y install python36

下载Datax：https://github.com/alibaba/DataX

二、案例实操

2.1 安装

1）将下载好的datax.tar.gz上传到hadoop101的/opt/softwares
2）解压datax.tar.gz到/opt/module

[atguigu@hadoop102 software]$ tar-zxvf datax.tar.gz-C /opt/module/

3）运行自检脚本

[atguigu@hadoop102 bin]cd /opt/module/datax/bin/
[atguigu@hadoop102 bin] python datax.py /opt/module/datax/job/job.json

2.2 查看官方模板

python /opt/module/datax/bin/datax.py - r mysqlreader - w

hdfswriter {
	"job": {
		"content": [{
			"reader": {
				"name": "mysqlreader",
				"parameter": {
					"column": [],
					"connection": [{
						"jdbcUrl": [],
						"table": []
					}],
					"password": "",
					"username": "",
					"where": ""
				}
			},
			"writer": {
				"name": "hdfswriter",
				"parameter": {
					"column": [],
					"compress": "",
					"defaultFS": "",
					"fieldDelimiter": "",
					"fileName": "",
					"fileType": "",
					"path": "",
					"writeMode": ""
				}
			}
		}],
		"setting": {
			"speed": {
				"channel": ""
			}
		}
	}
}

在这里插入图片描述

2.3 准备数据

1）创建 student 表

mysql> create database datax;
mysql> use datax;
mysql> create table student(id int,name varchar(20));

2）插入数据

mysql> insert into student values(1001,'zhangsan'),(1002,'lisi'),(1003,'wangwu');

2.4 编写配置文件

vim /opt/module/datax/job/mysql2hdfs.json

{
	"job": {
		"content": [{
			"reader": {
				"name": "mysqlreader",
				"parameter": {
					"column": [
						"id",
						"name"
					],
					"connection": [{
						"jdbcUrl": [
							"jdbc:mysql://hadoop102:3306/datax"
						],
						"table": [
							"student"
						]
					}],
					"username": "root",
					"password": "000000"
				}
			},
			"writer": {
				"name": "hdfswriter",
				"parameter": {
					"column": [{
							"name": "id",
							"type": "int"
						},
						{
							"name": "name",
							"type": "string"
						}
					],
					"defaultFS": "hdfs://hadoop102:9000",
					"fieldDelimiter": "\t",
					"fileName": "student.txt",
					"fileType": "text",
					"path": "/",
					"writeMode": "append"
				}
			}
		}],
		"setting": {
			"speed": {
				"channel": "1"
			}
		}
	}
}