sqoop之hive数据导入mysql

最新推荐文章于 2021-04-30 02:42:08 发布
最新推荐文章于 2021-04-30 02:42:08 发布
阅读量538 收藏 4
点赞数 1
分类专栏： # Sqoop 大数据 文章标签： sqoop hive mysql
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/hyj_king/article/details/107024526
版权

1、查询hive表中的student数据及信息

hive> select * from student;
OK
001	xiaohong	002
002	xiaolan	002
001	xiaohong	003
002	xiaolan	003
Time taken: 0.114 seconds, Fetched: 4 row(s)
hive> desc formatted student;
OK
# col_name            	data_type           	comment             
	 	 
id                  	string              	                    
name                	string              	                    
	 	 
# Partition Information	 	 
# col_name            	data_type           	comment             
	 	 
classroom           	string              	                    
	 	 
# Detailed Table Information	 	 
Database:           	default             	 
Owner:              	root                	 
CreateTime:         	Thu Mar 19 14:00:19 CST 2020	 
LastAccessTime:     	UNKNOWN             	 
Retention:          	0                   	 
Location:           	hdfs://master:9000/user/hive/warehouse/student	 
Table Type:         	MANAGED_TABLE       	 
Table Parameters:	 	 
	transient_lastDdlTime	1584597619          
	 	 
# Storage Information	 	 
SerDe Library:      	org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe	 
InputFormat:        	org.apache.hadoop.mapred.TextInputFormat	 
OutputFormat:       	org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat	 
Compressed:         	No                  	 
Num Buckets:        	-1                  	 
Bucket Columns:     	[]                  	 
Sort Columns:       	[]                  	 
Storage Desc Params:	 	 
	field.delim         	,                   
	serialization.format	,                   
Time taken: 0.065 seconds, Fetched: 32 row(s)

从上面的信息可知，student是以classroom为分区进行存储的；数据存储位置为/user/hive/warehouse/student；

2、查看hive中student表数据的存储位置

[root@master ~]# hdfs dfs -ls /user/hive/warehouse/student/
Found 2 items
drwxr-xr-x   - root supergroup          0 2020-03-19 14:04 /user/hive/warehouse/student/classroom=002
drwxr-xr-x   - root supergroup          0 2020-03-19 14:10 /user/hive/warehouse/student/classroom=003
[root@master ~]# hdfs dfs -ls /user/hive/warehouse/student/classroom=002
Found 1 items
-rwxr-xr-x   1 root supergroup         25 2020-03-19 14:04 /user/hive/warehouse/student/classroom=002/stu.txt
[root@master ~]# hdfs dfs -cat /user/hive/warehouse/student/classroom=002/stu.txt
001,xiaohong
002,xiaolan

由此可知根据逗号分隔数据；

3、先在mysql中创建字段和类型相同的表

mysql> create table student(id varchar(255),name varchar(255));
Query OK, 0 rows affected (0.02 sec)

mysql> select * from student;
Empty set (0.00 sec)

mysql>

4、sqoop导入数据

[root@master bin]# ./sqoop export --connect "jdbc:mysql://192.168.230.21:3306/mysql" --username root --password 123456 --table student  --fields-terminated-by "," --export-dir "/user/hive/warehouse/student/classroom=002";
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/06/29 17:27:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
20/06/29 17:27:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/06/29 17:27:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
20/06/29 17:27:58 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/softWare/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hbase/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hive/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/06/29 17:27:58 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:27:58 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:27:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/softWare/hadoop/hadoop-2.7.3
Note: /tmp/sqoop-root/compile/ee585e722d7b5a438d67d53c369ed36e/student.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
20/06/29 17:28:00 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/ee585e722d7b5a438d67d53c369ed36e/student.jar
20/06/29 17:28:00 INFO mapreduce.ExportJobBase: Beginning export of student
20/06/29 17:28:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
20/06/29 17:28:01 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
20/06/29 17:28:01 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:28:01 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/06/29 17:28:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
20/06/29 17:28:06 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:28:06 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:28:06 INFO mapreduce.JobSubmitter: number of splits:4
20/06/29 17:28:06 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:28:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1593394152340_0027
20/06/29 17:28:07 INFO impl.YarnClientImpl: Submitted application application_1593394152340_0027
20/06/29 17:28:07 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1593394152340_0027/
20/06/29 17:28:07 INFO mapreduce.Job: Running job: job_1593394152340_0027
20/06/29 17:28:16 INFO mapreduce.Job: Job job_1593394152340_0027 running in uber mode : false
20/06/29 17:28:16 INFO mapreduce.Job:  map 0% reduce 0%
20/06/29 17:28:37 INFO mapreduce.Job:  map 50% reduce 0%
20/06/29 17:28:38 INFO mapreduce.Job:  map 100% reduce 0%
20/06/29 17:28:38 INFO mapreduce.Job: Job job_1593394152340_0027 completed successfully
20/06/29 17:28:38 INFO mapreduce.Job: Counters: 30
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=551488
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=764
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=19
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=0
	Job Counters 
		Launched map tasks=4
		Data-local map tasks=4
		Total time spent by all maps in occupied slots (ms)=71778
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=71778
		Total vcore-milliseconds taken by all map tasks=71778
		Total megabyte-milliseconds taken by all map tasks=73500672
	Map-Reduce Framework
		Map input records=2
		Map output records=2
		Input split bytes=681
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=799
		CPU time spent (ms)=3130
		Physical memory (bytes) snapshot=445382656
		Virtual memory (bytes) snapshot=8325513216
		Total committed heap usage (bytes)=170655744
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=0
20/06/29 17:28:38 INFO mapreduce.ExportJobBase: Transferred 764 bytes in 36.6371 seconds (20.8532 bytes/sec)
20/06/29 17:28:38 INFO mapreduce.ExportJobBase: Exported 2 records.
[root@master bin]# ./sqoop export --connect "jdbc:mysql://192.168.230.21:3306/mysql" --username root --password 123456 --table student  --fields-terminated-by "," --export-dir "/user/hive/warehouse/student/classroom=003";
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/softWare/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/06/29 17:29:21 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
20/06/29 17:29:21 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/06/29 17:29:22 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
20/06/29 17:29:22 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/softWare/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hbase/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/softWare/hive/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/06/29 17:29:22 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:29:22 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1
20/06/29 17:29:22 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/softWare/hadoop/hadoop-2.7.3
Note: /tmp/sqoop-root/compile/b4bafb65b47b2cded3caa3989b6e4d00/student.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
20/06/29 17:29:24 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/b4bafb65b47b2cded3caa3989b6e4d00/student.jar
20/06/29 17:29:24 INFO mapreduce.ExportJobBase: Beginning export of student
20/06/29 17:29:24 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
20/06/29 17:29:25 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
20/06/29 17:29:25 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:29:25 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/06/29 17:29:25 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.230.21:8032
20/06/29 17:29:31 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:29:31 INFO input.FileInputFormat: Total input paths to process : 1
20/06/29 17:29:31 INFO mapreduce.JobSubmitter: number of splits:4
20/06/29 17:29:31 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
20/06/29 17:29:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1593394152340_0028
20/06/29 17:29:32 INFO impl.YarnClientImpl: Submitted application application_1593394152340_0028
20/06/29 17:29:32 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1593394152340_0028/
20/06/29 17:29:32 INFO mapreduce.Job: Running job: job_1593394152340_0028
20/06/29 17:29:41 INFO mapreduce.Job: Job job_1593394152340_0028 running in uber mode : false
20/06/29 17:29:41 INFO mapreduce.Job:  map 0% reduce 0%
20/06/29 17:29:52 INFO mapreduce.Job:  map 50% reduce 0%
20/06/29 17:29:54 INFO mapreduce.Job:  map 75% reduce 0%
20/06/29 17:29:55 INFO mapreduce.Job:  map 100% reduce 0%
20/06/29 17:29:55 INFO mapreduce.Job: Job job_1593394152340_0028 completed successfully
20/06/29 17:29:56 INFO mapreduce.Job: Counters: 31
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=551488
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=764
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=19
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=0
	Job Counters 
		Launched map tasks=4
		Data-local map tasks=2
		Rack-local map tasks=2
		Total time spent by all maps in occupied slots (ms)=35902
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=35902
		Total vcore-milliseconds taken by all map tasks=35902
		Total megabyte-milliseconds taken by all map tasks=36763648
	Map-Reduce Framework
		Map input records=2
		Map output records=2
		Input split bytes=681
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=500
		CPU time spent (ms)=2940
		Physical memory (bytes) snapshot=448909312
		Virtual memory (bytes) snapshot=8319729664
		Total committed heap usage (bytes)=166592512
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=0
20/06/29 17:29:56 INFO mapreduce.ExportJobBase: Transferred 764 bytes in 31.064 seconds (24.5944 bytes/sec)
20/06/29 17:29:56 INFO mapreduce.ExportJobBase: Exported 2 records.

5、查看mysql中student表数据结果

mysql> select * from student;
+------+----------+
| id   | name     |
+------+----------+
| 001  | xiaohong |
| 002  | xiaolan  |
| 001  | xiaohong |
| 002  | xiaolan  |
+------+----------+
4 rows in set (0.00 sec)

mysql>

 

郝少
关注
  • 1
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论
专栏目录
sqoop将hive中数据导入mysql（实例讲解）
Saint
07-02 3934
假设这样一个场景： 数据已经导入hive（hive实例讲解看这里），现在要把hive中数据库导入出到（mysql）关系型数据库中，这里使用的工具是sqoop(sqoop详细讲解看这里)。 该怎么处理呢？ 首先，我们需要的是在mysql中建立对应的表，数据字段类型与hive数据表中要导出的数据类型保持一致； 然后，我们需要在假如你不是需要原样输出而是需要经过计算统计的话，就需要写在h...
《从0开始学大数据》之如何自己开发一个大数据SQL引擎
CJJ的博客
01-30 1278
极客时间《从0开始学大数据》学习笔记
使用Sqoop实现HIve分析结果数据入mysql库
lrf2454224026的博客
09-11 1079
1.将hive作业中的结果，使用sqoop 导入mysql数据库。        创建stock表并导入数据                        创建stock_result表来存放stock的查询结果              自定义jar包并创建相应的函数（这一步走可以省略，可以使用hive自带的函数）                     将查询结果导入stoc...
Hive中使用sql的注意事项
weixin_30572613的博客
09-13 328
一、别名的使用 1 定义别名：columnA as X 不需要使用单引号 2 使用别名：不与where同时使用 花式报错-->有说hive不支持where后使用别名 二、GROUP BY select m.makeName ,SUM(c) from (select t.makeId as id ,count(o...
sqoop hive mysql 拒接连接
weixin_39712314的博客
12-13 620
2111 JobHistoryServer [hadoop@namenode1 ~]$ sqoop export --connect jdbc:mysql://localhost:3306/dbtaobao --username root --table user_log --export-dir '/usr/local/hive/warehouse/dbtaobao.db/inner_user_...
Apache Sqoop数据转换安装配置与基本操作命令
ZeroTeam_麒麟
05-25 2932
系统环境CentOS 6.5组件版本 name version Sqoop 1.4.6 Sqoop安装解压 懒的敲了配置环境变量 # SQOOP SQOOP_HOME=/home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0 PATH=$PATH:$SQOOP_HOME/bin export PATHcd 到Sqoop根目录 配置Sqoop
2、sqoop导入（RMDB-mysql、sybase到HDFS-hive）
最新发布
05-29
2、sqoop导入（RMDB-mysql、sybase到HDFS-hive） 网址：https://blog.csdn.net/chenwewi520feng/article/details/130572275 介绍sqoop从关系型数据库mysql、sybase同步到hdfs、hive中
sqoop从mysql中导入数据到parquet格式的hive中
07-07
sqoop导入数据到hive
sqoop 从 hive 导到mysql遇到的问题.docx
01-12
在将数据从 Hive 导入到 MySQL 时，Sqoop 可能会遇到一些问题。例如，在描述文件中，Sqoop 的版本为 1.4.4，Hadoop 的版本为 1.0.0，Hive 的版本为 0.11.0。 Sqoop 在连接 MySQL 时，需要指定 JDBC 连接器的路径，...
BigData:Sqoop+Hive+MySQL 纽约证券交易所数据集
07-05
在mysql中创建表，将NYSE数据导入MySQL Sqoop 将数据从mysql导入HIVE 操作hive中的数据来计算每个股票代码的股票量 将此数据存储到 hive 上的 stock_volume 表中 使用 sqoop 将此数据从 hive 导出回 mysql 进行...
数据同步Sqoop用法之mysql与Hive导入导出.docx
11-12
Sqoop 用法之 MySQL 与 Hive 导入导出 Sqoop 是一个专门用于将 Hadoop 和关系型数据库中的数据相互转移的工具，可以将一个关系型数据库（例如：MySQL、Oracle、Postgres 等）中的数据导进到 Hadoop 的 HDFS 中，也...
sqoop数据采集
枣泥馅的博客
07-03 1356
什么是sqoop sqoop的全称是SQL-To-Hadoop，它是一个关系型数据库和Hadoop之间数据交换的工具，它从sql导入到hadoop的过程叫做import，从hadoop导出sql的过程叫做export。import和export都是基于mapreduce的 安装Sqoop sqoop的安装非常节点，只需要把它的包解压后，再配置一下环境变量就可以了 如果采集Oracle中的数据，需要把Oracle安装包中lib的jar放入sqoop的lib目录中我的虚拟机中oracle的jdbc包ojdbc1
SQL注入—我是如何一步步攻破一家互联网公司的
wetest_tencent的博客
04-15 1127
本文由腾讯WeTest团队提供，更多资讯可直接戳链接查看：http://wetest.qq.com/lab/ 微信号：TencentWeTest最近在研究Web安全相关的知识，特别是SQL注入类的相关知识。接触了一些与SQL注入相关的工具。周末在家闲着无聊，想把平时学的东东结合起来攻击一下身边某个小伙伴去的公司，看看能不能得逞。不试不知道，一试还真TM得逞了，内心有些小激动，特在此写博文一篇，...
SQL Programming（3）：Executing SQL Statements
u012501350的专栏
02-25 480
QSqlQuery 类为执行SQL语句和浏览查询结果上提供了交互接口。     QSqlQueryModel类和QSqlTableModel在访问数据库上提供了更高水平的接口，这两个类将在下一部分进行说明。如果你不熟悉SQL，那么你可以直接查看下一部分（Using the SQL Model Class）。 执行一条语句 要执行一条SQL语句，你可以创建一个QSqlQuery
linux查看sqoop目录,Sqoop安装和简单使用
weixin_39639568的博客
04-30 317
下载sqoop因为官方并不建议在生产环境中使用sqoop2，即1.99.7，所以这里我用的是sqoop1,即1.4.7点击下载：http://mirror.bit.edu.cn/apache/sqoop/解压和上传我用的xshell辅助工具，先在本地解压改了名称之后上传到Linux上的sqoop.png上传linuxlinux下.png也可上传压缩包，执行解压缩，执行命令：cd /home/...
Sqoop向hdfs上传文件时报错
qq_42714108的博客
07-15 517
INFO client.RMProxy: Connecting to ResourceManager at master/192.168.49.200:8032 INFO ipc.Client: Retrying connect to server: master/192.168.49.200:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=100, sleepTime
sqoop1.4.7部署及其使用之旅
sunnyJam的专栏
03-31 861
为什么使用sqoop? 1、对于hadoop数据的处理有时候要用户关系型数据库（mysql，oracle）中的数据进行清洗，或处理的数据需要导入到关系型数据库中； 2、由于没有工具的支持hadoop的hdfs和数据库之间的交互，手工写map-reduce中来处理复杂繁琐，维护成功高。 3、sqoop是连接关系型数据库和hadoop的桥梁，主要有两个方面(导入和导出)； 开始部署...
linux查看sqoop目录,Linux安装部署Sqoop
weixin_42356844的博客
04-30 176
下载sqoopwget http://apache-mirror.8birdsvideo.com/sqoop/1.4.7/sqoop-1.4.7.tar.gz解压重命名[root@hadoop001 software]# tar -xzvf sqoop-1.4.7.tar.gz[root@hadoop001 software]# mv sqoop-1.4.7/ sqoopMySQL JDBC驱动包...
sqoop-介绍以及安装
daxiongwuwuwuw的博客
07-29 515
sqoop介绍以及安装简介底层原理安装下载地址部署测试 简介 sqoop是一款开源工具，用于在hive和传统数据库间mysql做数据传递，也就是可以从hive导入到关系型数据库也可以从关系型数据库导入到hive这意思 底层原理 事实上sqoop就是把底层原理翻译成了mapreduce程序 翻译出的mapreduce程序主要是对inputformat和outputformat进行定制 安装 前提：有java，有hadoop。 下载地址 http://mirrors.hust.edu.cn/apache/sqo
sqoop将hive数据导入mysql
03-16
sqoop是一种用于在Hadoop和关系型数据库之间传输数据的工具。要将Hive数据导入MySQL，可以使用以下步骤： 1. 确保已经安装了sqoop和MySQL驱动程序。 2. 在Hive中创建一个表，并将其导出为一个文件。例如，可以使用以下命令将Hive表导出为一个CSV文件： ``` hive -e 'SELECT * FROM my_table' | sed 's/[\t]/,/g' > my_table.csv ``` 3. 使用sqoop将CSV文件导入MySQL。例如，可以使用以下命令将CSV文件导入MySQL中的一个表： ``` sqoop import --connect jdbc:mysql://localhost/my_database --username my_username --password my_password --table my_table --fields-terminated-by ',' --lines-terminated-by '\n' --input-null-string '\\N' --input-null-non-string '\\N' --delete-target-dir --target-dir /tmp/my_table --bindir /tmp/my_table/bin --outdir /tmp/my_table/out --input-fields-terminated-by ',' ``` 其中，--connect指定MySQL连接字符串，--username和--password指定MySQL用户名和密码，--table指定要导入的MySQL表名，--fields-terminated-by指定CSV文件中字段之间的分隔符，--lines-terminated-by指定CSV文件中行之间的分隔符，--input-null-string和--input-null-non-string指定CSV文件中的空值表示方式，--delete-target-dir指定在导入之前删除目标目录，--target-dir指定HDFS中的目标目录，--bindir、--outdir和--input-fields-terminated-by指定生成的Java类的位置和属性分隔符。 4. 确认数据已经成功导入MySQL中的表中。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

郝少

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值