sqoop(4)—— Mysql数据导入Hive
-
官方文档
http://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_importing_data_into_hive_2
-
示例
-
集群和版本规划
-
hadoop-2.6.5
sqoop-1.4.7.bin__hadoop-2.6.0
zookeeper-3.4.10
apache-hive-1.2.2-bin
faith-Fedora | faith-Ubuntu | faith-openSUSE | faith-Kylin | faith-Mint | faith-Fedora2 | ||
HDFS/YARN | NN1(ZKFC1) | ✔ | |||||
NN2(ZKFC2) | ✔ | ||||||
JN1 | ✔ | ||||||
JN2 | ✔ | ||||||
JN3 | ✔ | ||||||
DN1(NM1) | ✔ | ||||||
DN2(NM2) | ✔ | ||||||
DN3(NM3) | ✔ | ||||||
RS1 | ✔ | ||||||
RS2 | ✔ | ||||||
ZOOKEEPER | ZK1 | ✔ | |||||
ZK2 | ✔ | ||||||
ZK3 | ✔ | ||||||
SQOOP | ✔ | ||||||
MYSQL | ✔ | ||||||
HIVE | Metastore | ✔ | |||||
Hiveserver2 | ✔ |
-
启动集群
-
启动ZK
-
启动HDFS
-
非生产环境下,让HDFS快速离开安全模式
-
启动YARN
-
启动Hive的metastore服务和hiveservice2服务
这两个服务开启一个就可以。
-
启动metastore服务
-
启动hiveservice2服务
-
启动MR、YARN、SPARK的历史日志服务
-
启动MR的HistoryServer
-
-
启动YARN的timelineserver
-
启动SPARK的HistoryServer
-
启动mysql
-
SQOOP导入脚本
import
--connect
jdbc:mysql://faith-Fedora2:3306/test
--username
root
--password
--as-textfile
--query
'SELECT id, name, msg from psn WHERE id LIKE "1%" AND $CONDITIONS'
--delete-target-dir
--target-dir
/sqoop/tmp
-m
1
--hive-home
/home/faith/Repository/Programs/apache-hive-1.2.2-bin
--hive-import
--create-hive-table
--hive-table
t_test