Sqoop环境安装
Sqoop是Hadoop和关系数据库服务器之间传送数据的一种工具。它是用来从关系数据库如:MySQL,Oracle到Hadoop的HDFS,并从Hadoop的文件系统导出数据到关系数据库。
0. Sqoop源码包下载
http://ftp.wayne.edu/apache/sqoop/
1. 集群环境
Master 192.168.253.10
Slave1 192.168.253.11
Slave2 192.168.253.12
2. 下载安装包
Master
安装sqoop的前提是已经具备java和hadoop的环境
wget http://ftp.wayne.edu/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
3. 配置.bashrc
Master
必须设置Sqoop环境通过附加以下行到〜/.bashrc文件:
vim ~/.bashrc
# set Sqoop conf
export SQOOP_HOME=/home/master/tools/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin
source ~/.bashrc
4. 修改Sqoop配置文件
Master
cd $SQOOP_HOME/conf
mv sqoop-env-template.sh sqoop-env.sh
打开sqoop-env.sh并编辑下面几行:
export HADOOP_COMMON_HOME=/home/hadoop/apps/hadoop-2.6.1
export HADOOP_MAPRED_HOME=/home/hadoop/apps/hadoop-2.6.1
export HIVE_HOME=/home/hadoop/apps/hive-1.2.1
5. 加入mysql的jdbc驱动包、hive-comon包、hive-shims-*.jar
Master
cp $HIVE_HOME/lib/mysql-connector-java-5.1.41-bin.jar $SQOOP_HOME/lib
cp $HIVE_HOME/lib/hive-common-1.2.2.jar $SQOOP_HOME/lib
cp $HIVE_HOME/lib/hive-shims-*.jar $SQOOP_HOME/lib
6. 验证启动
Master
cd $SQOOP_HOME/bin
sqoop-version
预期的输出:
18/07/06 23:44:35 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
到这里,整个Sqoop安装工作完成!