摘要:超详细讲解Sqoop2部署过程,1.下载Sqoop2,增加Sqoop2变量,修改Sqoop2配置文件,默认已经安装好了hadoop。另外提供Sqoop2自动部署源码
1.下载解压sqoop2
tar -zxf sqoop-1.99.7-bin-hadoop200.tar.gz
sudo mv sqoop-1.99.7-bin-hadoop200 /opt/
2.配置变量
2.1.设置hadoop环境变量
sudo vim /etc/profile.d/hadoop-2.7.3
#!/bin/sh
# Author:wangxiaolei 王小雷
# Blog: http://blog.csdn.net/dream_an
# Github: https://github.com/wangxiaoleiai
# Date: 20170729
# Path: /etc/profile.d/
# Organization: https://github.com/whaleai
export HADOOP_HOME=/opt/hadoop-2.7.3
export PATH=$HADOOP_HOME/bin:$PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
2.2.设置sqoop2环境变量
sudo vim /etc/profile.d/sqoop-1.99.7-bin-hadoop200.sh
# Author:wangxiaolei 王小雷
# Blog: http://blog.csdn.net/dream_an
# Github: https://github.com/wangxiaoleiai
# Date: 201707
# Organization: https://github.com/whaleai
export SQOOP_HOME=/opt/sqoop-1.99.7-bin-hadoop200
export PATH=$SQOOP_HOME/bin:$PATH
eexport SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/lib
2.3.修改文件sqoop.properties
xiaolei@wang:~$ vim /opt/sqoop-1.99.7-bin-hadoop200/conf/sqoop.properties
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop-2.7.3/etc/hadoop
2.4.修改hadoop的mapred-site.xml文件
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
</property>
3.sqoop2元初始化和安装验证
3.1.元数据初始化
# 进入到sqoop目录是因为初始化会生成@BASEDIR@和@LOGDIR@
xiaolei@wang:~$ cd /opt/sqoop-1.99.7-bin-hadoop200/
# 初始化
xiaolei@wang:/opt/sqoop-1.99.7-bin-hadoop200$ sqoop2-tool upgrade
3.2.验证sqoop2-tool verify
xiaolei@wang:/opt/sqoop-1.99.7-bin-hadoop200$ sqoop2-tool verify
3.3.开启server
xiaolei@wang:/opt/sqoop-1.99.7-bin-hadoop200$ sqoop2-server start
3.4.通过jps可以看到SqoopJettyServery
已经启动
6026 SqoopJettyServer
6077 Jps
3.5. 如何应用超详细讲解Sqoop2应用与实践
4.关闭Sqoop2
sqoop2-server stop
5.添加第三方jdbc
mkdir /opt/sqoop-1.99.7/lib
wget http://central.maven.org/maven2/org/postgresql/postgresql/42.1.1/postgresql-42.1.1.jar
mv postgresql-42.1.1.jar /opt/sqoop-1.99.7/lib/
添加环境变量
export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/sqoop-1.99.7/lib