cdh jar包 sqoop2_sqoop2 安装部署、源码编译

1.安装准备工作：

已经装好的 hadoop 环境是cdh版本 hadoop-2.6.0 64位

下载的sqoop安装包 (不知道这个安装包支不支持64位，所以我下载的源码，然后自己编译源码)

http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.5.1.tar.gz

源码下载地址：

http://archive.cloudera.com/cdh5/cdh/5/sqoop2-1.99.5-cdh5.5.1-src.tar.gz

编译源码

编译环境与hadoop的编译环境一致，详细配置请查看《Hadoop学习笔记 6 Hadoop源码编译》

maven编译命令：

mvn clean package -Pbinary -DskipTests

如果报内存溢出：

set MAVEN_OPTS=-XX:MaxPermSize=128M

编译好的安装包在 sqoop2-1.99.5-cdh5.5.1/dist/target/

2.解压文件到工作目录:

tar -xzvf sqoop-1.99.5-bin-hadoop200.tar.gz -C /usr/hadoop

3.修改环境变量：

vim /etc/profile

添加如下内容：

#sqoop

export SQOOP_HOME=/usr/hadoop/sqoop-1.99.5-bin-hadoop200

export PATH=SQOOPHOME/bin:PATH

export CATALINA_HOME=SQOOPHOME/server

LOGDIR=SQOOP_HOME/logs

保存退出即时生效：

source /etc/profile

4.修改sqoop配置：

vim /usr/hadoop/sqoop-1.99.5-bin-hadoop200/server/conf/sqoop.properties

#修改指向我的hadoop安装目录

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hadoop/hadoop-2.6.0-cdh5.5.1

#把hadoop目录下的jar包都引进来

vim /usr/hadoop/sqoop-1.99.4-bin-hadoop200/server/conf/catalina.properties

common.loader=/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/common/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/common/lib/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/hdfs/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/hdfs/lib/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/mapreduce/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/mapreduce/lib/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/tools/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/tools/lib/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/yarn/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/yarn/lib/*.jar,

/usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/httpfs/tomcat/lib/*.jar

或者

在$SQOOP_HOME中建个文件夹例如hadoop_lib，然后将这些jar包cp到此文件夹中，最后将此文件夹路径添加到common.loader属性

中，这种方法更加直观些

5.下载mysql驱动包

mysql-connector-java-5.1.32-bin.jar

并放到 /usr/hadoop/hadoop-2.6.0-cdh5.5.1/share/hadoop/httpfs/tomcat/lib/ 目录下

6.启动/停止sqoop2

/usr/hadoop/sqoop-1.99.5-bin-hadoop200/bin/sqoop.sh server start/stop

查看启动日志：

tail -500 /usr/hadoop/sqoop-1.99.5-bin-hadoop200/server/logs/catalina.out

7.进入客户端交互目录

/usr/hadoop/sqoop-1.99.5-bin-hadoop200/bin/sqoop.sh client

分享到：

2016-01-06 11:22

浏览 1518

评论