sqoop安装配置

sqoop安装配置
sqoop是一个把关系型数据库数据抽向hadoop的工具。同时，也支持将hive、pig等查询的结果导入关系型数据库中存储。
由于，笔者部署的hadoop版本是2.2.0，所以sqoop的版本是：sqoop-1.99.6-bin-hadoop200.tar.gz
1. 下载sqoop  
http://www.apache.org/dyn/closer.lua/sqoop/
2. 解压sqoop安装文件 
[node0 Documents]# tar -zxvf sqoop-1.99.6-bin-hadoop200.tar.gz  -C /usr/local
[root@node0 local]# ln -s /usr/local/sqoop-1.99.6-bin-hadoop200/  sqoop
3. 配置sqoop环境变量
修改/etc/profile和.bash_profile文件，添加Hadoop_Home,调整PATH
export JAVA_HOME=/usr/lib/jvm/java
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME=/root/hadoop/hadoop-2.6.4
export HBASE_HOME=/usr/local/hbase
export SQOOP_HOME=/usr/local/sqoop
export PATH=${JAVA_HOME}/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$HBASE_HOME/conf:$SQOOP_HOME/bin:$PATH
export CATALINA_BASE=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs/

/etc/profile和/root/.bash_profile的区别
/etc/profile是全局的，/root/.bash_profile是私有的
/etc/profile用于整个系统所有用户， ~/.bash_profile, ~/.profile和~/.bashrc 用于各个用户，这里的"~"符号就是各当前用户的$HOME
~/.bash_profile 和 ~/.profile 只在登陆时读取一次。
~/.bashrc 每次都读取
~/.bash_profile 和 ~/.

4. 重命名配置文件sqoop-site.xml
在${SQOOP_HOME}/conf中执行命令
mv  sqoop-env-template.sh  sqoop-env.sh
在conf目录下，有两个文件sqoop-site.xml和sqoop-site-template.xml内容是完全一样的，不必在意，我们只关心sqoop-site.xml即可。

5. 修改配置文件sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/root/hadoop/hadoop-2.6.4/
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/root/hadoop/hadoop-2.6.4
#set the path to where bin/hbase is available
export HBASE_HOME=/usr/local/hbase
#Set the path to where bin/hive is available
export HIVE_HOME=/usr/local/hive
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zookeeper

修改sqoop——server关于hadoop共享jar的引用
vi server/conf/catalina.properties
找到common.loader行，把/usr/lib/hadoop/lib/*.jar改成你的hadoop jar 包目录

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar,/usr/lib/hive/lib/*.jar
修改为：
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/root/hadoop/hadoop-2.6.4/*.jar,/root/hadoop/hadoop-2.6.4/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/hdfs/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/hdfs/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/mapreduce/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/mapreduce/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/yarn/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/yarn/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/common/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/common/lib/*.jar,/usr/local/hive/lib/*.jar

vi server/conf/sqoop.properties
找到：mapreduce.configuration.directory行，修改值为你的hadoop配置文件目录
如：org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
替换成
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/root/hadoop/hadoop-2.6.4/etc/hadoop/

并且替换@LOGDIR@ 和@BASEDIR@ ：
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g

org.apache.sqoop.log4j.appender.file.File=@LOGDIR@/sqoop.log
替换成
org.apache.sqoop.log4j.appender.file.File=0,$ s/@LOGDIR@/logs/g/sqoop.log

org.apache.sqoop.auditlogger.default.file=@LOGDIR@/default.audit
替换成
org.apache.sqoop.auditlogger.default.file=0,$ s/@LOGDIR@/logs/g/default.audit

org.apache.sqoop.repository.sysprop.derby.stream.error.file=@LOGDIR@/derbyrepo.log
替换成
org.apache.sqoop.repository.sysprop.derby.stream.error.file=0,$ s/@LOGDIR@/logs/g/derbyrepo.log

org.apache.sqoop.log4j.appender.file.File=@LOGDIR@/sqoop.log
替换成
org.apache.sqoop.log4j.appender.file.File=0,$ s/@LOGDIR@/logs/g/sqoop.log

org.apache.sqoop.repository.jdbc.url=jdbc:derby:@BASEDIR@/repository/db;create=true
替换成
org.apache.sqoop.repository.jdbc.url=jdbc:derby: 0,$ s/@BASEDIR@/base/g/repository/db;create=true

6. 启动服务
启动 
[root@node0 sqoop]#  ./bin/sqoop.sh server start
Sqoop home directory: /usr/local/sqoop
Setting SQOOP_HTTP_PORT:     12000
Setting SQOOP_ADMIN_PORT:     12001
Using   CATALINA_OPTS:       
Adding to CATALINA_OPTS:    -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE:   /usr/local/sqoop/server
Using CATALINA_HOME:   /usr/local/sqoop/server
Using CATALINA_TMPDIR: /usr/local/sqoop/server/temp
Using JRE_HOME:        /usr/lib/jvm/java/jre
Using CLASSPATH:       /usr/local/sqoop/server/bin/bootstrap.jar

关闭 sqoop server : ./bin/sqoop.sh server stop
使用sqoop客户端：
[root@db12c sqoop]# bin/sqoop.sh client
Sqoop home directory: /home/likehua/sqoop/sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> show version --all#显示版本：show version –all
显示连接器：show connector --all创建连接：create connection --cid 1
client version:
  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
server version:
  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
Protocol version:
  [1]
sqoop:000>
标签: hadoop

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/29050044/viewspace-2127428/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/29050044/viewspace-2127428/