sqoop安装配置
sqoop是一个把关系型数据库数据抽向hadoop的工具。同时,也支持将hive、pig等查询的结果导入关系型数据库中存储。
由于,笔者部署的hadoop版本是2.2.0,所以sqoop的版本是:sqoop-1.99.6-bin-hadoop200.tar.gz
1. 下载sqoop
http://www.apache.org/dyn/closer.lua/sqoop/
2. 解压sqoop安装文件
[node0 Documents]# tar -zxvf sqoop-1.99.6-bin-hadoop200.tar.gz -C /usr/local
[root@node0 local]# ln -s /usr/local/sqoop-1.99.6-bin-hadoop200/ sqoop
3. 配置sqoop环境变量
修改/etc/profile和.bash_profile文件,添加Hadoop_Home,调整PATH
export JAVA_HOME=/usr/lib/jvm/java
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME=/root/hadoop/hadoop-2.6.4
export HBASE_HOME=/usr/local/hbase
export SQOOP_HOME=/usr/local/sqoop
export PATH=${JAVA_HOME}/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$HBASE_HOME/conf:$SQOOP_HOME/bin:$PATH
export CATALINA_BASE=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs/
/etc/profile和/root/.bash_profile的区别
/etc/profile是全局的,/root/.bash_profile是私有的
/etc/profile用于整个系统所有用户, ~/.bash_profile, ~/.profile和~/.bashrc 用于各个用户,这里的"~"符号就是各当前用户的$HOME
~/.bash_profile 和 ~/.profile 只在登陆时读取一次。
~/.bashrc 每次都读取
~/.bash_profile 和 ~/.
4. 重命名配置文件sqoop-site.xml
在${SQOOP_HOME}/conf中执行命令
mv sqoop-env-template.sh sqoop-env.sh
在conf目录下,有两个文件sqoop-site.xml和sqoop-site-template.xml内容是完全一样的,不必在意,我们只关心sqoop-site.xml即可。
5. 修改配置文件sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/root/hadoop/hadoop-2.6.4/
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/root/hadoop/hadoop-2.6.4
#set the path to where bin/hbase is available
export HBASE_HOME=/usr/local/hbase
#Set the path to where bin/hive is available
export HIVE_HOME=/usr/local/hive
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zookeeper
修改sqoop——server关于hadoop共享jar的引用
vi server/conf/catalina.properties
找到common.loader行,把/usr/lib/hadoop/lib/*.jar改成你的hadoop jar 包目录
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar,/usr/lib/hive/lib/*.jar
修改为:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/root/hadoop/hadoop-2.6.4/*.jar,/root/hadoop/hadoop-2.6.4/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/hdfs/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/hdfs/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/mapreduce/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/mapreduce/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/yarn/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/yarn/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/common/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/common/lib/*.jar,/usr/local/hive/lib/*.jar
vi server/conf/sqoop.properties
找到:mapreduce.configuration.directory行,修改值为你的hadoop配置文件目录
如:org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
替换成
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/root/hadoop/hadoop-2.6.4/etc/hadoop/
并且替换@LOGDIR@ 和@BASEDIR@ :
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g
org.apache.sqoop.log4j.appender.file.File=@LOGDIR@/sqoop.log
替换成
org.apache.sqoop.log4j.appender.file.File=0,$ s/@LOGDIR@/logs/g/sqoop.log
org.apache.sqoop.auditlogger.default.file=@LOGDIR@/default.audit
替换成
org.apache.sqoop.auditlogger.default.file=0,$ s/@LOGDIR@/logs/g/default.audit
org.apache.sqoop.repository.sysprop.derby.stream.error.file=@LOGDIR@/derbyrepo.log
替换成
org.apache.sqoop.repository.sysprop.derby.stream.error.file=0,$ s/@LOGDIR@/logs/g/derbyrepo.log
org.apache.sqoop.log4j.appender.file.File=@LOGDIR@/sqoop.log
替换成
org.apache.sqoop.log4j.appender.file.File=0,$ s/@LOGDIR@/logs/g/sqoop.log
org.apache.sqoop.repository.jdbc.url=jdbc:derby:@BASEDIR@/repository/db;create=true
替换成
org.apache.sqoop.repository.jdbc.url=jdbc:derby: 0,$ s/@BASEDIR@/base/g/repository/db;create=true
6. 启动服务
启动
[root@node0 sqoop]# ./bin/sqoop.sh server start
Sqoop home directory: /usr/local/sqoop
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE: /usr/local/sqoop/server
Using CATALINA_HOME: /usr/local/sqoop/server
Using CATALINA_TMPDIR: /usr/local/sqoop/server/temp
Using JRE_HOME: /usr/lib/jvm/java/jre
Using CLASSPATH: /usr/local/sqoop/server/bin/bootstrap.jar
关闭 sqoop server : ./bin/sqoop.sh server stop
使用sqoop客户端:
[root@db12c sqoop]# bin/sqoop.sh client
Sqoop home directory: /home/likehua/sqoop/sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> show version --all#显示版本:show version –all
显示连接器:show connector --all创建连接:create connection --cid 1
client version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
server version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
Protocol version:
[1]
sqoop:000>
标签: hadoop
sqoop是一个把关系型数据库数据抽向hadoop的工具。同时,也支持将hive、pig等查询的结果导入关系型数据库中存储。
由于,笔者部署的hadoop版本是2.2.0,所以sqoop的版本是:sqoop-1.99.6-bin-hadoop200.tar.gz
1. 下载sqoop
http://www.apache.org/dyn/closer.lua/sqoop/
2. 解压sqoop安装文件
[node0 Documents]# tar -zxvf sqoop-1.99.6-bin-hadoop200.tar.gz -C /usr/local
[root@node0 local]# ln -s /usr/local/sqoop-1.99.6-bin-hadoop200/ sqoop
3. 配置sqoop环境变量
修改/etc/profile和.bash_profile文件,添加Hadoop_Home,调整PATH
export JAVA_HOME=/usr/lib/jvm/java
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME=/root/hadoop/hadoop-2.6.4
export HBASE_HOME=/usr/local/hbase
export SQOOP_HOME=/usr/local/sqoop
export PATH=${JAVA_HOME}/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$HBASE_HOME/conf:$SQOOP_HOME/bin:$PATH
export CATALINA_BASE=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs/
/etc/profile和/root/.bash_profile的区别
/etc/profile是全局的,/root/.bash_profile是私有的
/etc/profile用于整个系统所有用户, ~/.bash_profile, ~/.profile和~/.bashrc 用于各个用户,这里的"~"符号就是各当前用户的$HOME
~/.bash_profile 和 ~/.profile 只在登陆时读取一次。
~/.bashrc 每次都读取
~/.bash_profile 和 ~/.
4. 重命名配置文件sqoop-site.xml
在${SQOOP_HOME}/conf中执行命令
mv sqoop-env-template.sh sqoop-env.sh
在conf目录下,有两个文件sqoop-site.xml和sqoop-site-template.xml内容是完全一样的,不必在意,我们只关心sqoop-site.xml即可。
5. 修改配置文件sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/root/hadoop/hadoop-2.6.4/
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/root/hadoop/hadoop-2.6.4
#set the path to where bin/hbase is available
export HBASE_HOME=/usr/local/hbase
#Set the path to where bin/hive is available
export HIVE_HOME=/usr/local/hive
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zookeeper
修改sqoop——server关于hadoop共享jar的引用
vi server/conf/catalina.properties
找到common.loader行,把/usr/lib/hadoop/lib/*.jar改成你的hadoop jar 包目录
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar,/usr/lib/hive/lib/*.jar
修改为:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/root/hadoop/hadoop-2.6.4/*.jar,/root/hadoop/hadoop-2.6.4/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/hdfs/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/hdfs/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/mapreduce/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/mapreduce/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/yarn/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/yarn/lib/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/common/*.jar,/root/hadoop/hadoop-2.6.4/share/hadoop/common/lib/*.jar,/usr/local/hive/lib/*.jar
vi server/conf/sqoop.properties
找到:mapreduce.configuration.directory行,修改值为你的hadoop配置文件目录
如:org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
替换成
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/root/hadoop/hadoop-2.6.4/etc/hadoop/
并且替换@LOGDIR@ 和@BASEDIR@ :
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g
org.apache.sqoop.log4j.appender.file.File=@LOGDIR@/sqoop.log
替换成
org.apache.sqoop.log4j.appender.file.File=0,$ s/@LOGDIR@/logs/g/sqoop.log
org.apache.sqoop.auditlogger.default.file=@LOGDIR@/default.audit
替换成
org.apache.sqoop.auditlogger.default.file=0,$ s/@LOGDIR@/logs/g/default.audit
org.apache.sqoop.repository.sysprop.derby.stream.error.file=@LOGDIR@/derbyrepo.log
替换成
org.apache.sqoop.repository.sysprop.derby.stream.error.file=0,$ s/@LOGDIR@/logs/g/derbyrepo.log
org.apache.sqoop.log4j.appender.file.File=@LOGDIR@/sqoop.log
替换成
org.apache.sqoop.log4j.appender.file.File=0,$ s/@LOGDIR@/logs/g/sqoop.log
org.apache.sqoop.repository.jdbc.url=jdbc:derby:@BASEDIR@/repository/db;create=true
替换成
org.apache.sqoop.repository.jdbc.url=jdbc:derby: 0,$ s/@BASEDIR@/base/g/repository/db;create=true
6. 启动服务
启动
[root@node0 sqoop]# ./bin/sqoop.sh server start
Sqoop home directory: /usr/local/sqoop
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE: /usr/local/sqoop/server
Using CATALINA_HOME: /usr/local/sqoop/server
Using CATALINA_TMPDIR: /usr/local/sqoop/server/temp
Using JRE_HOME: /usr/lib/jvm/java/jre
Using CLASSPATH: /usr/local/sqoop/server/bin/bootstrap.jar
关闭 sqoop server : ./bin/sqoop.sh server stop
使用sqoop客户端:
[root@db12c sqoop]# bin/sqoop.sh client
Sqoop home directory: /home/likehua/sqoop/sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> show version --all#显示版本:show version –all
显示连接器:show connector --all创建连接:create connection --cid 1
client version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
server version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
Protocol version:
[1]
sqoop:000>
标签: hadoop
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/29050044/viewspace-2100927/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/29050044/viewspace-2100927/