Sqoop-1.99.7安装部署

Sqoop安装部署

官方文档：http://sqoop.apache.org/

下载地址

[hadoop@slavenode8 hadoop]$ wget http://apache.fayea.com/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz 

设置环境变量

[hadoop@slavenode8 sqoop-1.99.7]$ vi ~/.bash_profile 

export SQOOP_HOME=/opt/hadoop/sqoop-1.99.7

export PATH=$PATH:$SQOOP_HOME/bin

export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extra

export CATALINA_BASE=$SQOOP_HOME/server

export LOGDIR=$SQOOP_HOME/logs/

 

安装sqoop

[hadoop@slavenode8 hadoop]$ tar -zxf sqoop-1.99.7-bin-hadoop200.tar.gz 

[hadoop@slavenode8 hadoop]$ mv sqoop-1.99.7-bin-hadoop200 sqoop-1.99.7

[hadoop@slavenode8 hadoop]$ cd sqoop-1.99.7

配置参数文件

[hadoop@slavenode2 ~]$ vi /opt/hadoop/hadoop-2.7.2/etc/hadoop/container-executor.cfg 

yarn.nodemanager.linux-container-executor.group=#configured value of yarn.nodemanager.linux-container-executor.group

banned.users=#comma separated list of users who can not run applications

min.user.id=1000#Prevent other super-users

allowed.system.users=hadoop

vi core-site.xml

<property>

       <name>hadoop.proxyuser.hadoop.hosts</name>

          <value>*</value>

       </property>

        <property>

        <name>hadoop.proxyuser.hadoop.groups</name> 运行sqoop的用户名

          <value>*</value>

         </property>

 

 

[hadoop@masternode2 sqoop-1.99.7]$ vi conf/sqoop.properties 

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop/hadoop-2.7.2/etc/hado

op

org.apache.sqoop.security.authentication.type=SIMPLE

org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler

org.apache.sqoop.security.authentication.anonymous=true

并且替换@LOGDIR@ 和@BASEDIR@ ：
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g

 

创建目录

mkdir /opt/hadoop/sqoop-1.99.7/extra

mkdir /opt/hadoop/sqoop-1.99.7/logs

mysql的驱动jar文件复制到这个目录下。 

cp mysql-connector-java-5.1.36-bin.jar /opt/hadoop/sqoop-1.99.7/extra

 

验证配置是否有效

使用bin中的sqoop2-tool工具进行验证：

 

[hadoop@masternode2 sqoop-1.99.7]$ bin/sqoop2-tool verify 

Setting conf dir: /opt/hadoop/sqoop-1.99.7/bin/../conf

Sqoop home directory: /opt/hadoop/sqoop-1.99.7

Sqoop tool executor:

        Version: 1.99.7

        Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb

        Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine

Running tool: class org.apache.sqoop.tools.tool.VerifyTool

0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.

7    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread

Verification was successful.

Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

 

开启服务器：

[hadoop@masternode2 sqoop-1.99.7]$ bin/sqoop2-server start

Setting conf dir: /opt/hadoop/sqoop-1.99.7/bin/../conf

Sqoop home directory: /opt/hadoop/sqoop-1.99.7

Starting the Sqoop2 server...

0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.

7    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread

Sqoop2 server started.