sqoop1.99.3安装与教程

测试环境hadoop版本2.4.0，sqoop版本1.99.3，其他高版本 1.99.4、1.99.5均安装失败。

安装文档    http://sqoop.apache.org/docs/1.99.3/Installation.html

1.下载sqoop-1.99.3-bin-hadoop200.tar.gz

2.解压   tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz

3.配置文件/home/hadoop/sqoop/server/conf   

catalina.properties 

修改：（依赖jar）

 common.loader=common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar

sqoop.properties（设置hadoop配置文件路径）

修改

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/

4.添加数据库驱动包

mysql-connector-java-5.1.26-bin.jar --mysql

ojdbc6.jar --oracle

4.启动 /bin/sqoop.sh server start     

   停止 ./bin/sqoop.sh server stop

5.添加服务端

set server --host 132.77.210.201 --port 12000 --webapp sqoop

6.验证：

 -> ./sqoop.sh client

截图

7. 常用命令：  注意 --jid 表示 job ID   --xid  表示 连接ID

create connection --cid 1 --创建连接

delete connection --xid 1 

update connection --xid 1

create job --xid 1 --type import 创建job 从数据库导入hdfs     --xid 1 表示连接id

create job --xid 1 --type export  创建job  从hdfs导入数据库

status job --jid 2   --查看job状态

start job --jid 2   --执行job

update job -jid 2 --修改job

show job --显示job

delete job --jid 4 --删除job  --jid 4 要删除的job id

8.链接oracle:

sqoop:000>  create connection --cid 1

Creating connection for connector with id 1

Please fill following values to create new connection object

Name:  wgh_test

Connection configuration

JDBC Driver Class:   oracle.jdbc.driver.OracleDriver

JDBC Connection String:    jdbc:oracle:thin:@132.77.119.185:1528:wghdbts1

Username:  wgh

Password:  ***

JDBC Connection Properties:

There are currently 0 values in the map:

entry#

Security related configuration options

Max connections:   10

New connection was successfully created with validation status FINE and persistent id 2

9.修改jop/新建  

sqoop:000> update job --jid 1      --修改      -- create job --xid 1 --type import --创建 从数据库导入hdfs

Updating job with id 1

Please update job metadata:

Name:   first_job

Database configuration

Schema name:   wgh    --数据库用户名

Table name:   eval_template

Table SQL statement:

Table column names: EVAL_TEMPLATE_ID,TEMPLATE_NAME  --字段

Partition column name:                

Nulls in partition column:

Boundary query:

Output configuration

Storage type:

  0 : HDFS

Choose:   0

Output format:

  0 : TEXT_FILE

  1 : SEQUENCE_FILE

Choose:   0

Compression format:

  0 : NONE

  1 : DEFAULT

  2 : DEFLATE

  3 : GZIP

  4 : BZIP2

  5 : LZO

  6 : LZ4

  7 : SNAPPY

Choose:  0

Output directory:   /test --hdfs文件目录

Throttling resources

Extractors:

Loaders:

Job was successfully updated with status FINE

--创建导入job,从hdfs导入oracle

sqoop:000> create job --xid 1 --type export

Creating job for connection with id 1

Please fill following values to create new job object

Name:   export_eval

Database configuration

Schema name:   wgh

Table name:   eval_template_001

Table SQL statement:

Table column names:   EVAL_TEMPLATE_ID,TEMPLATE_NAME

Stage table name:

Clear stage table:

Input configuration

Input directory:  /test2

Throttling resources

Extractors:

Loaders:

New job was successfully created with validation status FINE  and persistent id 4

sqoop:000> show job        

+----+--------------------------+--------+-----------+---------+

| Id |           Name           |  Type  | Connector | Enabled |

+----+--------------------------+--------+-----------+---------+

| 1  | first_job                | IMPORT | 1         | true    |

| 2  | export_vip_customer_info | EXPORT | 1         | true    |

| 4  | export_eval              | EXPORT | 1         | true    |

+----+--------------------------+--------+-----------+---------+

sqoop:000> 

日志目录：/home/hadoop/sqoop/@LOGDIR@

server启动日志目录：/home/hadoop/sqoop/server/logs