sqoop2 1.99.3安装

最新推荐文章于 2022-03-26 03:23:11 发布

笔尖的痕

最新推荐文章于 2022-03-26 03:23:11 发布

阅读量2.1k

点赞数

分类专栏：记录 hadoop 文章标签：转载

hadoop 同时被 2 个专栏收录

90 篇文章 0 订阅

订阅专栏

记录

49 篇文章 0 订阅

订阅专栏

HADOOP环境为hadoop2.2.0

下载的sqoop包为sqoop-1.99.3-bin-hadoop200,下载地址：

1.解压文件到工作目录:

tar -xzvf sqoop-1.99.3-bin-hadoop200.tar.gz

mv sqoop-1.99.3-bin-hadoop200 /usr/app/sqoop

2.修改环境变量：

vim /etc/profile

增加SQOOP主目录

export SQOOP_HOME=/use/app/sqoop

export PATH=$PATH:$SQOOP_HOME/bin

export CATALINA_BASE=$SQOOP_HOME/server

export LOGDIR=$SQOOP_HOME/logs/

即时生效：

source /etc/profile

3.修改sqoop配置：

vi server/conf/sqoop.properties

将org.apache.sqoop.submission.engine.mapreduce.configuration.directory后面hadoop的位置修改为自己安装的hadoop配置文件位置，我的为:/usr/app/hadoop-2.2.0/etc/hadoop/

4.修改sqoop读取hadoop的jar包的路径：

vi /sqoop/server/conf/catalina.properties

将common.loader行后的/usr/lib/hadoop/lib/*.jar改成自己的hadoop jar 包目录，我的为：

/usr/app/hadoop-2.2.0/share/hadoop/common/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/common/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/hdfs/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/hdfs/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/mapreduce/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/mapreduce/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/tools/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/tools/lib/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/yarn/*.jar,

/usr/app/hadoop-2.2.0/share/hadoop/yarn/lib/*.jar

到此sqoop就基本配置完成可以直接运行./bin/sqoop.sh server start启动sqoop了

5.使用SQOOP：

启动SQOOP:

$./bin/sqoop.sh server start

停止SQOOP：

$./bin/sqoop.sh server stop

进入客户端交互模式:

$./bin/sqoop.sh client

Sqoop home directory: /usr/lib/sqoop

Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000>

为客户端配置服务器：

sqoop:000> set server --host master --port 12000 --webapp sqoop

Server is set successfully

查看版本信息：

sqoop:000> show version --all

client version:

Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

server version:

Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

Protocol version:

[1]

显示连接器：

sqoop:000> show connector --all

1 connector(s) to show:

Connector with id 1:

Name: generic-jdbc-connector

Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector

Supported job types: [EXPORT, IMPORT]

...

创建数据库连接

sqoop:000> create connection --cid 1

Creating connection for connector with id 1

Please fill following values to create new connection object

Name: Mysql-H216

Connection configuration

JDBC Driver Class: com.mysql.jdbc.Driver

JDBC Connection String: jdbc:mysql://192.168.1.120:3306/test

Username: admin

Password: *****

JDBC Connection Properties:

There are currently 0 values in the map:

entry#

Security related configuration options

Max connections: 100

New connection was successfully created with validation status FINE and persistent id 1

创建导入任务

sqoop:000> create job --xid 1 --type import

Creating job for connection with id 1

Please fill following values to create new job object

Name: HeartBeat

Database configuration

Schema name: mic_db_out

Table name: t_heart_beat

Table SQL statement:

Table column names:

Partition column name:

Nulls in partition column:

Boundary query:

Output configuration

Storage type:

0 : HDFS

Choose: 0

Output format:

0 : TEXT_FILE

1 : SEQUENCE_FILE

Choose: 0

Compression format:

0 : NONE

1 : DEFAULT

2 : DEFLATE

3 : GZIP

4 : BZIP2

5 : LZO

6 : LZ4

7 : SNAPPY

Choose: 0

Output directory: /user/jarcec/users

Throttling resources

Extractors:

Loaders:

New job was successfully created with validation status FINE and persistent id 1

sqoop:000>

查看导入状态：

sqoop:000> status job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: dev

Creation date: 2014-04-19 18:54:25 CST

Lastly updated by: dev

External ID: job_local1638775039_0002

2014-04-19 18:54:50 CST: UNKNOWN