cdh sqoop 配置_Sqoop 1.4.6 安装配置关联Hadoop Hive Zookeeper

系统环境

组件版本

nameversion

Sqoop

1.4.6

Sqoop安装

解压

懒的敲了

配置环境变量

# SQOOP

SQOOP_HOME=/home/Hadoop/development/src/sqoop-1.4.6-cdh5.6.0

PATH=$PATH:$SQOOP_HOME/bin

export PATH

cd 到Sqoop根目录

配置Sqoop核心文件

主要配置${SQOOP_HOME}/conf/sqoop-env-template.sh 文件

cp ${SQOOP_HOME}/conf/sqoop-env-template.sh ${SQOOP_HOME}/conf/sqoop-env.sh

主要修改参数

# 指定 Hadoop 安装目录

export HADOOP_COMMON_HOME=

# 指定 Hadoop MapReduce 技术jar存放目录

export HADOOP_MAPRED_HOME=

# Hbase 安装目录

export HBASE_HOME=

# Zookeeper 安装目录

export ZOOCFGDIR=

如果没有安装的可以不用填写

例如

#Set path to where bin/hadoop is available

export HADOOP_COMMON_HOME=/home/hadoop/development/src/hadoop-2.6.0-cdh5.6.0

#Set path to where hadoop-*-core.jar is available

"这里之所以和Hadoop安装目录一样，是由于上次我配置成mapreduce jar地址时,sqoop报错"

export HADOOP_MAPRED_HOME=/home/hadoop/development/src/hadoop-2.6.0-cdh5.6.0

#set the path to where bin/hbase is available

"我没有安装Hbase所以这里直接 注释掉就可以"

#export HBASE_HOME=

#Set the path to where bin/hive is available

export HIVE_HOME=/home/hadoop/development/src/hive-1.1.0-cdh5.6.0

#Set the path for where zookeper config dir is

export ZOOCFGDIR=/home/hadoop/development/src/zookeeper-3.4.5-cdh5.6.0

测试Sqoop是否配置成功

sqoop list-databases --connect jdbc:mysql://192.168.2.251:3306/canbot --username root --password root

Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../hbase does not exist! HBase imports will fail.

Please set $HBASE_HOME to the root of your HBase installation.

Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0/bin/../../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

16/05/23 10:34:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.6.0

16/05/23 10:34:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

16/05/23 10:34:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

"information_schema

canbot

from_66internet

from_fhmysql

hive

mysql

Sqoop 数据导入到Hive中报错 HIVE_CONF_DIR

解决方法：export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*

相关阅读：

Hadoop生态系统搭建(hadoop hive hbase zookeeper oozie Sqoop) http://www.linuxidc.com/Linux/2012-03/55721.htm

Sqoop 的详细介绍：请点这里

Sqoop 的下载地址：请点这里