sqoop1.4.6相关操作及问题汇总

1 sqoop环境准备
sqoop官网推荐使用稳定的1.4.6,之前在csdn误操作把之前写的删除掉了，现在重新补上
1.1 安装

tar -zxvf sqoop-1.4.6.tar.gz
ln -s sqoop-1.4.6 sqoop

文件可以从sqoop-1.4.6.tar.gz下载
1.2 环境变量

export SQOOP_HOME=/home/hadoop/application/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}

1.3 更改$SQOOP_HOME/sqoop
将

exec ${HADOOP_COMMON_HOME}/bin/hadoop  org.apache.sqoop.Sqoop "$@"

更改为

exec ${HADOOP_COMMON_HOME}/bin/hadoop jar $SQOOP_HOME/sqoop-1.4.6.jar org.apache.sqoop.Sqoop "$@"

其中sqoop-1.4.6.jar可以在sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz中找到
1.4 注释$SQOOP_HOME/configure-sqoop中部分

#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi

1.5 将mysql-connector-java-5.1.43.jar放入到$SQOOP_HOME/lib
mysql-connector-java-5.1.43.jar可以从maven仓库中获取
2 sqoop 导入数据到mysql
参考使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟

sqoop import \
> --connect jdbc:mysql://192.168.5.135:3306/toolbox --username root -P \
> --query "select CONCAT(id,'_',xhdwsbh,'_',kprq) row_key,tit.* from t_invoice_ticket tit where kprq>='20170401' and kprq<='20170430' and \$CONDITIONS " \
> --split-by xhdwsbh -m 5 \
> --hbase-table iw:test01 \
> --column-family i \
> --hbase-row-key row_key

参数说明关于sqoop --split-by 及 -m的理解