1 sqoop环境准备
sqoop官网推荐使用稳定的1.4.6,之前在csdn误操作把之前写的删除掉了,现在重新补上
1.1 安装
tar -zxvf sqoop-1.4.6.tar.gz
ln -s sqoop-1.4.6 sqoop
文件可以从sqoop-1.4.6.tar.gz下载
1.2 环境变量
export SQOOP_HOME=/home/hadoop/application/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
1.3 更改$SQOOP_HOME/sqoop
将
exec ${HADOOP_COMMON_HOME}/bin/hadoop org.apache.sqoop.Sqoop "$@"
更改为
exec ${HADOOP_COMMON_HOME}/bin/hadoop jar $SQOOP_HOME/sqoop-1.4.6.jar org.apache.sqoop.Sqoop "$@"
其中sqoop-1.4.6.jar
可以在sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz中找到
1.4 注释$SQOOP_HOME/configure-sqoop中部分
#if [ ! -d "${HCAT_HOME}" ]; then
# echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
# echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi
#if [ ! -d "${ACCUMULO_HOME}" ]; then
# echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
# echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi
1.5 将mysql-connector-java-5.1.43.jar放入到$SQOOP_HOME/lib
mysql-connector-java-5.1.43.jar可以从maven仓库中获取
2 sqoop 导入数据到mysql
参考使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟
sqoop import \
> --connect jdbc:mysql://192.168.5.135:3306/toolbox --username root -P \
> --query "select CONCAT(id,'_',xhdwsbh,'_',kprq) row_key,tit.* from t_invoice_ticket tit where kprq>='20170401' and kprq<='20170430' and \$CONDITIONS " \
> --split-by xhdwsbh -m 5 \
> --hbase-table iw:test01 \
> --column-family i \
> --hbase-row-key row_key