Sqoop导入mysql数据到HDFS

Sqoop导入mysql数据到HDFS

打算把mysql中爬虫爬到的数据导入到HDFS中供后续学习kylin以及研究使用，以前导数据经常会自己写一段java脚本去导入，这次打算用sqoop试一试效果

首先看了下mysql的数据量，不到900万条数据，不多不少

从官网下载好sqoop进行安装

http://sqoop.apache.org/docs/1.99.7/admin/Installation.html
中间也经历了一点波折，一个是hadoop的conf文件目录要配置正确
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/apps/hadoop-2.7.7/etc/hadoop
将mysql connector jar包丢到/apps/sqoop-1.99.7-bin-hadoop200/server/lib目录

启动sqoop

root@hadoop:/apps/sqoop-1.99.7-bin-hadoop200/bin# ./sqoop2-server start
Setting conf dir: /apps/sqoop-1.99.7-bin-hadoop200/bin/../conf
Sqoop home directory: /apps/sqoop-1.99.7-bin-hadoop200
Starting the Sqoop2 server...
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
15   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Sqoop2 server started.

使用sqoopshell