1.问题描述
在CDH集群中我们需要将Hive表的数据导入到RDBMS数据库中,使用Sqoop工具可以方便的将Hive表数据抽取到RDBMS数据库中,在使用Sqoop抽取Hive Parquet表时作业执行异常。
Sqoop抽数脚本:
sqoop export \
--connect jdbc:mysql://localhost:3306/test_db \
--username root \
--password 123456 \
--table mytest_parquet \
--export-dir /user/hive/warehouse/mytest_parquet \
-m 1
异常日志:
17/12/17 01:18:52 ERROR sqoop.Sqoop: Got exception running Sqoop: org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist: hdfs://nameservice1/user/hive/warehouse/mytest_parquet/.metadata org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist: hdfs://nameservice1/user/hive/warehouse/mytest_parquet/.metadata at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.checkExists(FileSystemMetadataProvider.java:562) at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.find(FileSystemMetadataProvider.java:605) at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.load(FileSystemMetadataProvider.java:114) at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:197) at org.kitesdk.data.Datasets.load(Datasets.java:108) at org.kitesdk.data.Datasets.load(Datasets.java:140) at org.kitesdk.data.mapreduce.DatasetKeyInputFormat$ConfigBuilder.readFrom(DatasetKeyInputFormat.java:92) at org.kitesdk.data.mapreduce.DatasetKeyInputFormat$ConfigBuilder.readFrom(DatasetKeyInputFormat.java:139) at org.apache.sqoop.mapreduce.JdbcExportJob.configureInputFormat(JdbcExportJob.java:84) at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:429) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) [ec2-user@ip-172-31-22-86 ~]$
2.解决方法
1.将Sqoop抽数脚本修改为如下:
sqoop export \ --connect jdbc:mysql://ip-172-31-22-86.ap-southeast-1.compute.internal:3306/test_db \ --username testuser \ --password password \ --table mytest_parquet \ --hcatalog-database default \ --hcatalog-table mytest_parquet --num-mappers 1
参数说明:
--table:MySQL库中的表名
--hcatalog-database:Hive中的库名
--hcatalog-table:Hive库中的表名,需要抽数的表
--num-mappers:执行作业的Map数
2.修改后执行抽数作业
作业执行成功。
3.查看MySQL表数据
3.总结
目前通过Sqoop从Hive的parquet抽数到关系型数据库的时候会报kitesdk找不到文件的错,这是Sqoop已知的问题,参考SQOOP-2907:
https://issues.apache.org/jira/browse/SQOOP-2907
该jira目前并没有修复,如果要实现该功能,需要参考第二章的做法,使用hcatalog参数指定到Hive表。