Sqoop导出到MySQL错误赏析

最新推荐文章于 2024-05-12 08:14:48 发布

weixin_33949359

最新推荐文章于 2024-05-12 08:14:48 发布

阅读量757

点赞数

文章标签：数据库大数据 java

原文链接：https://my.oschina.net/u/4017963/blog/2994195

版权

2019独角兽企业重金招聘Python工程师标准>>>

Group FileSystemCounters is deprecated

具体错误

18/12/27 15:04:30 INFO mapreduce.Job: Running job: job_1543155456252_6606
18/12/27 15:04:35 INFO mapreduce.Job: Job job_1543155456252_6606 running in uber mode : false
18/12/27 15:04:35 INFO mapreduce.Job:  map 0% reduce 0%
18/12/27 15:04:41 INFO mapreduce.Job:  map 100% reduce 0%
18/12/27 15:04:41 INFO mapreduce.Job: Job job_1543155456252_6606 failed with state FAILED due to: Task failed task_1543155456252_6606_m_000002
Job failed as tasks failed. failedMaps:1 failedReduces:0

18/12/27 15:04:41 INFO mapreduce.Job: Counters: 8
	Job Counters 
		Failed map tasks=3
		Launched map tasks=3
		Data-local map tasks=3
		Total time spent by all maps in occupied slots (ms)=20104
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=10052
		Total vcore-milliseconds taken by all map tasks=10052
		Total megabyte-milliseconds taken by all map tasks=20586496
18/12/27 15:04:41 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/12/27 15:04:41 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 13.822 seconds (0 bytes/sec)
18/12/27 15:04:41 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/12/27 15:04:41 INFO mapreduce.ExportJobBase: Exported 0 records.
18/12/27 15:04:41 ERROR tool.ExportTool: Error during export: 
Export job failed!
	at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:439)
	at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
	at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
	at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

找到了其中的关键信息 Group FileSystemCounters is deprecated , 在CDH的坑之Sqoop导出数据到MySQL 这里找到了一些信息
里面说是Hive和MySQL的时间字段不匹配导致的,但是我的字段都匹配
在经历了两天的困扰之后,突然(真的是突然) --- 执行命令hadoop fs -text /user/hive/warehouse/database.db/table/dt=20181212/* | head -100之后,看到一连串乱码,想到了把hive的表储存成了orc的格式了.
于是重新建了个表储存成textfile格式的,终于好了.

转载于:https://my.oschina.net/u/4017963/blog/2994195