datax从hive导数据到mysql时hadoop版本不兼容异常

最新推荐文章于 2022-10-13 21:14:33 发布

着凉的皮皮虾

最新推荐文章于 2022-10-13 21:14:33 发布

阅读量1.1k

点赞数

分类专栏： datax 大数据文章标签： datax

本文链接：https://blog.csdn.net/weixin_37641413/article/details/103423899

版权

在使用DataX从Hive迁移数据到MySQL时遇到Hadoop版本不兼容的异常，异常源于DataX HDFSReader使用的Hadoop版本与集群版本不一致。通过下载DataX源码，修改POM文件将Hadoop版本改为2.6.0，重新打包并替换相关依赖。尽管解决了一些问题，但最终仍遇到JSON配置和路径错误。尝试在HDFSReader中添加特定文件未成功，考虑转向使用Sqoop进行数据迁移。

摘要由CSDN通过智能技术生成

异常原因
写在前面:解决了一些异常情况但是最后没彻底打通,换个方向继续搞,参考的朋友直接跳过

java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.get(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)Lorg/apache/hadoop/tracing/SpanReceiverHost;
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
	at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.getHDFSAllFiles(DFSUtil.java:123)
	at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.getAllFiles(DFSUtil.java:112)
	at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.prepare(HdfsReader.java:169)
	at com.alibaba.datax.core.job.JobContainer.prepareJobReader(JobContainer.java:715)
	at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:308)
	at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
	at com.alibaba.datax.core.Engine.start(Engine.java:92)
	at com.alibaba.datax.core.Engine.entry(Engine.java:171)
	at com.daqsoft.bdsp.service.impl.PushServiceImpl.test(PushServiceImpl.java:32)
	at com.daqsoft.bdsp.controller.PushController.test(PushController.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:8