dolphinscheduler调度datax脚本执行错误原因（我见）

最新推荐文章于 2024-05-16 05:55:19 发布
最新推荐文章于 2024-05-16 05:55:19 发布
阅读量1.7k 收藏
点赞数
文章标签： hadoop 大数据 hdfs
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Linmeng001/article/details/127294626
版权 

2022-10-13 00:05:06.188 [job-0] ERROR HdfsWriter$Job - 判断文件路径[message:filePath =/junzun_recruit_origin/recruit_web/db_spider/position_info/2022-10-12]是否存在时发生网络IO异常,请检查您的网络是否正常！
	2022-10-13 00:05:06.191 [job-0] ERROR JobContainer - Exception when job run
	com.alibaba.datax.common.exception.DataXException: Code:[HdfsWriter-06], Description:[与HDFS建立连接时出现IO异常.]. - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
		at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
		at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2017)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1441)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3125)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)
	
		at org.apache.hadoop.ipc.Client.call(Client.java:1476)
		at org.apache.hadoop.ipc.Client.call(Client.java:1407)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
		at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
		at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
		at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
		at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
		at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:156)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.prepare(HdfsWriter.java:151)
		at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724)
		at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309)
		at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
		at com.alibaba.datax.core.Engine.start(Engine.java:92)
		at com.alibaba.datax.core.Engine.entry(Engine.java:171)
		at com.alibaba.datax.core.Engine.main(Engine.java:204)
	 - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
		at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
		at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2017)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1441)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3125)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)
	
		at org.apache.hadoop.ipc.Client.call(Client.java:1476)
		at org.apache.hadoop.ipc.Client.call(Client.java:1407)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
		at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
		at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
		at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
		at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
		at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:156)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.prepare(HdfsWriter.java:151)
		at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724)
		at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309)
		at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
		at com.alibaba.datax.core.Engine.start(Engine.java:92)
		at com.alibaba.datax.core.Engine.entry(Engine.java:171)
		at com.alibaba.datax.core.Engine.main(Engine.java:204)
	
		at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:161) ~[hdfswriter-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.prepare(HdfsWriter.java:151) ~[hdfswriter-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
		at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
	Caused by: org.apache.hadoop.ipc.RemoteException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
		at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
		at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2017)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1441)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3125)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)
	
		at org.apache.hadoop.ipc.Client.call(Client.java:1476) ~[hadoop-common-2.7.1.jar:na]
		at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[hadoop-common-2.7.1.jar:na]
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[hadoop-common-2.7.1.jar:na]
		at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) ~[na:na]
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) ~[hadoop-hdfs-2.7.1.jar:na]
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_212]
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_212]
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212]
		at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) ~[hadoop-common-2.7.1.jar:na]
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[hadoop-common-2.7.1.jar:na]
		at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source) ~[na:na]
		at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116) ~[hadoop-hdfs-2.7.1.jar:na]
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) ~[hadoop-hdfs-2.7.1.jar:na]
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) ~[hadoop-hdfs-2.7.1.jar:na]
		at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.7.1.jar:na]
		at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317) ~[hadoop-hdfs-2.7.1.jar:na]
		at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) ~[hadoop-common-2.7.1.jar:na]
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:156) ~[hdfswriter-0.0.1-SNAPSHOT.jar:na]
		... 7 common frames omitted
	2022-10-13 00:05:06.194 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
	2022-10-13 00:05:06.297 [job-0] ERROR Engine - 
	
	经DataX智能分析,该任务最可能的错误原因是:
	com.alibaba.datax.common.exception.DataXException: Code:[HdfsWriter-06], Description:[与HDFS建立连接时出现IO异常.]. - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
		at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
		at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2017)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1441)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3125)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)
	
		at org.apache.hadoop.ipc.Client.call(Client.java:1476)
		at org.apache.hadoop.ipc.Client.call(Client.java:1407)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
		at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
		at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
		at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
		at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
		at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:156)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.prepare(HdfsWriter.java:151)
		at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724)
		at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309)
		at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
		at com.alibaba.datax.core.Engine.start(Engine.java:92)
		at com.alibaba.datax.core.Engine.entry(Engine.java:171)
		at com.alibaba.datax.core.Engine.main(Engine.java:204)
	 - org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
		at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
		at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2017)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1441)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3125)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)
	
		at org.apache.hadoop.ipc.Client.call(Client.java:1476)
		at org.apache.hadoop.ipc.Client.call(Client.java:1407)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
		at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
		at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
		at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
		at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
		at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:156)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.prepare(HdfsWriter.java:151)
		at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724)
		at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309)
		at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
		at com.alibaba.datax.core.Engine.start(Engine.java:92)
		at com.alibaba.datax.core.Engine.entry(Engine.java:171)
		at com.alibaba.datax.core.Engine.main(Engine.java:204)
	
		at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:161)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.prepare(HdfsWriter.java:151)
		at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724)
		at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309)
		at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
		at com.alibaba.datax.core.Engine.start(Engine.java:92)
		at com.alibaba.datax.core.Engine.entry(Engine.java:171)
		at com.alibaba.datax.core.Engine.main(Engine.java:204)
	Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
		at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
		at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2017)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1441)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3125)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)
	
		at org.apache.hadoop.ipc.Client.call(Client.java:1476)
		at org.apache.hadoop.ipc.Client.call(Client.java:1407)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
		at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
		at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
		at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
		at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
		at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
		at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
		at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
		at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.isPathexists(HdfsHelper.java:156)
		... 7 more

dolphin会随机在节点所在位置进行任务分配，而我的json文件主要集中在hadoop master节点，所以需要同步到每台节点，这样其他任务节点接收到任务，才能访问到配置文件。

林蒙_
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Dolphinscheduler中提交DATAX任务报错
qq_51620976的博客
12-07 1840
如题，谁能帮忙解决一下，下面是报错的日志
Ubuntu Dolphinscheduler 执行命令 source: 未找到
qq_44820964的博客
11-04 747
服务器为Ubuntu，dolphinscheduler执行普通的shell命令，提示source未找到。时，无法识别source命令，需要切换为。原因：ubuntu服务器，当指定为
最新DolphinScheduler海豚调度教程_海豚调度使用说明，2024年最新2024年百度大数据开发面试真题
最新发布
2401_84186026的博客
05-16 1211
dolphinscheduler-alert 告警模块，提供 AlertServer 服务。dolphinscheduler-api web应用模块，提供 ApiServer 服务。dolphinscheduler-common 通用的常量枚举、工具类、数据结构或者基类dolphinscheduler-dao 提供数据库访问等操作。dolphinscheduler-remote 基于 netty 的客户端、服务端。
Dolphinscheduler中DataX任务报错anaconda3/bin/python2.7: No such file or directory
weixin_42258633的博客
10-18 1824
2.不安装2.7的也可以，方法如下。1.安装2.7版本的Python。
dolphinscheduler执行datax任务报错：/bin/sh: java: 未找到命令
weixin_44407691的博客
11-07 954
dolphinscheduler执行datax任务
基于dolphinscheduler on datax的自动数据同步
Combinerz的博客
10-08 1656
在使用dolphinscheduler基于datax进行mysql表数据同步到hive时, 当mysql表数量较大时，配置的dolphin datax任务链的工作量大且容易出错，所有开发了此简易web项目自动生成datax json文件，自动创建hive表，一键生成dolophin任务链实现自动化数据同步。 源码github地址：https://github.com/zz-big/gather.git 如果觉得还不错，请star下。 dolphinscheduler官网：https://dolphinsc
dolphinScheduler调度.rar
06-17
- **任务调度**：DolphinScheduler支持定时调度和事件驱动调度，可以根据业务需求灵活设定执行策略，如立即执行、周期性执行、依赖执行等。 - **依赖关系管理**：任务之间可以设置前置和后置依赖，确保任务按照...
dolphinscheduler配置
07-13
Python和Scala支持使Dolphinscheduler能执行脚本任务。在Dolphinscheduler中，你需要确保系统环境已经安装了Python和Scala，并且在`conf/dolphinscheduler-env.sh`中配置了它们的路径。Python任务可以直接在Dolphin...
dolphinscheduler parameter and date compute test.json
07-16
dolphinscheduler 日期转换及参数测试流程
java通过Python命令执行datax任务调度
03-23
1、通过java调用Python命令执行datax任务调度 2、自动开始任务和调度结束关闭任务调度释放内存 3、如果我们在cmd使用命令调度，执行完毕后无法释放内存，会造成内存泄露不足，出现报错“Error occurred during ...
datax抽取数据到hdfs的null值变成''(引号)的问题
02-20
mysql的null值通过datax抽取到hdfs，会变成引号，这不是我们所需要的，所以需要修改一下datax的源码
关于通过java调用datax,返回任务执行的方法
08-25
在实际业务中，我们需要使用 DataX 把数据从文本写入到数据库，而原来的做法都是使用 Python 通过 datax.py 去调用脚本。为了更好地管控 DataX 的任务，我们需要使用 Java 集成的方式去调用 DataX，並返回任务执行的...
datax安装使用及使用过程中遇到的问题
世樹のブログ
12-30 1万+
DataX 是一个异构数据源离线同步工具，致力于实现包括关系型数据库(MySQL、Oracle等)、HDFS、Hive、ODPS、HBase、FTP等各种异构数据源之间稳定高效的数据同步功能。 相关概念和设计理念不再赘述，这里记录个人初次使用时的过程。 datax环境要求: Linux JDK(1.8以上，推荐1.8) Python(推荐Python2.6.X) Apache Mave...
Datax MySQL2Hive抽数ClassCastException: java.lang.String cannot be cast to java.lang.Integer问题解决
欲望以提升热忱 毅力以磨平高山
04-10 2690
1、现象 com.alibaba.datax.common.exception.DataXException: Code:[HdfsWriter-04], Description:[您配置的文件在写入时出现IO异常.]. - java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer at ...
Dolphinscheduler配置Datax踩坑记录
热门推荐
快乐星球
11-09 1万+
tmp/dolphinscheduler/exec/process 下文件创建失败问题 dolphinscheduler调度datax任务时需要在tmp/dolphinscheduler/exec/process 目录下创建一系列临时文件，但是在worker运行日志中/opt/soft/dolphinscheduler/logs/dolphinscheduler-worker.log看到创建失败的报错 [taskAppId=TASK-1-10-13]:[178] - datax task failure j
Datax往ClickHouse数据库插入数据前执行truncate table遇到超时问题Read timed out
qq_45124566的博客
01-11 2517
该问题是由于数据量过大(也有可能是网络问题)，处理请求太久，导致 ClickHouse 连接超时，建议在 ClickHouse 链接字符串后面加上参数：`?socket_timeout=600000`
datax同步es数据
qq_39887331的博客
05-22 6822
xshell登录到服务器 进入到/home/datax目录底下 cd /home/datax 使用ps -auxww 查看下有多少datax进程 java -server -Xmx1g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./log -Dfile.encoding=UTF-8 -Dlogback.statusListenerCl...
dolphinscheduler datax
03-16
DolphinScheduler是一个分布式调度系统，可以用于调度DataX任务。DataX是阿里巴巴开源的数据同步工具，支持多种数据源和目标数据存储，可以实现不同数据源之间的数据同步和迁移。DolphinScheduler可以通过Web界面配置和管理DataX任务，支持任务依赖、任务流程控制、任务监控等功能。同时，DolphinScheduler还支持多种调度方式，如定时调度、手动触发、依赖触发等，可以满足不同场景下的需求。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值