Hadoop3.1.4（apache发行版）HA搭建时遇到的问题及解决方案

最新推荐文章于 2022-11-25 17:10:57 发布
最新推荐文章于 2022-11-25 17:10:57 发布
阅读量1.1k 收藏 3
点赞数 1
分类专栏： Hadoop 文章标签： hadoop 指针 hdfs java gwt
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_38258720/article/details/112135976
版权

目录

主要参照的两篇博客：
Hadoop3.2.0使用详解.
hadoop3.1.0 HA高可用完全分布式集群的安装部署（详细教程）.

搭建环境：

  • centos8（5台）
  • JDK 11.0.9
  • hadoop 3.1.4

一.端口占用问题

2021-01-02 19:16:35,007 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping JournalNode metrics system...
2021-01-02 19:16:35,008 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JournalNode metrics system stopped.
2021-01-02 19:16:35,008 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JournalNode metrics system shutdown complete.
2021-01-02 19:16:35,008 ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNode: Failed to start journalnode.
java.net.BindException: Port in use: 0.0.0.0:8480
	at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1226)
	at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1248)
	at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1307)
	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1162)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNodeHttpServer.start(JournalNodeHttpServer.java:86)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.start(JournalNode.java:234)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.run(JournalNode.java:205)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.main(JournalNode.java:415)
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:8480
	at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
	at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1213)
	at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1244)
	... 8 more
Caused by: java.net.BindException: 地址已在使用
	at java.base/sun.nio.ch.Net.bind0(Native Method)
	at java.base/sun.nio.ch.Net.bind(Net.java:461)
	at java.base/sun.nio.ch.Net.bind(Net.java:453)
	at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
	at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
	at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
	... 11 more

解决方案：查看端口号是否被占用，netstat -anp |grep 端口号。
如果端口没被占用，重启解决一切。

二.权限问题

2021-01-02 18:58:22,430 INFO org.apache.hadoop.hdfs.qjournal.server.JournalNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-01-02 18:58:22,572 ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNode: Failed to start JournalNode.
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not writable: /data/hdata/dfs/journal
	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:167)
	at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:100)
	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:77)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.validateAndCreateJournalDir(JournalNode.java:195)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.start(JournalNode.java:218)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.run(JournalNode.java:205)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.main(JournalNode.java:415)
2021-01-02 18:58:22,584 ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNode: Failed to start journalnode.
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not writable: /data/hdata/dfs/journal
	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:167)
	at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:100)
	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:77)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.validateAndCreateJournalDir(JournalNode.java:195)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.start(JournalNode.java:218)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.run(JournalNode.java:205)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.main(JournalNode.java:415)

解决方案：权限问题，hadoop的数据我存储在了根目录下，用户和用户组都是root
drwxr-xr-x. 2 root root 6 1月 2 20:17 journal
hadoop用户没有w权限，文件写不进去。
我直接把hadoop的数据存到hadoop用户目录下，省时省力。
drwxrwxr-x. 4 hadoop hadoop 37 1月 3 13:46 journal

2021-01-02 20:18:04,650 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-01-02 20:18:05,321 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/data/hdata/dfs/data
2021-01-02 20:18:05,386 WARN org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/data/hdata/dfs/data
EPERM: Operation not permitted
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:381)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:836)
	at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:508)
	at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:489)
	at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:511)
	at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:234)
	at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141)
	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239)
	at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52)
	at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-01-02 20:18:05,388 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
	at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:231)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2804)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2719)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2761)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2905)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2929)

解决方案：和上面问题相似，权限问题，启动不了datanode，因为data目录数据写不了。
drwxr-xr-x. 2 root root 6 1月 2 20:17 data

三.缺包或找不到主类问题

Caused by: java.lang.NoClassDefFoundError: javax/activation/DataSource
	at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.<clinit>(RuntimeBuiltinLeafInfoImpl.java:457)
	at com.sun.xml.bind.v2.model.impl.RuntimeTypeInfoSetImpl.<init>(RuntimeTypeInfoSetImpl.java:65)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:133)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:85)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.<init>(ModelBuilder.java:156)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.<init>(RuntimeModelBuilder.java:93)
	at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:473)
	at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:319)
	at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170)
	at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:262)
	at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:249)
	at javax.xml.bind.ContextFinder.find(ContextFinder.java:456)
	at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:656)
	at com.sun.jersey.api.json.JSONJAXBContext.<init>(JSONJAXBContext.java:255)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:69)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver$$FastClassByGuice$$6a7be7f6.newInstance(<generated>)
	at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
	at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
	... 54 more
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 88 more

解决方案：https://blog.csdn.net/slopop12/article/details/108710328
这种问题比较常见，缺啥就补啥。

[2021-01-03 14:52:02.446]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
原因: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster


[2021-01-03 14:52:02.447]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
原因: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster

解决方案：https://blog.csdn.net/qq_41684957/article/details/81710190

四.目录变文件问题

hdfs dfs -put /tmp/wc /tmp

org.apache.hadoop.fs.ParentNotDirectoryException: /tmp (is not a directory)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkIsDirectory(FSPermissionChecker.java:638)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkSimpleTraverse(FSPermissionChecker.java:629)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:604)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1801)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTraverse(FSDirectory.java:1819)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePath(FSDirectory.java:679)
	at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:50)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3192)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1157)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:714)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943)

解决方案：hdfs dfs -put /tmp/wc /tmp之前，先hdfs dfs -mkdir /tmp，不然tmp会被创成文件类型。

五.java空指针问题

2021-01-03 15:11:50,386 INFO mapreduce.Job:  map 0% reduce 0%
2021-01-03 15:11:50,408 INFO mapreduce.Job: Job job_1609657572378_0002 failed with state FAILED due to: Application application_1609657572378_0002 failed 2 times due to AM Container for appattempt_1609657572378_0002_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2021-01-03 15:11:49.602]Exception from container-launch.
Container id: container_1609657572378_0002_02_000001
Exit code: 1

[2021-01-03 15:11:49.606]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


[2021-01-03 15:11:49.606]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


For more detailed output, check the application tracking page: http://hadoop001:8088/cluster/app/application_1609657572378_0002 Then click on links to logs of each attempt.
. Failing the application.

终端显示的错误看不出来啥，查看web端历史日志，发现是java空指针问题。

2021-01-03 15:06:53,508 ERROR [Listener at 0.0.0.0/45119] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
	at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
	... 14 more

解决方案：https://blog.csdn.net/yj2434/article/details/107817519

程研板
关注
专栏目录
遇到问题--hadoop--DiskErrorException:Could not find any valid local directory for
直到世界的尽头
03-23 2879
情况 我们在运行importTsv命令时执行mapreduce，在output阶段报错org.apache.hadoop.util.DiskChecker$DiskErrorException:Could not find any valid local directory for。 完整报错如下: 2018-03-22 10:34:12,001 INFO [LocalJobRunne...
Hadoop集群搭建日记
chenhanl的博客
09-25 502
一.Hadoop的搭建与配置 1.环境配置要求 1.Centos7 2.jdk1.8.0_261-i586 3.hadoop 3.1.4 2.设置机器名称，在任意一台机器Centos7上修改域名映射 1.使用hostname命令查看本机名称。 2. vi /etc/hosts 域名映射 使用scp命令发送其他克隆机上 scp /etc/hosts root@
Hadoop执行Job时报DiskChecker$DiskErrorException异常原因
smallnetvisitor的博客
05-07 1198
版本：   $ hadoop version Hadoop 0.20.2-cdh3u4 Subversion git://ubuntu-slave01/var/lib/jenkins/workspace/CDH3u4-Full-RC/build/cdh3/hadoop20/0.20.2-cdh3u4/source -r 214dd731e3bdb687cb55988d3f47dd9e2...
Flink项目开发、上线全流程报错集合加处理办法（持续更新，附原因解决办法）
weixin_42642502的博客
09-27 4368
1、由于自己启动任务用的是root用户、未使用FlinkProject 用户产生以下报错 org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result. (JobID: 9ebe0758cd287126758d57b15fb5e5a3) at org.apache.flink.client.program.rest.RestClusterClient.submitJob
异常 Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local
_木子_简单_
12-04 3533
查看你的配置文件，找到mapred.local.dir这个参数，查看这个参数对应的文件夹的状态，如容量，可读性等 设置mapred-site.xml property> name>mapred.local.dirname> value>/data/hadoop/mapred/localvalue> property> mapred.l
docker-fastdfs
weixin_40944846的博客
08-13 177
docker部署fastdfs 下载代码 git clone https://github.com/LowEntropyBody/docker-fastdfs-with-nginx.git 切换目录 cd docker-fastdfs-with-nginx 修改配置文件 vi config/fdfs.conf 配置文件如下 # Configure the tracker address, it must be an external address (does not support DNS) TRA
hadoop 3.1.4
10-03
Hadoop 3.1.4 是 Apache 软件基金会发布的一个重要版本，它作为大数据处理的核心框架，为用户提供了高效、可靠的分布式计算能力。在这个版本中，Hadoop 进行了多方面的优化和增强，使得它在处理海量数据时性能更优，...
apache-dolphinscheduler-3.1.4-bin
最新发布
02-20
Apache DolphinScheduler是一款强大的分布式工作流管理系统，用于大数据任务的工作流编排和调度。它提供了图形化的工作流设计界面，使得用户可以方便地构建复杂的任务流程。在这个“apache-dolphinscheduler-3.1.4-...
3.1.4 YARN资源调度、Apache Hadoop 核心源码剖析、调优、Hadoop二次开发
weixin_47134119的博客
01-27 1600
3.1.4 YARN资源调度、Apache Hadoop 核心源码剖析 文章目录3.1.4 YARN资源调度、Apache Hadoop 核心源码剖析七、YARN资源调度7.1 Yarn架构7.2 Yarn任务提交(工作机制)7.3 Yarn调度策略7.4Yarn多租户资源隔离配置八、Apache Hadoop 核心源码剖析8.1 源码阅读准备8.2 NameNode 启动流程 七、YARN资源调度 7.1 Yarn架构 ResourceManager(rm)：处理客户端请求、启动/监控Ap
hadoop.dll 和 winutils.exe
07-10
标题中的"hadoop.dll"和"winutils.exe"是Hadoop在Windows环境中运行时的重要组成部分。 首先，我们来了解一下"hadoop.dll"。在Windows系统中，.dll（动态链接库）文件是一种共享库，包含了一系列函数和资源，可供多...
Hadoop 数据节点DataNode异常
玄火的专栏
09-27 1万+
使用start-all.sh脚本或者先后执行start-dfs.sh、start-yarn.sh脚本启动Hadoop后，通过jps命令检测DataNode节点进程发现有DataNode进程，数秒后DataNode进程莫名其妙“消失”了，只有Jps、NodeManager两个进程。简而言之就是DataNode进程异常，由于某种原因导致无法启动或者启动后数据节点DataNode进程自动关闭。
hadoop常见错误及处理方法
热门推荐
yonghutwo的专栏
06-30 5万+
如果大家在安装的时候遇到问题，或者按步骤安装完后却不能运行Hadoop，那么建议仔细查看日志信息，Hadoop记录了详尽的日志信息，日志文件保存在logs文件夹内。 无论是启动，还是以后会经常用到的MapReduce中的每一个job，以及HDFS等相关信息，Hadoop均存有日志文件以供分析。 1、hadoop-root-datanode-master.log 中有如下错误：
Hadoop datanode启动异常 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode
Kason_iWiki
11-25 2563
发现这两个id不一致，经查阅相关资料，id不一致会导致Initialization failed for Block pool。线上收到hadoop集群datanode掉线告警。然后再次尝试启动datanode进程。发现未存在datanode进程。
Hadoop常见错误及解决办法汇总
大讲台内测员的博客
08-31 1796
Hadoop常见错误及解决办法汇总   1，错误一：java.io.IOException: Incompatible clusterIDs 时常出现在namenode重新格式化之后9 `7 k# I: L2 |9 U* @6 d 2014-04-29 14:32:53,877 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode:
在搭建hadoop3.1.4分布式集群遇到集群启动时分节点datanode不启动且无日志的问题**
theoldy的博客
04-08 448
项目场景： 在搭建hadoop3.x分布式集群遇到集群启动是分节点datanode不启动且无日志的问题 问题描述： 在搭建hadoop分布式集群是按照教程进行配置后 ，启动集群两个分节点的datanode都没有正常启动 原因分析： ，搜索相关问题后，删除tmp ，logs 格式化namenode依然启动不了 ，且分节点都没有日志信息。 在经过多次搜索文章后 ，发现可能是hadoop目录下etc/hadoop/slaves的配置出了问题。但我的slaves文件是正确配置且分发到了其他两台主机，最后发现e
hadoop2.7.2中datanode无法启动，日志报错java.net.BindException: Port in use: localhost:0
caoyaqi的博客
07-27 5901
namenode启动正常，每个从节点上nodemanager也正常启动，就是从节点上服务datanode无法全部正常启动，有时候只有一个节点上的datanode起来了，其他都没启动，或者从节点上的datanode全部无法启动，在slave节点上查看日志  vi hadoop-root-datanode-slave2.log 发现日志中报有如下错误： java.net.BindException:
Linux Centos7 listen tcp 0.0.0.0:15672: bind: address already in use 端口被占用，解决办法
路西法Lucifer
10-22 2万+
问题：       从下图可以看到15672的端口被占用了，那怎么解决呢？？？  解决： netstat -nap 查看程序运行的pid，查找15672端口，可以发现确实被占用了，记住PID 当然也可以直接找到某个端口是否被占用，命令：netstat -nap | grep 15672 kill PID,杀死该进程 ，这里是kill 2407    ...
集群配置的错误
qq_41927934的博客
11-05 2965
在集群上运行jar程序的时候，出现以下这个错误信息： Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild 解决方法 1.打开配置文件yarn
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值