Hadoop+Spark 大数据集群日常1
由于项目涉及Hadoop+Spark大数据集群,特写此文档,方便将来处理类似问题参照,也为后人提供解决方案。
本人才疏学浅,文档难免有错漏与不妥之处,欢迎与本人进行交流。
问题描述
Hadoop 2.7.7,搭建了一个生产环境,集群内有一台master机,一台slave机。使用一个脚本来启动集群
#!/bin/bash
cd /usr/local/
rm -rf ./hadoop/tmp
rm -rf ./hadoop/logs/*
cd /
/usr/local/hadoop/bin/hdfs namenode -format
/usr/local/hadoop/sbin/start-all.sh
/usr/local/spark/sbin/start-master.sh
/usr/local/spark/sbin/start-slaves.sh
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/30/logs
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark-python
/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark-python.zip /user/spark-python
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark/lib_jars/
/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark/jars/*.jar /user/spark/lib_jars/
hdfs dfs -chmod 777 /user/30/logs
/bin/bash
在执行到/usr/local/hadoop/bin/hadoop fs -put
的时候出现以下错误:
put: File /user/spark/lib_jars/spark-repl_2.11-2.2.0.jar._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
20/08/14 11:15:25 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/spark/lib_jars/spark-sketch_2.11-2.2.0.jar._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1620)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3135)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3059)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:493)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1603)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1388)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:554)
在slave节点上使用指令jps
查看java进程,发现并无Datanode进程,可知是Datanode未启动,发生了错误。
网上各类论坛已有文针对本问题给出解决方案。大致分为以下两类:
1、namenode与datanode下VERSION文件的clusterID值不匹配。
2、master节点的9000端口绑定在了不同的地址上。
经排除本文所讨论情况均不属于这两种情况的范围,本文不再讨论。
查看$HADOOP_HOME/logs/下的日志文件,发现是各个配置文件core-site.xml,hdfs-site.xml,mapred-site.xml,yarn-site.xml中的master节点的hosts出现了错误,才回忆起,由于项目功能要求,本人搭建的是弹性集群,若干个集群根据要求,其中包含不同的slaves。即某个特定的系统在不同的任务下充当不同集群的slaves。此需求要求在每一次启动Hadop集群之前自动化更新各配置文件中的hosts地址。
本着先定位问题的原则,手动修改各配置文件后使用脚本重启集群,发现上述
could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
问题仍然存在。但是slave节点上的Datanode进程已成功启动。
于是手动执行指令/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark-python.zip /user/spark-python
发现成功。
这使我有充分的理由怀疑是脚本刚执行完start-all.sh,该进程在slaves节点上启动Datanode需要时间,而slaves上的Datanode还未成功启动,master就开始hdfs put操作造成的。于是修改启动脚本为
#!/bin/bash
cd /usr/local/
rm -rf ./hadoop/tmp
rm -rf ./hadoop/logs/*
cd /
/usr/local/hadoop/bin/hdfs namenode -format
/usr/local/hadoop/sbin/start-all.sh
/usr/local/spark/sbin/start-master.sh
/usr/local/spark/sbin/start-slaves.sh
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/30/logs
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark-python
sleep 10s
/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark-python.zip /user/spark-python
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark/lib_jars/
/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark/jars/*.jar /user/spark/lib_jars/
hdfs dfs -chmod 777 /user/30/logs
/bin/bash
再一次重新启动脚本,发现hdfs -put .../spark-python
时出现了上述错误,而hdfs -put .../lib_jars/
成功了,完美的验证了我的想法。
将启动脚本最后修改为下文以后,问题完美解决。
#!/bin/bash
cd /usr/local/
rm -rf ./hadoop/tmp
rm -rf ./hadoop/logs/*
cd /
/usr/local/hadoop/bin/hdfs namenode -format
/usr/local/hadoop/sbin/start-all.sh
/usr/local/spark/sbin/start-master.sh
/usr/local/spark/sbin/start-slaves.sh
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/30/logs
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark-python
sleep 20s
/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark-python.zip /user/spark-python
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark/lib_jars/
/usr/local/hadoop/bin/hadoop fs -put /usr/local/spark/jars/*.jar /user/spark/lib_jars/
hdfs dfs -chmod 777 /user/30/logs
/bin/bash