前面分享了hadoop伪分布式的搭建方法,这次来分享一下分布式方式的搭建方法。
主机安装或虚拟机安装这里就不再赘述,下面直接进入正题:
1.下面是网络配置,全部为静态:
master : 192.168.80.128
slave-1 : 192.168.80.129
slave-2 : 192.168.80.130
slave-3 : 192.168.80.131
slave-4 : 192.168.80.132
2.修改hosts,加入如下内容:
192.168.80.128 master
192.168.80.129 slave1
192.168.80.130 slave2
192.168.80.131 slave3
192.168.80.132 slave4
3.修改各个主机的hostname,为对应的名字
4.配置SSH免密码登陆,保证五台主机任意之间可以免密码访问
5.关闭五台主机防火墙
$ sudo systemctl stop firewalld.service
$ sudo systemctl disable firewalld.service
必须关闭防火网,否则在存储文件时,可能会出现以下错误:
[zhoupan@master ~]$ hadoop-2.8.0/bin/hadoop fs -put hadoop-2.8.0.tar.gz /data
17/07/14 01:47:58 WARN hdfs.DataStreamer: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /input/hadoop-2.8.0.tar.gz._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1733)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2496)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:828)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRp