局域网下连接其他电脑的HDFS集群

你还在为内存不够用而发愁?
感觉不感觉台式机用着不顺手?

参看本教程只需四步让你开发的web项目不再依赖自己的HDFS集群
本文详细介绍自己电脑上开发的项目如何连接其他机器的HDFS集群
最终成果:
团队内共用一个HDFS集群开发项目
条件:
同一局域网内

第一步:

参看我的局域网下用其他电脑连接VM虚拟机方法
这个是必须的,如果你会了那可以跳过

第二步:

学会怎么配置虚拟网路之后我们看一下这个配置文件
hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>dfs.nameservices</name>
 		<value>mycluster</value>
 	</property>
 	<property>
 		<name>dfs.ha.namenodes.mycluster</name>
 		<value>nn1,nn2</value>
 	</property>
        <property>  
		 <name>dfs.namenode.rpc-address.mycluster.nn1</name>
                 <value>node01:8020</value>
        </property>
	<property>
       		 <name>dfs.namenode.rpc-address.mycluster.nn2</name>     		                
       		 <value>node02:8020</value>
       	</property>
	<property>
             <name>dfs.namenode.http-address.mycluster.nn1</name>
      		 <value>node01:50070</value>
        </property>
        <property>
        	<name>dfs.namenode.http-address.mycluster.nn2</name>
                <value>node02:50070</value>
 	</property>
 	<property>
 		<name>dfs.namenode.shared.edits.dir</name>
 		<value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
 	</property>
	<property>
                <name>dfs.journalnode.edits.dir</name>
                <value>/var/abc/hadoop/ha/jn</value>
        </property>
        <property>
 	       <name>dfs.client.failover.proxy.provider.mycluster</name>  	
 	       <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
 	</property>
        <property>
               <name>dfs.ha.fencing.methods</name>
               <value>sshfence</value>
        </property>
        <property>
               <name>dfs.ha.fencing.ssh.private-key-files</name>
               <value>/root/.ssh/id_rsa</value>
        </property>
        <property>
 	       <name>dfs.ha.automatic-failover.enabled</name>
               <value>true</value>
        </property>

</configuration>

第三步 修改虚拟机和本机的Ip端口映射

可以看到上面有许多端口号,是虚拟机的端口.我们去VMware修改这些端口的映射
修改方法参看:局域网下用其他电脑连接VM虚拟机方法

如此一来我们就把虚拟机node01的50070端口映射为本机的50071端口下了,同样的我们将上面代码所有的端口全部用本机端口映射一遍,




这里就不全部贴出来了,修改完成之后

第四步 修改配置文件

为本机的Ip加对应的端口号

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>dfs.nameservices</name>
 		<value>mycluster</value>
 	</property>
 	<property>
 		<name>dfs.ha.namenodes.mycluster</name>
 		<value>nn1,nn2</value>
 	</property>
        <property>  
		 <name>dfs.namenode.rpc-address.mycluster.nn1</name>
                 <value>172.17.36.93:18020</value>
        </property>
	<property>
       		 <name>dfs.namenode.rpc-address.mycluster.nn2</name>     		                
       		 <value>172.17.36.93:28020</value>
       	</property>
	<property>
             <name>dfs.namenode.http-address.mycluster.nn1</name>
      		 <value>172.17.36.93:50073</value>
        </property>
        <property>
        	<name>dfs.namenode.http-address.mycluster.nn2</name>
                <value>172.17.36.93:50072</value>
 	</property>
 	<property>
 		<name>dfs.namenode.shared.edits.dir</name>
 		<value>qjournal://172.17.36.93:18485;172.17.36.93:28485;172.17.36.93:38485/mycluster</value>
 	</property>
	<property>
                <name>dfs.journalnode.edits.dir</name>
                <value>/var/abc/hadoop/ha/jn</value>
        </property>
        <property>
 	       <name>dfs.client.failover.proxy.provider.mycluster</name>  	
 	       <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
 	</property>
        <property>
               <name>dfs.ha.fencing.methods</name>
               <value>sshfence</value>
        </property>
        <property>
               <name>dfs.ha.fencing.ssh.private-key-files</name>
               <value>/root/.ssh/id_rsa</value>
        </property>
        <property>
 	       <name>dfs.ha.automatic-failover.enabled</name>
               <value>true</value>
        </property>

</configuration>

到这里还没完,还有一个配置文件需要修改
core-site.xml
同样的操作,将原来2181端口映射为本机的端口,修改配置文件为

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

	<property> 
		<name>fs.defaultFS</name>
      		<value>hdfs://mycluster</value>
      </property>
      <property>
         	<name>ha.zookeeper.quorum</name>
         	<value>172.17.36.93:21812,172.17.36.93:21813,172.17.36.93:21814</value>
      </property>


</configuration>

完成上述操作后在本机启动你的web项目已经和原来一样可以用了
但是用其他电脑运行web项目仍然报404什么原因呢?
答案是本机的防火墙需要放出刚才设置的那些端口,或者直接将防火墙关闭

如此一来,在同一局域网下的所有电脑,只要带着你修改后的配置文件,就可以访问你的HDFS集群了.

感谢您的浏览