Hadoop机架感知-----数据存放副本所在datanodes是放在哪个节点?

 

 

 

 

这两张图说明了,集群之间的网络拓扑距离,经过一次网络交换机的话,网络拓扑

hadoop的数据副本是怎么放得呢?

hadoop副本的存放遵循以下的原则:

1.可靠性：block存储在两个机架上以保证一个机架故障导致整个数据丢失.
2.写带宽：写操作仅仅穿过一个网络交换机,从上图中可以看出,网络拓扑距离如果是在一个机架上是的距离是最短的,但是不满足可靠性的原则.
3.读操作：选择其中得一个机架去读
4.block分布在整个集群上。

所以hadoop 在副本存放的时候为了满足以上原则,客户端上放置个副本的数据,不同机架上再放置副本.但是hadoop本事不能识别网络的拓扑的http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-common/RackAwareness.html hadoop 官方的文档,说明机架感知,以下文字摘自Apache hadoop 翻译而来,建议去看官方文档:

机架感知

Hadoop组件具有机架感知功能。例如，通过将一个块放在不同的机架上，HDFS块放置将使用机架感知来实现容错。这可在网络交换机发生故障或群集内分区时提供数据可用性。

Hadoop主守护程序通过调用配置文件指定的外部脚本或java类来获取集群从属的机架ID。使用java类或拓扑的外部脚本，输出必须遵循java org.apache.hadoop.net.DNSToSwitchMapping接口。接口期望保持一对一的对应关系，并且拓扑信息的格式为'/ myrack / myhost'，其中'/'是拓扑定界符，'myrack'是机架标识符，'myhost'是个人host。假设每个机架有一个/ 24个子网，可以使用'/192.168.100.0/192.168.100.5'格式作为唯一的机架 - 主机拓扑映射。

要使用java类进行拓扑映射，类名由配置文件中的topology.node.switch.mapping.impl参数指定。例如，NetworkTopology.java包含在hadoop发行版中，可以由Hadoop管理员自定义。使用Java类而不是外部脚本具有性能优势，因为当新的从属节点注册自身时，Hadoop不需要分叉外部进程。

 

Rack Awareness

Hadoop components are rack-aware. For example, HDFS block placement will use rack awareness for fault tolerance by placing one block replica on a different rack. This provides data availability in the event of a network switch failure or partition within the cluster.

Hadoop master daemons obtain the rack id of the cluster slaves by invoking either an external script or java class as specified by configuration files. Using either the java class or external script for topology, output must adhere to the java org.apache.hadoop.net.DNSToSwitchMapping interface. The interface expects a one-to-one correspondence to be maintained and the topology information in the format of ‘/myrack/myhost’, where ‘/’ is the topology delimiter, ‘myrack’ is the rack identifier, and ‘myhost’ is the individual host. Assuming a single /24 subnet per rack, one could use the format of ‘/192.168.100.0/192.168.100.5’ as a unique rack-host topology mapping.

To use the java class for topology mapping, the class name is specified by the topology.node.switch.mapping.impl parameter in the configuration file. An example, NetworkTopology.java, is included with the hadoop distribution and can be customized by the Hadoop administrator. Using a Java class instead of an external script has a performance benefit in that Hadoop doesn’t need to fork an external process when a new slave node registers itself.

java 中实现接口DNSToSwitchMapping

1.打包源代码,将jar包发送到所以集群节点hadoop/common/lib下

import org.apache.hadoop.net.DNSToSwitchMapping;

import java.util.ArrayList;
import java.util.List;

public class TestRack implements DNSToSwitchMapping {

    /**
     * @param names 传入的是一个主机名或ip地址的列表
     * @return 返回网络拓扑路径/rack1/192.168.23.102
     */
    public List<String> resolve(List<String> names) {

        List<String> list = new ArrayList<String>();

        for(String name : names){
            //如果参数是主机名s101 s102 s103 s104 s105
            if(name.startsWith("s")){
                //获取主机名后缀
                int suffix = Integer.parseInt(name.substring(1));

                //如果后缀是101-103，则在rack1中
                if(suffix < 104){
                    String path = "/rack1/";
                    list.add(path);
                }
                else {
                    String path = "/rack2/";
                    list.add(path);
                }

            }
            //参数是ip地址192.168.23.101
            else{
                //获取后缀
                int suffix = Integer.parseInt(name.split("\\.")[3]);

                //如果后缀是101-103，则在rack1中
                if(suffix < 104){
                    String path = "/rack1/";
                    list.add(path);
                }
                else {
                    String path = "/rack2/";
                    list.add(path);
                }
            }
        }
        return list;

    }

    public void reloadCachedMappings() {

    }

    public void reloadCachedMappings(List<String> names) {

    }
}

2.配置core-site.xml,添加如下配置

<property>

         <name>net.topology.node.switch.mapping.impl</name>

         <value>TestRack类名的全路径</value>

       </property>