HDFS机架感知配置（block放置策略/NodeGroup分配策略配置）

HDFS机架感知配置（block放置策略/NodeGroup分配策略配置）

1、配置hdfs-site和core-site文件

• core-site.xml

<property>
    <name>net.topology.impl</name>
    <value>org.apache.hadoop.net.NetworkTopologyWithNodeGroup</value>
</property>
<property>
    <name>net.topology.script.file.name</name>
    <value>/tmp/rack.py</value>	
</property>
<property>
    <name>net.topology.nodegroup.aware</name>
    <value>true</value>
</property>

• hdfs-site.xml

<property>
    <name>dfs.block.replicator.classname</name>
    <value>
        org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup
    </value>
</property>
<property>
    <name>dfs.use.dfs.network.topology</name>
    <value>false</value>
</property>

2、nodegroup示例

192.168.0.1 /rack1/nodegroup1
192.168.0.2 /rack1/nodegroup1
192.168.0.3 /rack1/nodegroup2
192.168.0.4 /rack1/nodegroup2
192.168.0.5 /rack2/nodegroup3
192.168.0.6 /rack2/nodegroup3

添加脚本

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import sys
rack = {
    "192.168.0.1": "/rack1/nodegroup1",
    "192.168.0.2": "/rack1/nodegroup1",
    "192.168.0.3": "/rack1/nodegroup2",
    "192.168.0.4": "/rack1/nodegroup2",
    "192.168.0.5": "/rack2/nodegroup3",
    "192.168.0.6": "/rack2/nodegroup3",
}
if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit(1)
        ip_address = sys.argv[1]
        result = rack.get(ip_address, "/default-rack")
        print(result)

给脚本授权

chmod +777 /tmp/rack.py

3、通过命令查看是否配置成功

hdfs dfsadmin -printTopology