How-to: configure hadoop rack awareness

Update/genrate following files in ${HADOOP_CONF_DIR}: 
  • core-site.xml:
            <!--Rack aware-->
            <property>
                    <name>net.topology.script.file.name</name>
                    <value>${HADOOP_CONF_DIR}/rack-topology.sh</value>
            </property>
  • Add file ${HADOOP_CONF_DIR}/rack-topology.sh and make it executable:
    #!/bin/bash

    # Supply appropriate rack prefix
    RACK_PREFIX=default

    # To test, supply a hostname as script input:
    if [ $# -gt 0 ]; then
    CTL_FILE=rack_topology.data

    if [ ! -f ${HADOOP_CONF_DIR}/${CTL_FILE} ]; then
      echo -n "/$RACK_PREFIX/rack "
      exit 0
    fi

    while [ $# -gt 0 ] ; do
      nodeArg=$1
      exec< ${HADOOP_CONF_DIR}/${CTL_FILE}
      result=""
      while read line ; do
        ar=( $line )
        if [ "${ar[0]}" = "$nodeArg" ] ; then
          result="${ar[1]}"
        fi
      done
      shift
      if [ -z "$result" ] ; then
        echo -n "/$RACK_PREFIX/rack "
      else
        echo -n "/$RACK_PREFIX/rack_$result "
      fi
    done

    else
      echo -n "/$RACK_PREFIX/rack "
    fi
  • Generate ${HADOOP_CONF_DIR}/rack_topology.data:
    # Add Hostnames to this file. Format <host ip> <rack_location>
    ip1 01
    ip2 02
  • Test to make sure could get rack information:
    [hadoop@master01 hadoop]$ ./rack-topology.sh master01.dc.tj
    /default/rack_01
  • rsync the ${HADOOP_CONF_DIR} in all nodes od hadoop cluster  
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值