Java Yarn 获取集群队列负载

背景：任务调度时候，需要检测集群上某个队列的资源使用情况，当资源很紧张的时候，禁止任务提交。
           本贴基于 springboot 实现一个简单的检测

1.在某个类中声明一个创建YarnClient的实体函数，@bean 功能等同于新建一个实体类

@Bean
public YarnClient createYarnClient() {
    YarnClient yarnClient = YarnClient.createYarnClient();
    yarnClient.init(LocalHadoopConfiguration.load());
    yarnClient.start();
    return yarnClient;
}

import org.apache.hadoop.conf.Configuration;
public class LocalHadoopConfiguration {
    private final static String CORE_SITE = "core-site.xml";
    private final static String YARN_SITE = "yarn-site.xml";
    private final static String HDFS_SITE = "hdfs-site.xml";
    private final static String MAPRED_SITE = "mapred-site.xml";
    public static Configuration load() {
        Configuration conf = new Configuration();
        conf.addResource(CORE_SITE);
        conf.addResource(YARN_SITE);
        conf.addResource(HDFS_SITE);
        conf.addResource(MAPRED_SITE);
        return conf;
    }
}

2.在需要使用的类中，注入YarnClient对象

@Autowired
private YarnClient yarnClient;

3. 编写一个 check函数，检测资源上限。使用cache缓存，减少rm的http请求。规则自定，如下是使用capacity限制和pendingApps和runningApps比例来做限制

// QUEUE_LIMIT_METRIC = 0.5   
  @Cacheable(cacheNames = CLUSTER_LOAD_METRICS, key = "#userQueue")
  public long checkQueueLoadMetrics(String userQueue) {
    
    QueueInfo queueInfo;
    try {
      queueInfo = yarnClient.getQueueInfo(userQueue);
    } catch (Exception e) {
      e.printStackTrace();
      return 0;
    }

    if (queueInfo == null) {
      LOG.info("Cluster queue info is null, waiting a period time");
      return 0;
    }

    QueueStatistics queueStatistics = queueInfo.getQueueStatistics();
    double capacity = queueInfo.getCapacity();
    long pendingApps = queueStatistics.getNumAppsPending();
    long runningApps = queueStatistics.getNumAppsRunning();

    LOG.info("Current cluster queue:{} metrics, capacity:{} %, running apps:{}, pending apps:{}",
        userQueue, capacity * 100, runningApps, pendingApps);

    if (pendingApps < runningApps * BackTraceConfig.PENDING_DIVID_RUNING_METRIC
        || capacity < BackTraceConfig.QUEUE_CAPACITY_LIMIT) {
      return BackTraceConfig.MAX_RUN_LIMIT;
    }

    return 0;
  }

4.在提交application的地方进行检测即可
5.执行springboot的机器需有hadoop客户端，脚本类似如下，将需要的配置引入，环境相关的配置在CLASSPATH，-cp引入，执行就行。脚本根据实际修改

nohup java -Dspring.config.location=/path/to/dag-backtrace.properties ${JAVA_OPTS} \
        -cp ${CLASSPATH}:`echo $(hadoop classpath 2>/dev/null | awk 'BEGIN{RS=":"}{printf("%s\n",$0)}') \
        | sed 's/ /\n/g' | grep -v jersey | grep -v javax.ws.rs-api |  awk 'BEGIN{ORS=":"}{print $NF}'`  \
        ${MAIN_CLASS} >${STDOUT_LOG} 2>&1 &

参考文献
https://dzone.com/articles/how-to-use-the-yarn-api-to-determine-resources-ava-1
https://hadoop.apache.org/docs/r2.8.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_Queue_API