Hadoop-3.1.1 Yarn 对GPU的调度和管理

GPU ON YARN上的配置

hadoop-3.1.1版本的yarn支持对gpu的调度和管理，主要分为两种模式：

• yarn自动获取gpu的资源，进行分配；
• 由用户指定使用哪些gpu资源；

具体的配置方法详见文档 Using GPU On YARN.md

GPU ON YARN 上的执行

执行命令：

yarn jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -shell_script /root/smitest.sh \
  -container_memory 1024 \
  -container_resources memory-mb=1024,vcores=1,yarn.io/gpu=1 \
  -num_containers 2

或者直接执行

yarn jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -jar /home/nht/hadoop-3.1.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.1.jar \
  -shell_script /root/smitest.sh 

脚本smitest.sh里面的内容：

# !/bin/bash

/usr/bin/nvidia-smi > /tmp/smitest

输出文件smitest里面的内容：

Fri Sep 28 14:43:09 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.66                 Driver Version: 384.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0 Off |                  N/A |
| 27%   33C    P8    12W / 180W |     10MiB /  8112MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    Off  | 00000000:02:00.0 Off |                  N/A |
| 27%   33C    P8    12W / 180W |     10MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Process