问题 Cannot run program /etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py No such file

错误日志

在使用cdh大数据平台运行spark任务时，在ApplicationMaster 端出现如下错误日志

java.io.IOException: Cannot run program "/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/topology.py" (in directory "/data/yarn/nm/usercache/azkaban/appcache/application_1567507254359_44080/container_e11_1567507254359_44080_01_000001"): error=2, No such file or directory
   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
   at org.apache.hadoop.util.Shell.runCommand(Shell.java:548)
   at org.apache.hadoop.util.Shell.run(Shell.java:504)
   at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
   at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
   at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
   at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
   at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
   at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:81)
   at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$handleAllocatedContainers$2.apply(YarnAllocator.scala:356)
   at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$handleAllocatedContainers$2.apply(YarnAllocator.scala:355)
   at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
   at org.apache.spark.deploy.yarn.YarnAllocator.handleAllocatedContainers(YarnAllocator.scala:355)
   at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:255)
   at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:368)
Caused by: java.io.IOException: error=2, No such file or directory
   at java.lang.UNIXProcess.forkAndExec(Native Method)
   at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
   at java.lang.ProcessImpl.start(ProcessImpl.java:134)
   at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

问题原因

出现这种问题，是没有安装spark客户端的Gateway实例，所以在ApplicationMaster 所在节点没有生成配置文件目录 /etc/spark/conf.cloudera.spark_on_yarn，在cdh扩容节点时，spark也需要增加Gateway实例，进行客户端配置

解决办法

增加spark Gateway实例，部署客户端配置