Scala(程序化方式):
getExecutorStorageStatus和getExecutorMemoryStatus都返回执行程序的数量,包括驱动程序.
如下面的示例代码段.
/** Method that just returns the current active/registered executors
* excluding the driver.
* @param sc The spark context to retrieve registered executors.
* @return a list of executors each in the form of host:port.
*/
def currentActiveExecutors(sc: SparkContext): Seq[String] = {
val allExecutors = sc.getExecutorMemoryStatus.map(_._1)
val driverHost: String = sc.getConf.get("spark.driver.host")
allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList
}
sc.getConf.getInt("spark.executor.instances", 1)
同样获取所有属性并打印如下,您也可以获得核心信息..
sc.getConf.getAll.mkString("\n")
要么
sc.getConf.toDebugString
执行程序spark.driver.cores驱动程序的spark.executor.cores应该具有此值.
Python:
编辑
但是可以使用SparkSession公开的Py4J绑定来访问它.
sc._jsc.sc().getExecutorMemoryStatus()