spark和python的关系_spark程序与参数的关系

What is spark.python.worker.memory?

Spark on YARN resource manager: Relation between YARN Containers and Spark Executors？

When running Spark on YARN, each Spark executor runs as a YARN container

所以有，--executor-memory <= yarn.scheduler.maximum-allocation-mb(一个container的最大值)

同时有：

yarn.scheduler.maximum-allocation-mb <= yarn.nodemanager.resource.memory-mb (每个节点yarn可以使用的内存资源上线)

所以最终三者之间的关系为：

--executor-memory <= yarn.scheduler.maximum-allocation-mb(一个container的最大值) <= yarn.nodemanager.resource.memory-mb (每个节点yarn可以使用的内存资源上线)

可以启动的executor数量:

- execuoterNum = spark.cores.max/spark.executor.cores

每个executor上可以执行多少个task

- taskNum = spark.executor.cores/ spark.task.cpus

spark.python.worker.memory is a subset of the memory from spark.executor.memory

spark.python.worker.memory is used for Python worker in executor

spark.python.worker.memory <= spark.executor.memory(--executor-memory)

Because of GIL, pyspark use multiple python process in the executor, one for each task.

spark.python.worker.memory will tell the python worker to when to

spill the data into disk.

If you have enough memory in executor, increase spark.python.worker.memory will

let python worker to use more memory during shuffle.which will increase the performance.

综上， pyspark运行时会在executor中起多个python进程task，每个task多少内存由spark.python.worker.memory控制

那么什么参数控制一个executor中其多少个python-task？只需要控制spark.python.worker.memory就可以吗？会exector/worker?

一个Executor上同时运行多少个Task，就会有多少个对应的pyspark.worker进程

spark.yarn.executor.memoryoverhead 的内存从哪里去？与spark.executor.memory和container的关系是什么？