Running Spark on Kubernetes
- 注意,Spark 2.3已经正式发布,原生支持Kubernetes,这里的Spark 2.2已经过时。
Support for running on Kubernetes is available in experimental status. The feature set is currently limited and not well-tested. This should not be used in production environments.
Prerequisites
- You must have a running Kubernetes cluster with access configured to it using kubectl. If you do not already have a working Kubernetes cluster, you may setup a test cluster on your local machine using minikube.
- We recommend that minikube be updated to the most recent version (0.19.0 at the time of this documentation), as some earlier versions may not start up the kubernetes cluster with all the necessary components.
- The minimum version of Kubernetes currently supported is 1.6. So your Kubernetes cluster must be running Kubernetes 1.6 and up.
- You must have the
kube-dns
cluster add-on available in your Kubernetes cluster. You can verify that thekube-dns
service is up by running commandkubectl get svc --namespace=kube-system
. - You must have appropriate permissions to create and list pods, ConfigMaps and secrets in your cluster. You can verify that you can create these resources by running
kubectl auth can-i create pods
,kubectl auth can-i create configmap
, andkubectl auth can-i create secrets
which should give you a responseyes
otherwise you can’t create these resources. To test if you can list these resources repeat the commands withlist
instead ofcreate
e.q.kubectl auth can-i list pods
. - You must have a spark distribution with Kubernetes support. The following documentation corresponds to v2.2.0-kubernetes-0.5.0.
This may be obtained from the release tarball or by building Spark with Kubernetes support.
Docker Images
Kubernetes requires users to supply images that can be deployed into containers within pods. The images are built to be run in a container runtime environment that Kubernetes supports. Docker is a container runtime environment that is frequently used with Kubernetes, so Spark provides some support for working with Docker to get started quickly.
If you wish to use pre-built docker images, you may use the images published in kubespark. The images are as follows:
Component | Image |
---|---|
Spark Driver Image | kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 |
Spark Executor Image | kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 |
Spark Initialization Image | kubespark/spark-init:v2.2.0-kubernetes-0.5.0 |
Spark Staging Server Image | kubespark/spark-resource-staging-server:v2.2.0-kubernetes-0.5.0 |
PySpark Driver Image | kubespark/spark-driver-py:v2.2.0-kubernetes-0.5.0 |
PySpark Executor Image | kubespark/spark-executor-py:v2.2.0-kubernetes-0.5.0 |
SparkR Driver Image | kubespark/spark-driver-r:v2.2.0-kubernetes-0.5.0 |
SparkR Executor Image | kubespark/spark-executor-r:v2.2.0-kubernetes-0.5.0 |
You may also build these docker images from sources, or customize them as required.
In addition to the above, there are default images supplied for auxiliary components, like the Resource Staging Server and Spark External Shuffle Service.
Component | Image |
---|---|
Spark Resource Staging Server | kubespark/spark-resource-staging-server:v2.2.0-kubernetes-0.5.0 |
Spark External Shuffle Service | kubespark/spark-shuffle:v2.2.0-kubernetes-0.5.0 |
There is a script, sbin/build-push-docker-images.sh
that you can use to build and push customized spark distribution images consisting of all the above components.
Example usage is:
./sbin/build-push-docker-images.sh -r docker.io/myusername -t my-tag build
./sbin/build-push-docker-images.sh -r docker.io/myusername -t my-tag push
Docker files are under the dockerfiles/
and can be customized further before building using the supplied script, or manually.
Submitting Applications to Kubernetes
Kubernetes applications can be executed via spark-submit
. For example, to compute the value of pi, assuming the images are set up as described above:
bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
--kubernetes-namespace default \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
The Spark master, specified either via passing the --master
command line argument to spark-submit
or by setting spark.master
in the application’s configuration, must be a URL with the format k8s://<api_server_url>
. Prefixing the master string with k8s://
will cause the Spark application to launch on the Kubernetes cluster, with the API server being contacted at api_server_url
. If no HTTP protocol is specified in the URL, it defaults to https
. For example, setting the master to k8s://example.com:443
is equivalent to setting it to k8s://https://example.com:443
, but to connect without TLS on a different port, the master would be set to k8s://http://example.com:8443
.
If you have a Kubernetes cluster setup, one way to discover the apiserver URL is by executing kubectl cluster-info
.
> kubectl cluster-info
Kubernetes master is running at http://127.0.0.1:8080
In the above example, the specific Kubernetes cluster can be used with spark submit by specifying --master k8s://http://127.0.0.1:8080
as an argument to spark-submit.
Note that applications can currently only be executed in cluster mode, where the driver and its executors are running on the cluster.
Finally, notice that in the above example we specify a jar with a specific URI with a scheme of local://
. This URI is the location of the example jar that is already in the Docker image. Using dependencies that are on your machine’s local disk is discussed below.
Python Support
With the ever growing support for Python by data scientists, we have supported the submission of PySpark applications. These applications follow the general syntax that you would expect from other cluster managers. The submission of a PySpark job is similar to the submission of Java/Scala applications except you do not supply a class, as expected. Here is how you would execute a Spark-Pi example:
bin/spark-submit \
--deploy-mode cluster \
--master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
--kubernetes-namespace <k8s-namespace> \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver-py:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor-py:v2.2.0-kubernetes-0.5.0 \
--jars local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar \
local:///opt/spark/examples/src/main/python/pi.py 10
With Python support it is expected to distribute .egg
, .zip
and .py
libraries to executors via the --py-files
option. We support this as well, as seen with the following example:
bin/spark-submit \
--deploy-mode cluster \
--master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
--kubernetes-namespace <k8s-namespace> \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver-py:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor-py:v2.2.0-kubernetes-0.5.0 \
--jars local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar \
--py-files local:///opt/spark/examples/src/main/python/sort.py \
local:///opt/spark/examples/src/main/python/pi.py 10
You may also customize your Docker images to use different pip
packages that suit your use-case. As you can see with the current spark-driver-py
Docker image we have commented out the current pip module support that you can uncomment to use:
...
ADD examples /opt/spark/examples
ADD python /opt/spark/python
RUN apk add --no-cache python && \
python -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip install --upgrade pip setuptools && \
rm -r /root/.cache
# UNCOMMENT THE FOLLOWING TO START PIP INSTALLING PYTHON PACKAGES
# RUN apk add --update alpine-sdk python-dev
# RUN pip install numpy
...
And bake into your docker image whichever PySpark files you wish to include by merely appending to the following exec command with your appropriate file (i.e. MY_SPARK_FILE)
...
CMD SPARK_CLASSPATH="${SPARK_HOME}/jars/*" && \
if ! [ -z ${SPARK_MOUNTED_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_MOUNTED_CLASSPATH:$SPARK_CLASSPATH"; fi && \
if ! [ -z ${SPARK_SUBMIT_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_SUBMIT_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi && \
if ! [ -z ${SPARK_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi && \
if ! [ -z ${SPARK_MOUNTED_FILES_DIR} ]; then cp -R "$SPARK_MOUNTED_FILES_DIR/." .; fi && \
exec /sbin/tini -- ${JAVA_HOME}/bin/java $SPARK_DRIVER_JAVA_OPTS -cp $SPARK_CLASSPATH \
-Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY \
$SPARK_DRIVER_CLASS $PYSPARK_PRIMARY MY_PYSPARK_FILE,$PYSPARK_FILES $SPARK_DRIVER_ARGS
Dependency Management
Application dependencies that are being submitted from your machine need to be sent to a resource staging server that the driver and executor can then communicate with to retrieve those dependencies. A YAML file denoting a minimal set of Kubernetes resources that runs this service is located in the file conf/kubernetes-resource-staging-server.yaml
. This YAML file configures a Deployment with one pod running the resource staging server configured with a ConfigMap, and exposes the server through a Service with a fixed NodePort. Deploying a resource staging server with the included YAML file requires you to have permissions to create Deployments, Services, and ConfigMaps.
To run the resource staging server with default configurations, the Kubernetes resources can be created:
kubectl create -f conf/kubernetes-resource-staging-server.yaml
and then you can compute the value of Pi as follows:
bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://<k8s-apiserver-host>:<k8s-apiserver-port> \
--kubernetes-namespace default \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.initcontainer.docker.image=kubespark/spark-init:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.resourceStagingServer.uri=http://<address-of-any-cluster-node>:31000 \
./examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
The Docker image for the resource staging server may also be built from source, in a similar manner to the driver and executor images. The Dockerfile is provided in dockerfiles/resource-staging-server/Dockerfile
.
The provided YAML file specifically sets the NodePort to 31000 on the service’s specification. If port 31000 is not available on any of the nodes of your cluster, you should remove the NodePort field from the service’s specification and allow the Kubernetes cluster to determine the NodePort itself. Be sure to provide the correct port in the resource staging server URI when submitting your application, in accordance to the NodePort chosen by the Kubernetes cluster.
Dependency Management Without The Resource Staging Server
Note that this resource staging server is only required for submitting local dependencies. If your application’s dependencies are all hosted in remote locations like HDFS or http servers, they may be referred to by their appropriate remote URIs. Also, application dependencies can be pre-mounted into custom-built Docker images. Those dependencies can be added to the classpath by referencing them with local://
URIs and/or setting the SPARK_EXTRA_CLASSPATH
environment variable in your Dockerfiles. For any remote dependencies (not using the local:// scheme), the init-container (spark.kubernetes.initcontainer.docker.image
) must be specified during submission.
Accessing Kubernetes Clusters
Spark-submit also supports submission through the local kubectl proxy. One can use the authenticating proxy to communicate with the api server directly without passing credentials to spark-submit.
The local proxy can be started by running:
kubectl proxy
If our local proxy were listening on port 8001, we would have our submission looking like the following:
bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://http://127.0.0.1:8001 \
--kubernetes-namespace default \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
Communication between Spark and Kubernetes clusters is performed using the fabric8 kubernetes-client library. The above mechanism using kubectl proxy
can be used when we have authentication providers that the fabric8 kubernetes-client library does not support. Authentication using X509 Client Certs and OAuth tokens is currently supported.
Dynamic Executor Scaling
Spark on Kubernetes supports Dynamic Allocation with cluster mode. This mode requires running an external shuffle service. This is typically a daemonset with a provisioned hostpath volume. This shuffle service may be shared by executors belonging to different SparkJobs. Using Spark with dynamic allocation on Kubernetes assumes that a cluster administrator has set up one or more shuffle-service daemonsets in the cluster.
A sample configuration file is provided in conf/kubernetes-shuffle-service.yaml
which can be customized as needed for a particular cluster. It is important to note that spec.template.metadata.labels
are setup appropriately for the shuffle service because there may be multiple shuffle service instances running in a cluster. The labels give Spark applications a way to target a particular shuffle service.
For example, if the shuffle service we want to use is in the default namespace, and has pods with labels app=spark-shuffle-service
and spark-version=2.2.0
, we can use those tags to target that particular shuffle service at job launch time. In order to run a job with dynamic allocation enabled, the command may then look like the following:
bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.GroupByTest \
--master k8s://<k8s-master>:<port> \
--kubernetes-namespace default \
--conf spark.app.name=group-by-test \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.4.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.4.0 \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.shuffle.service.enabled=true \
--conf spark.kubernetes.shuffle.namespace=default \
--conf spark.kubernetes.shuffle.labels="app=spark-shuffle-service,spark-version=2.2.0" \
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar 10 400000 2
Advanced
Configuring Kubernetes Roles and Service Accounts
In Kubernetes clusters with RBAC enabled, users can configure Kubernetes RBAC roles and service accounts used by the various Spark on Kubernetes components to access the Kubernetes API server.
Driver
The Spark driver pod uses a Kubernetes service account to access the Kubernetes API server to create and watch executor pods. The service account used by the driver pod must have the appropriate permission for the driver to be able to do its work.
Specifically, at minimum, the service account must be granted a Role
or ClusterRole
that allows driver pods to create pods and services. By default, the driver pod is automatically assigned the default
service account in the namespace specified by --kubernetes-namespace
, if no service account is specified when the pod gets created.
Depending on the version and setup of Kubernetes deployed, this default
service account may or may not have the role that allows driver pods to create pods and services under the default Kubernetes RBAC policies. Sometimes users may need to specify a custom service account that has the right role granted. Spark on Kubernetes supports specifying a custom service account to be used by the driver pod through the configuration property spark.kubernetes.authenticate.driver.serviceAccountName=<service account name>
. For example to make the driver pod to use the spark
service account, a user simply adds the following option to the spark-submit
command:
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
To create a custom service account, a user can use the kubectl create serviceaccount
command. For example, the following command creates a service account named spark
:
kubectl create serviceaccount spark
To grant a service account a Role
or ClusterRole
, a RoleBinding
or ClusterRoleBinding
is needed. To create a RoleBinding
or ClusterRoleBinding
, a user can use the kubectl create rolebinding
(or clusterrolebinding
for ClusterRoleBinding
) command. For example, the following command creates an edit
ClusterRole
in the default
namespace and grants it to the spark
service account created above:
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
Note that a Role
can only be used to grant access to resources (like pods) within a single namespace, whereas a ClusterRole
can be used to grant access to cluster-scoped resources (like nodes) as well as namespaced resources (like pods) across all namespaces. For Spark on Kubernetes, since the driver always creates executor pods in the same namespace, a Role
is sufficient, although users may use a ClusterRole
instead. For more information on RBAC authorization and how to configure Kubernetes service accounts for pods, please refer to Using RBAC Authorization and Configure Service Accounts for Pods.
Resource Staging Server
The Resource Staging Server (RSS) watches Spark driver pods to detect completed Spark applications so it knows when to safely delete resource bundles of the applications. When running as a pod in the same Kubernetes cluster as the Spark applications, by default (spark.kubernetes.authenticate.resourceStagingServer.useServiceAccountCredentials
defaults to true
), the RSS uses the default Kubernetes service account token located at /var/run/secrets/kubernetes.io/serviceaccount/token
and the CA certificate located at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
. Note that the locations referred to here are both within the RSS pod and are used by Kubernetes by default.
When running outside the Kubernetes cluster or when spark.kubernetes.authenticate.resourceStagingServer.useServiceAccountCredentials
is set to false
, the credentials for authenticating with the Kubernetes API server can be specified using other configuration properties as documented in Spark Properties. Regardless of which credential is used, the credential must allow the RSS to view pods in any namespace.
Shuffle Service
The shuffle service runs as a Kubernetes DaemonSet
. Each pod of the shuffle service watches Spark driver pods so at minimum it needs a role that allows it to view pods. Additionally, the shuffle service uses a hostPath
volume for shuffle data. Writing to a hostPath
volume requires either that the shuffle service process runs as root in a privileged container or that the user is able to modify the file permissions on the host to be able to write to a hostPath
volume. Even in the first case, a pod may or may not be able to use a hostPath
volume, depending on the types of volumes usable in the pod, which are controlled by PodSecurityPolicy
.
In Kubernetes 1.5 and newer, one can use PodSecurityPolicy
to control access to privileged containers based on user role and groups. To enable hostPath
volume using a PodSecurityPolicy
, a user needs to create a new or use an existing PodSecurityPolicy
that has hostPath
listed in the .spec.volumes
field as this example shows.
Then the user needs to create a Role
(or a ClusterRole
if necessary) that is allowed to use
the PodSecurityPolicy
. Finally, the user needs a RoleBinding
(or ClusterRoleBinding
in case of a ClusterRole
) to grant the Role
(or ClusterRole
) to the service account used by the shuffle service pods. For more details on how to use PodSecurityPolicy
and RBAC to control access to PodSecurityPolicy
, please refer to this doc.
To specify a custom service account for the shuffle service pods, add the following to the pod template in the shuffle service DaemonSet
defined in conf/kubernetes-shuffle-service.yaml
:
spec:
serviceAccountName: <service account name>
Securing the Resource Staging Server with TLS
The default configuration of the resource staging server is not secured with TLS. It is highly recommended to configure this to protect the secrets and jars/files being submitted through the staging server.
The YAML file in conf/kubernetes-resource-staging-server.yaml
includes a ConfigMap resource that holds the resource staging server’s configuration. The properties can be adjusted here to make the resource staging server listen over TLS. Refer to the security page for the available settings related to TLS. The namespace for the resource staging server is kubernetes.resourceStagingServer
, so for example the path to the server’s keyStore would be set by spark.ssl.kubernetes.resourceStagingServer.keyStore
.
In addition to the settings specified by the previously linked security page, the resource staging server supports the following additional configurations:
Property Name | Default | Meaning |
---|---|---|
spark.ssl.kubernetes.resourceStagingServer.keyPem | (none) | Private key file encoded in PEM format that the resource staging server uses to secure connections over TLS. If this is specified, the associated public key file must be specified inspark.ssl.kubernetes.resourceStagingServer.serverCertPem . PEM files and a keyStore file (set byspark.ssl.kubernetes.resourceStagingServer.keyStore ) cannot both be specified at the same time. |
spark.ssl.kubernetes.resourceStagingServer.serverCertPem | (none) | Certificate file encoded in PEM format that the resource staging server uses to secure connections over TLS. If this is specified, the associated private key file must be specified inspark.ssl.kubernetes.resourceStagingServer.keyPem . PEM files and a keyStore file (set byspark.ssl.kubernetes.resourceStagingServer.keyStore ) cannot both be specified at the same time. |
spark.ssl.kubernetes.resourceStagingServer.keyStorePasswordFile | (none) | Provides the KeyStore password through a file in the container instead of a static value. This is useful if the keyStore's password is to be mounted into the container with a secret. |
spark.ssl.kubernetes.resourceStagingServer.keyPasswordFile | (none) | Provides the keyStore's key password using a file in the container instead of a static value. This is useful if the keyStore's key password is to be mounted into the container with a secret. |
Note that while the properties can be set in the ConfigMap, you will still need to consider the means of mounting the appropriate secret files into the resource staging server’s container. A common mechanism that is used for this is to use Kubernetes secrets that are mounted as secret volumes. Refer to the appropriate Kubernetes documentation for guidance and adjust the resource staging server’s specification in the provided YAML file accordingly.
Finally, when you submit your application, you must specify either a trustStore or a PEM-encoded certificate file to communicate with the resource staging server over TLS. The trustStore can be set with spark.ssl.kubernetes.resourceStagingServer.trustStore
, or a certificate file can be set with spark.ssl.kubernetes.resourceStagingServer.clientCertPem
. For example, our SparkPi example now looks like this:
bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
--kubernetes-namespace default \
--conf spark.executor.instances=5 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.initcontainer.docker.image=kubespark/spark-init:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.resourceStagingServer.uri=https://<address-of-any-cluster-node>:31000 \
--conf spark.ssl.kubernetes.resourceStagingServer.enabled=true \
--conf spark.ssl.kubernetes.resourceStagingServer.clientCertPem=/home/myuser/cert.pem \
examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
Spark Properties
Below are some other common properties that are specific to Kubernetes. Most of the other configurations are the same from the other deployment modes. See the configuration page for more information on those.
</tr> </tr> </table> ## Current Limitations Running Spark on Kubernetes is currently an experimental feature. Some restrictions on the current implementation that should be lifted in the future include: * Applications can only run in cluster mode. * Only Scala, Java, and Python applications can be run.
Property Name | Default | Meaning |
---|---|---|
spark.kubernetes.namespace | default | The namespace that will be used for running the driver and executor pods. When using spark-submit in cluster mode, this can also be passed to spark-submit via the --kubernetes-namespace command line argument. |
spark.kubernetes.driver.docker.image | spark-driver:2.2.0 | Docker image to use for the driver. Specify this using the standard Docker tag format. |
spark.kubernetes.executor.docker.image | spark-executor:2.2.0 | Docker image to use for the executors. Specify this using the standardDocker tag format. |
spark.kubernetes.initcontainer.docker.image | spark-init:2.2.0 | Docker image to use for the init-container that is run before the driver and executor containers. Specify this using the standard Docker tag format. The init-container is responsible for fetching application dependencies from both remote locations like HDFS or S3, and from the resource staging server, if applicable. |
spark.kubernetes.shuffle.namespace | default | Namespace in which the shuffle service pods are present. The shuffle service must be created in the cluster prior to attempts to use it. |
spark.kubernetes.shuffle.labels | (none) | Labels that will be used to look up shuffle service pods. This should be a comma-separated list of label key-value pairs, where each label is in the format key=value . The labels chosen must be such that they match exactly one shuffle service pod on each node that executors are launched. |
spark.kubernetes.allocation.batch.size | 5 | Number of pods to launch at once in each round of executor pod allocation. |
spark.kubernetes.allocation.batch.delay | 1 | Number of seconds to wait between each round of executor pod allocation. |
spark.kubernetes.authenticate.submission.caCertFile | (none) | Path to the CA cert file for connecting to the Kubernetes API server over TLS when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
spark.kubernetes.authenticate.submission.clientKeyFile | (none) | Path to the client key file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
spark.kubernetes.authenticate.submission.clientCertFile | (none) | Path to the client cert file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
spark.kubernetes.authenticate.submission.oauthToken | (none) | OAuth token to use when authenticating against the Kubernetes API server when starting the driver. Note that unlike the other authentication options, this is expected to be the exact string value of the token to use for the authentication. |
spark.kubernetes.authenticate.driver.caCertFile | (none) | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
spark.kubernetes.authenticate.driver.clientKeyFile | (none) | Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). If this is specified, it is highly recommended to set up TLS for the driver submission server, as this value is sensitive information that would be passed to the driver pod in plaintext otherwise. |
spark.kubernetes.authenticate.driver.clientCertFile | (none) | Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
spark.kubernetes.authenticate.driver.oauthToken | (none) | OAuth token to use when authenticating against the against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this must be the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod. If this is specified, it is highly recommended to set up TLS for the driver submission server, as this value is sensitive information that would be passed to the driver pod in plaintext otherwise. |
spark.kubernetes.authenticate.driver.serviceAccountName | default | Service account that is used when running the driver pod. The driver pod uses this service account when requesting executor pods from the API server. Note that this cannot be specified alongside a CA cert file, client key file, client cert file, and/or OAuth token. |
spark.kubernetes.authenticate.resourceStagingServer.caCertFile | (none) | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the resource staging server when it monitors objects in determining when to clean up resource bundles. |
spark.kubernetes.authenticate.resourceStagingServer.clientKeyFile | (none) | Path to the client key file for authenticating against the Kubernetes API server from the resource staging server when it monitors objects in determining when to clean up resource bundles. The resource staging server must have credentials that allow it to view API objects in any namespace. |
spark.kubernetes.authenticate.resourceStagingServer.clientCertFile | (none) | Path to the client cert file for authenticating against the Kubernetes API server from the resource staging server when it monitors objects in determining when to clean up resource bundles. The resource staging server must have credentials that allow it to view API objects in any namespace. |
spark.kubernetes.authenticate.resourceStagingServer.oauthToken | (none) | OAuth token value for authenticating against the Kubernetes API server from the resource staging server when it monitors objects in determining when to clean up resource bundles. The resource staging server must have credentials that allow it to view API objects in any namespace. Note that this cannot be set at the same time asspark.kubernetes.authenticate.resourceStagingServer.oauthTokenFile . |
spark.kubernetes.authenticate.resourceStagingServer.oauthTokenFile | (none) | File containing the OAuth token to use when authenticating against the against the Kubernetes API server from the resource staging server, when it monitors objects in determining when to clean up resource bundles. The resource staging server must have credentials that allow it to view API objects in any namespace. Note that this cannot be set at the same time as spark.kubernetes.authenticate.resourceStagingServer.oauthToken . |
spark.kubernetes.authenticate.resourceStagingServer.useServiceAccountCredentials | true | Whether or not to use a service account token and a service account CA certificate when the resource staging server authenticates to Kubernetes. If this is set, interactions with Kubernetes will authenticate using a token located at /var/run/secrets/kubernetes.io/serviceaccount/token and the CA certificate located at/var/run/secrets/kubernetes.io/serviceaccount/ca.crt . Note that ifspark.kubernetes.authenticate.resourceStagingServer.oauthTokenFile is set, it takes precedence over the usage of the service account token file. Also, ifspark.kubernetes.authenticate.resourceStagingServer.caCertFile is set, it takes precedence over using the service account's CA certificate file. This generally should be set to true (the default value) when the resource staging server is deployed as a Kubernetes pod, but should be set to false if the resource staging server is deployed by other means (i.e. when running the staging server process outside of Kubernetes). The resource staging server must have credentials that allow it to view API objects in any namespace. |
spark.kubernetes.executor.memoryOverhead | executorMemory * 0.10, with minimum of 384 | The amount of off-heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%). |
spark.kubernetes.driver.label.[LabelName] | (none) | Add the label specified by LabelName to the driver pod. For example, spark.kubernetes.driver.label.something=true . Note that Spark also adds its own labels to the driver pod for bookkeeping purposes. |
spark.kubernetes.driver.annotation.[AnnotationName] | (none) | Add the annotation specified by AnnotationName to the driver pod. For example, spark.kubernetes.driver.annotation.something=true . |
spark.kubernetes.executor.label.[LabelName] | (none) | Add the label specified by LabelName to the executor pods. For example, spark.kubernetes.executor.label.something=true . Note that Spark also adds its own labels to the driver pod for bookkeeping purposes. |
spark.kubernetes.executor.annotation.[AnnotationName] | (none) | Add the annotation specified by AnnotationName to the executor pods. For example, spark.kubernetes.executor.annotation.something=true . |
spark.kubernetes.driver.pod.name | (none) | Name of the driver pod. If not set, the driver pod name is set to "spark.app.name" suffixed by the current timestamp to avoid name conflicts. |
spark.kubernetes.submission.waitAppCompletion | true | In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job. |
spark.kubernetes.resourceStagingServer.port | 10000 | Port for the resource staging server to listen on when it is deployed. |
spark.kubernetes.resourceStagingServer.uri | (none) | URI of the resource staging server that Spark should use to distribute the application's local dependencies. Note that by default, this URI must be reachable by both the submitting machine and the pods running in the cluster. If one URI is not simultaneously reachable both by the submitter and the driver/executor pods, configure the pods to access the staging server at a different URI by settingspark.kubernetes.resourceStagingServer.internal.uri as discussed below. |
spark.kubernetes.resourceStagingServer.internal.uri | Value of spark.kubernetes.resourceStagingServer.uri | URI of the resource staging server to communicate with when init-containers bootstrap the driver and executor pods with submitted local dependencies. Note that this URI must by the pods running in the cluster. This is useful to set if the resource staging server has a separate "internal" URI that must be accessed by components running in the cluster. |
spark.ssl.kubernetes.resourceStagingServer.internal.trustStore | Value of spark.ssl.kubernetes.resourceStagingServer.trustStore | Location of the trustStore file to use when communicating with the resource staging server over TLS, as init-containers bootstrap the driver and executor pods with submitted local dependencies. This can be a URI with a scheme of local:// , which denotes that the file is pre-mounted on the pod's disk. A uri without a scheme or a scheme of file:// will result in this file being mounted from the submitting machine's disk as a secret into the init-containers. |
spark.ssl.kubernetes.resourceStagingServer.internal.trustStorePassword | Value of
| Password of the trustStore file that is used when communicating with the resource staging server over TLS, as init-containers bootstrap the driver and executor pods with submitted local dependencies. |
spark.ssl.kubernetes.resourceStagingServer.internal.trustStoreType | Value of
| Type of the trustStore file that is used when communicating with the resource staging server over TLS, when init-containers bootstrap the driver and executor pods with submitted local dependencies. |
spark.ssl.kubernetes.resourceStagingServer.internal.clientCertPem | Value of spark.ssl.kubernetes.resourceStagingServer.clientCertPem | Location of the certificate file to use when communicating with the resource staging server over TLS, as init-containers bootstrap the driver and executor pods with submitted local dependencies. This can be a URI with a scheme of local:// , which denotes that the file is pre-mounted on the pod's disk. A uri without a scheme or a scheme of file:// will result in this file being mounted from the submitting machine's disk as a secret into the init-containers. |
spark.kubernetes.mountdependencies.jarsDownloadDir | /var/spark-data/spark-jars | Location to download jars to in the driver and executors. This will be mounted as an empty directory volume into the driver and executor containers. |
spark.kubernetes.mountdependencies.filesDownloadDir | /var/spark-data/spark-files | Location to download files to in the driver and executors. This will be mounted as an empty directory volume into the driver and executor containers. |
spark.kubernetes.report.interval | 1s | Interval between reports of the current Spark job status in cluster mode. |
spark.kubernetes.docker.image.pullPolicy | IfNotPresent | Docker image pull policy used when pulling Docker images with Kubernetes. |
spark.kubernetes.driver.limit.cores | (none) | Specify the hard cpu limit for the driver pod |
spark.kubernetes.executor.limit.cores | (none) | Specify the hard cpu limit for a single executor pod |
spark.kubernetes.node.selector.[labelKey] | (none) | Adds to the node selector of the driver pod and executor pods, with key labelKey and the value as the configuration's value. For example, setting spark.kubernetes.node.selector.identifier to myIdentifier will result in the driver pod and executors having a node selector with key identifier and value myIdentifier . Multiple node selector keys can be added by setting multiple configurations with this prefix. |
spark.executorEnv.[EnvironmentVariableName] | (none) | Add the environment variable specified by EnvironmentVariableName to the Executor process. The user can specify multiple of these to set multiple environment variables. |
spark.kubernetes.driverEnv.[EnvironmentVariableName] | (none) | Add the environment variable specified by EnvironmentVariableName to the Driver process. The user can specify multiple of these to set multiple environment variables. |
spark.kubernetes.driver.secrets.[SecretName] | (none) | Mounts the Kubernetes secret named SecretName onto the path specified by the value in the driver Pod. The user can specify multiple instances of this for multiple secrets. |
spark.kubernetes.executor.secrets.[SecretName] | (none) | Mounts the Kubernetes secret named SecretName onto the path specified by the value in the executor Pods. The user can specify multiple instances of this for multiple secrets. |