spark2.4.1 on Kubernetes

修改 dockerfile

参见 https://blog.csdn.net/zhixingheyi_tian/article/details/89028989

编译部署

./bin/docker-image-tool.sh -r zhixingheyitian -t spark2.4.1 build
./bin/docker-image-tool.sh -r zhixingheyitian -t spark2.4.1 push （这一步不能省略）

如何省略 push 操作，在提交 spark-submit 时，会有以下错误

status: [ContainerStatus(containerID=null, image=zhixingheyitian/spark:spark2.4.1, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = Unknown desc = Error response from daemon: manifest for zhixingheyitian/spark:spark2.4.1 not found, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]

执行 push 后

The push refers to a repository [docker.io/zhixingheyitian/spark]
5a642627eb7a: Pushed 
231ae8a129fc: Pushed 
39cbe825df21: Pushed 
b08fb9d91d57: Pushed 
47a1282ea643: Pushed 
14b31a13eccb: Pushed 
f4311143a246: Pushed 
864b41a15a91: Pushed 
b97e1a6c18dd: Layer already exists 
344fb4b275b7: Layer already exists 
bcf2f368fe23: Layer already exists 
spark2.4.1: digest: sha256:8f08f90c68a5c4806f6ba57d1eb675c239d6fd96ca7c83cec1b6fe0d1ff25d06 size: 2624
The push refers to a repository [docker.io/zhixingheyitian/spark-py]
An image does not exist locally with the tag: docker.io/zhixingheyitian/spark-py
The push refers to a repository [docker.io/zhixingheyitian/spark-r]
An image does not exist locally with the tag: docker.io/zhixingheyitian/spark-r

Run Spark-PI

export NO_PROXY=localhost,127.0.0.1,192.168.99.108,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24
bin/spark-submit \
    --master k8s://https://192.168.99.108:8443 \
    --deploy-mode cluster\
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=1 \
    --conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1 \
    local:///opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar

successful response

Container name: spark-kubernetes-driver
	 Container image: zhixingheyitian/spark:spark2.4.1
	 Container state: Terminated
	 Exit code: 0
19/04/11 10:49:31 INFO submit.Client: Application spark-pi finished.
19/04/11 10:49:31 INFO util.ShutdownHookManager: Shutdown hook called
19/04/11 10:49:31 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0ac279ba-7032-4cc5-844f-fe83ce923be9

Client 模式 Run

bin/spark-submit \
    --master k8s://https://192.168.99.108:8443 \
    --deploy-mode client \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=1 \
    --conf spark.kubernetes.container.image=zhixingheyitian/spark:spark2.4.1 \
    ./examples/jars/spark-examples_2.11-2.4.1.jar

successful response

19/04/11 10:57:02 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 10:57:02 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/04/11 10:57:02 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.17.0.7, executor 1, partition 0, PROCESS_LOCAL, 7742 bytes)
19/04/11 10:57:04 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.17.0.7:33025 (size: 1232.0 B, free: 413.9 MB)
19/04/11 10:57:04 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 172.17.0.7, executor 1, partition 1, PROCESS_LOCAL, 7744 bytes)
19/04/11 10:57:04 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1931 ms on 172.17.0.7 (executor 1) (1/2)
19/04/11 10:57:04 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 112 ms on 172.17.0.7 (executor 1) (2/2)
19/04/11 10:57:04 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/04/11 10:57:04 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 2.568 s
19/04/11 10:57:04 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 2.640299 s
Pi is roughly 3.142475712378562
19/04/11 10:57:04 INFO server.AbstractConnector: Stopped Spark@50b0bc4c{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
19/04/11 10:57:04 INFO ui.SparkUI: Stopped Spark web UI at http://bdpe822n2:4040
19/04/11 10:57:04 INFO k8s.KubernetesClusterSchedulerBackend: Shutting down all executors
19/04/11 10:57:04 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
19/04/11 10:57:04 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
19/04/11 10:57:04 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/04/11 10:57:04 INFO memory.MemoryStore: MemoryStore cleared
19/04/11 10:57:04 INFO storage.BlockManager: BlockManager stopped
19/04/11 10:57:04 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
19/04/11 10:57:04 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/04/11 10:57:04 INFO spark.SparkContext: Successfully stopped SparkContext
19/04/11 10:57:04 INFO util.ShutdownHookManager: Shutdown hook called
19/04/11 10:57:04 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0a5df120-8597-4b9e-bfe5-f607f1815fba
19/04/11 10:57:04 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-7b458ba7-6e45-47c4-a195-f17dfdc44c49