Spark on Kubernetes

http://spark.apache.org/docs/latest/running-on-kubernetes.html

The Kubernetes scheduler is currently experimental. In future versions, there may be behavioral changes around configuration, container images and entrypoints.

当前还属于实验性

架构图

1.制作镜像且推送

//比如在opt目录下
tar -xzvf spark-2.4.3-bin-hadoop2.6.tgz -C /opt/

//重命名
mv /opt/spark-2.4.3-bin-hadoop2.6 /opt/spark
cd /opt/spark  

//直到私有云仓库地址
./bin/docker-image-tool.sh -r 172.19.7.111/ruozedata -t v2.4.3 build
./bin/docker-image-tool.sh -r 172.19.7.111/ruozedata -t v2.4.3 push

2.查看

[root@ruozedata001 spark]#  docker images |grep spark
172.19.7.111/ruozedata/spark-r        v2.4.3                     fa814bd53474        12 minutes ago      738MB
172.19.7.111/ruozedata/spark-py       v2.4.3                     8ac3cd4cd046        16 minutes ago      444MB
172.19.7.111/ruozedata/spark          v2.4.3                     93dac9112e6a        31 minutes ago      352MB
[root@ruozedata001 spark]# 

打开: harbor https://47.103.136.192 查看是否push成功

3.创建serviceaccount及clusterrolebinding

kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default

4.创建key

kubectl create secret docker-registry sparkkey \
--namespace=default \
--docker-server=172.19.7.111 \
--docker-username=admin \
--docker-password=Harbor12345 \
--docker-email=jepson@ruozedata.com

5.提交

bin/spark-submit \
--master k8s://https://172.19.7.111:6443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=2 \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark  \
--conf spark.kubernetes.container.image.pullSecrets=sparkkey \
--conf spark.kubernetes.container.image=172.19.7.111/ruozedata/spark:v2.4.3 \
--conf spark.kubernetes.driver.pod.name=spark-pi-driver \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar 200

6.排查

kubectl cluster-info
kubectl logs -f spark-pi-driver
kubectl delete pods spark-pi-driver

response from daemon: pull access denied for 172.19.7.111/ruozedata/spark

我们通常是把standalone的Spark部署到K8S

总结

Kubernetes 配置Pod和容器(四)设置CPU和Mem限制https://www.jianshu.com/p/b2866ac8a470