spark on k8s 镜像构建

spark on k8s 基础镜像的构建

背景

这是跑spark on k8s任务的基础镜像，用来指明executor pod的基础镜像

构建步骤

• git clone spark特定的版本（假如是3.0.1版本），克隆完后，执行一下命令进行构建,构建出包含kubernetes模块的可运行包：

   ## spark 3.x兼容hadoop cdh版本，处理冲突
  git cherry-pick 8e8afb3a3468aa743d13e23e10e77e94b772b2ed
  
  ./dev/make-distribution.sh --name 2.6.0-cdh5.13.1  --pip --tgz -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests
  

• 安装并加入必要的jar包
  按照lzoCodec,安装native-lzo library(用来支持lzo),
  把包含libhadoop.so的目录下的文件复制到assembly/target/scala-2.12/jars/hadoop_native
  把包含libgplcompression.so的目录下的文件复制到assembly/target/scala-2.12/jars/native
  把hadoop-lzo-0.4.15-cdh5.13.1.jar复制到assembly/target/scala-2.12/jars
  配置环境变量

  ENV SPARK_DIST_CLASSPATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  ENV LD_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  ENV JAVA_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  

• 修改镜像代码为

  # distribution, the docker build command should be invoked from the top level directory
  # of the Spark distribution. E.g.:
  # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .
  
  RUN set -ex && \
      sed -i 's/http:\/\/deb.\(.*\)/https:\/\/deb.\1/g' /etc/apt/sources.list && \
      apt-get update && \
      ln -s /lib /lib64 && \
      apt install -y bash tini libc6 libpam-modules krb5-user libnss3 && \
      apt-get install liblzo2-dev -y && \
      mkdir -p /opt/spark && \
      mkdir -p /opt/spark/examples && \
      mkdir -p /opt/spark/work-dir && \
      touch /opt/spark/RELEASE && \
      rm /bin/sh && \
      ln -sv /bin/bash /bin/sh && \
      echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
      chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
      rm -rf /var/cache/apt/*
  
  COPY jars /opt/spark/jars
  COPY bin /opt/spark/bin
  COPY sbin /opt/spark/sbin
  COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
  COPY examples /opt/spark/examples
  COPY kubernetes/tests /opt/spark/tests
  COPY data /opt/spark/data
  
  ENV SPARK_HOME /opt/spark
  
  ENV SPARK_DIST_CLASSPATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  ENV LD_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  ENV JAVA_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  RUN ln -s $SPARK_HOME/jars/hadoop-lzo-0.4.15-cdh5.13.1.jar $SPARK_HOME/jars/hadoop-lzo.jar
  WORKDIR /opt/spark/work-dir
  RUN chmod g+w /opt/spark/work-dir
  
  ENTRYPOINT [ "/opt/entrypoint.sh" ]
  

• 构建包含k8s的镜像,执行如下命令：

  ./bin/docker-image-tool.sh -t spark-on-k8s-v3.0.1-cdh-2.6.0-5.13.1 build
  ## 按需进行修改镜像标签
  docker tag spark:spark-on-k8s-v3.0.1-cdh-2.6.0-5.13.1 xxx.xxx.xxx/xxx/spark-on-k8s:v3.0.1-cdh-2.6.0-5.13.1
  

任务镜像的构建

背景

这是用来跑spark on k8s任务的driver端的镜像

构建步骤

• 按照任务要求进行镜像增加
  注意对于spark on k8s client 在dockerfile中需配置

  ENV HADOOP_CONF_DIR=/opt/hadoop/conf
  
  RUN echo '\nexport SPARK_LOCAL_HOSTNAME=${POD_IP}' >> /path/to/spark/conf/spark-env.sh
  

  这样在driver通行的过程中就不会出现executor连接不上driver端的情况，原因是因为driver端以pod的名字作为host，而exeuctor直接访问该host是访问不了的,具体参考spark on k8s 与spark on k8s operator的对比

• 配置spark-default.conf
  在/path/to/spark/conf/spark-default.conf配置：

  spark.kubernetes.namespace                              dev
  spark.kubernetes.authenticate.driver.serviceAccountName lijiahong
  spark.kubernetes.authenticate.serviceAccountName        lijiahong
  ## 注意这里是之前构建的spark-on-k8s的基础镜像，如果是以cluster形式运行，则driver和executor的镜像分开配置
  ## spark.kubernetes.driver.container.image 
  ## spark.kubernetes.executor.container.image	
  spark.kubernetes.container.image                        xxx.xxx.xxx./xxx/spark-on-k8s:v3.0.1-cdh-2.6.0-5.13.1
  spark.kubernetes.container.image.pullSecrets            regsecret
  spark.kubernetes.file.upload.path                       hdfs://tmp
  spark.kubernetes.container.image.pullPolicy             Always
  

• 构建镜像

  docker build -f Dockerfile --pull -t "xxx/xxx/spark-on-k8s:xxx" .
  

• 提交任务的时候设置POD_IP
  如以下yaml文件：

  apiVersion: v1
  kind: Pod
  metadata:
  name: spark-on-k8s-demo
  labels:
    name: spark-on-k8s-demo
  spec:
   containers:
    - name: spark-on-k8s-demo
      image: xxx/xxx/spark-on-k8s:xxx
      imagePullPolicy: Always
      env:
      - name: POD_IP
        valueFrom:
          fieldRef:
              fieldPath: status.podIP
      - name: NODE_IP
        valueFrom:
          fieldRef:
              fieldPath: status.hostIP
   imagePullSecrets:
    - name: regsecret
   restartPolicy: Never
  

至此，spark on k8s构建完毕