如果关注 Spark 社区的同学已经知道，从 Spark 3.0 开始，就不再支持用 Scala 2.11 来编译了，Scala 2.11 相关的依赖也被 Owen 去掉了，所以如果要用 Spark 3.0 的同学就需要用 Scala 2.12 去编译打包了。
而如果还在用 Spark 2.x 版本的，至少在 2.4.5 的版本中，已经是将 Scala 2.12 作为默认的 Scala 版本了，如果用户的程序是用 Scala 2.11 写的，也需要将 Spark 用 Scala 2.11 编一次，而 Maven 的仓库里应该只有 Scala 2.12 编出来的 jar 依赖可以供下载，所以如果你的需求是下面那样的：
可以用 Spark 2.4.5，并且用户的代码是用 Scala 2.11 作为依赖的
那么就需要自己用 Scala 2.11 编一次 Spark 的依赖。具体的流程也比较简单，Spark 有提供一个脚本来修改项目 pom.xml 文件关于 Scala 依赖版本的工具，用户尽量不要自己去找 pom.xml 里的 Scala 依赖版本，还挺容易找错找漏的，用脚本 change-scala-version.sh 就可以了。
下面是我从编译打包到构建镜像完整流程的日志，可以供大家参考一下。
➜ spark git:(v2.4.5) dev/change-scala-version.sh --help
Usage: change-scala-version.sh [-h|--help] <version>
where :
-h| --help Display this help text
valid version values : 2.11 2.12
➜ spark git:(v2.4.5) dev/change-scala-version.sh 2.11
dev/../hadoop-cloud/pom.xml
dev/../assembly/pom.xml
dev/../repl/pom.xml
dev/../core/pom.xml
dev/../external/spark-ganglia-lgpl/pom.xml
dev/../external/flume/pom.xml
dev/../external/kafka-0-8/pom.xml
dev/../external/kafka-0-10-sql/pom.xml
dev/../external/kafka-0-8-assembly/pom.xml
dev/../external/kinesis-asl-assembly/pom.xml
dev/../external/flume-assembly/pom.xml
dev/../external/kinesis-asl/pom.xml
dev/../external/docker-integration-tests/pom.xml
dev/../external/kafka-0-10-assembly/pom.xml
dev/../external/kafka-0-10/pom.xml
dev/../external/avro/pom.xml
dev/../external/flume-sink/pom.xml
dev/../tools/pom.xml
dev/../streaming/pom.xml
dev/../mllib-local/pom.xml
dev/../launcher/pom.xml
dev/../pom.xml
dev/../examples/pom.xml
dev/../sql/core/pom.xml
dev/../sql/hive-thriftserver/pom.xml
dev/../sql/catalyst/pom.xml
dev/../sql/hive/pom.xml
dev/../common/network-common/pom.xml
dev/../common/tags/pom.xml
dev/../common/kvstore/pom.xml
dev/../common/network-yarn/pom.xml
dev/../common/unsafe/pom.xml
dev/../common/network-shuffle/pom.xml
dev/../common/sketch/pom.xml
dev/../mllib/pom.xml
dev/../resource-managers/kubernetes/core/pom.xml
dev/../resource-managers/kubernetes/integration-tests/pom.xml
dev/../resource-managers/yarn/pom.xml
dev/../resource-managers/mesos/pom.xml
dev/../graphx/pom.xml
dev/../docs/_plugins/copy_api_dirs.rb
➜ spark git:(v2.4.5) dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes
+++ dirname dev/make-distribution.sh
++ cd dev/..
++ pwd
+ SPARK_HOME=/root/tx/spark-all/spark
+ DISTDIR=/root/tx/spark-all/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/root/tx/spark-all/spark/build/mvn
+ (( 3 ))
+ case $1 in
+ MAKE_TGZ=true
+ shift
+ (( 2 ))
+ case $1 in
+ break
+ '[' -z /root/txKona-8.0.0-232 ']'
+ '[' -z /root/txKona-8.0.0-232 ']'
++ command -v git
+ '[' /usr/bin/git ']'
++ git rev-parse --short HEAD
+ GITREV=cee4ecb
+ '[' '!' -z cee4ecb ']'
+ GITREVSTRING=' (git revision cee4ecb)'
+ unset GITREV
++ command -v /root/tx/spark-all/spark/build/mvn
+ '[' '!' /root/tx/spark-all/spark/build/mvn ']'
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=project.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ VERSION=2.4.5
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=scala.binary.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ SCALA_VERSION=2.11
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=hadoop.version -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ tail -n 1
+ SPARK_HADOOP_VERSION=2.7.3
++ /root/tx/spark-all/spark/build/mvn help:evaluate -Dexpression=project.activeProfiles -pl sql/hive -Phadoop-2.7 -Pkubernetes
++ grep -v INFO
++ grep -v WARNING
++ fgrep --count '<id>hive</id>'
++ echo -n
+ SPARK_HIVE=0
+ '[' none == none ']'
+ NAME=2.7.3
+ echo 'Spark version is 2.4.5'
Spark version is 2.4.5
+ '[' true == true ']'
+ echo 'Making spark-2.4.5-bin-2.7.3.tgz'
Making spark-2.4.5-bin-2.7.3.tgz
+ cd /root/tx/spark-all/spark
+ export 'MAVEN_OPTS=-Xmx2g -XX:ReservedCodeCacheSize=1g'
+ MAVEN_OPTS='-Xmx2g -XX:ReservedCodeCacheSize=1g'
+ BUILD_COMMAND=("$MVN" -T 1C clean package -DskipTests $@)
+ echo -e '\nBuilding with...'
Building with...
+ echo -e '$ /root/tx/spark-all/spark/build/mvn' -T 1C clean package -DskipTests -Phadoop-2.7 '-Pkubernetes\n'
$ /root/tx/spark-all/spark/build/mvn -T 1C clean package -DskipTests -Phadoop-2.7 -Pkubernetes
+ /root/tx/spark-all/spark/build/mvn -T 1C clean package -DskipTests -Phadoop-2.7 -Pkubernetes
Using `mvn` from path: /root/apache-maven-3.6.2/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM [pom]
[INFO] Spark Project Tags [jar]
[INFO] Spark Project Sketch [jar]
[INFO] Spark Project Local DB [jar]
[INFO] Spark Project Networking [jar]
[INFO] Spark Project Shuffle Streaming Service [jar]
[INFO] Spark Project Unsafe [jar]
[INFO] Spark Project Launcher [jar]
[INFO] Spark Project Core [jar]
[INFO] Spark Project ML Local Library [jar]
[INFO] Spark Project GraphX [jar]
[INFO] Spark Project Streaming [jar]
[INFO] Spark Project Catalyst [jar]
[INFO] Spark Project SQL [jar]
[INFO] Spark Project ML Library [jar]
[INFO] Spark Project Tools [jar]
[INFO] Spark Project Hive [jar]
[INFO] Spark Project REPL [jar]
[INFO] Spark Project Kubernetes [jar]
[INFO] Spark Project Assembly [pom]
[INFO] Spark Integration for Kafka 0.10 [jar]
[INFO] Kafka 0.10+ Source for Structured Streaming [jar]
[INFO] Spark Project Examples [jar]
[INFO] Spark Integration for Kafka 0.10 Assembly [jar]
[INFO] Spark Avro [jar]
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 8
[INFO]
.....省略很多
.....省略很多
.....省略很多
.....省略很多
.....省略很多
[INFO] -----------------< org.apache.spark:spark-parent_2.11 >-----------------
[INFO] Building Spark Project Parent POM 2.4.5 [1/25]
[INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork (create-source-jar) @ spark-examples_2.11 ---
[INFO] Building jar: /root/tx/spark-all/spark/examples/target/spark-examples_2.11-2.4.5-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.5:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 47.105 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 27.619 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.121 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 13.466 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.274 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.603 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 13.363 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 7.332 s]
[INFO] Spark Project Core ................................. SUCCESS [03:08 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 20.801 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 17.927 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 34.502 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:58 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:38 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:02 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 26.929 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 50.893 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 4.578 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 16.327 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 5.431 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 15.950 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 14.692 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 18.164 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 4.847 s]
[INFO] Spark Avro ......................................... SUCCESS [ 10.145 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:40 min (Wall Clock)
[INFO] Finished at: 2020-07-27T09:11:44+08:00
[INFO] ------------------------------------------------------------------------
+ rm -rf /root/tx/spark-all/spark/dist
+ mkdir -p /root/tx/spark-all/spark/dist/jars
+ echo 'Spark 2.4.5 (git revision cee4ecb) built for Hadoop 2.7.3'
+ echo 'Build flags: -Phadoop-2.7' -Pkubernetes
+ cp /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/activation-1.1.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/aircompressor-0.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/antlr4-runtime-4.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/aopalliance-repackaged-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/apacheds-i18n-2.0.0-M15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/apacheds-kerberos-codec-2.0.0-M15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/api-asn1-api-1.0.0-M20.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/api-util-1.0.0-M20.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arpack_combined_all-0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-format-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-memory-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/arrow-vector-0.10.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/automaton-1.11-8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-1.8.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-ipc-1.8.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/avro-mapred-1.8.2-hadoop2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/breeze_2.11-0.13.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/breeze-macros_2.11-0.13.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/chill_2.11-0.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/chill-java-0.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-beanutils-1.9.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-cli-1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-codec-1.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-collections-3.2.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-compiler-3.0.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-compress-1.8.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-configuration-1.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-crypto-1.0.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-digester-1.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-httpclient-3.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-io-2.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-lang-2.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-lang3-3.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-math3-3.4.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/commons-net-3.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/compress-lzf-1.0.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/core-1.1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-client-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-framework-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/curator-recipes-2.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/flatbuffers-1.2.0-3f79e055.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/generex-1.0.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/gson-2.2.4.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/guava-14.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-annotations-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-auth-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-client-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-hdfs-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-app-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-core-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-api-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-client-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hadoop-yarn-server-common-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-api-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-locator-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hk2-utils-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/hppc-0.7.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/htrace-core-3.1.0-incubating.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/httpclient-4.5.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/httpcore-4.4.10.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/ivy-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-annotations-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-core-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-core-asl-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-databind-2.6.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-dataformat-yaml-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-jaxrs-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-mapper-asl-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-jaxb-annotations-2.6.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-paranamer-2.7.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-module-scala_2.11-2.6.7.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jackson-xc-1.9.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/janino-3.0.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javassist-3.18.1-GA.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.annotation-api-1.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.inject-2.4.0-b34.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.servlet-api-3.1.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/javax.ws.rs-api-2.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jaxb-api-2.2.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jcl-over-slf4j-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-client-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-common-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-container-servlet-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-container-servlet-core-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-guava-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-media-jaxb-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jersey-server-2.22.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jetty-util-6.1.26.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/joda-time-2.9.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-ast_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-core_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-jackson_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/json4s-scalap_2.11-3.5.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jsp-api-2.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jsr305-1.3.9.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jtransforms-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/jul-to-slf4j-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kryo-shaded-4.0.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-client-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-model-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/kubernetes-model-common-4.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/leveldbjni-all-1.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/log4j-1.2.17.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/logging-interceptor-3.12.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/lz4-java-1.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/machinist_2.11-0.6.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/macro-compat_2.11-1.1.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-core-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-graphite-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-json-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/metrics-jvm-3.1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/minlog-1.3.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/netty-3.9.9.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/netty-all-4.1.42.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/objenesis-2.5.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/okhttp-3.12.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/okio-1.15.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/opencsv-2.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-core-1.5.5-nohive.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-mapreduce-1.5.5-nohive.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/orc-shims-1.5.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/oro-2.0.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/osgi-resource-locator-1.0.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/paranamer-2.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-column-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-common-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-encoding-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-format-2.4.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-hadoop-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/parquet-jackson-1.10.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/protobuf-java-2.5.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/py4j-0.10.7.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/pyrolite-4.13.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/RoaringBitmap-0.7.45.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-compiler-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-library-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-parser-combinators_2.11-1.1.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-reflect-2.11.12.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/scala-xml_2.11-1.0.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/shapeless_2.11-2.3.2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/shims-0.7.45.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/slf4j-api-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/slf4j-log4j12-1.7.16.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/snakeyaml-1.15.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/snappy-java-1.1.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-catalyst_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-core_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-graphx_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-kubernetes_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-kvstore_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-launcher_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-mllib_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-mllib-local_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-network-common_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-network-shuffle_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-repl_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-sketch_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-sql_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-streaming_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-tags_2.11-2.4.5-tests.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spark-unsafe_2.11-2.4.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spire_2.11-0.13.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/spire-macros_2.11-0.13.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/stax-api-1.0-2.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/stream-2.7.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/univocity-parsers-2.7.3.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/validation-api-1.1.0.Final.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xbean-asm6-shaded-4.8.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xercesImpl-2.9.1.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xmlenc-0.52.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/xz-1.5.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zjsonpatch-0.3.0.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zookeeper-3.4.6.jar /root/tx/spark-all/spark/assembly/target/scala-2.11/jars/zstd-jni-1.3.2-2.jar /root/tx/spark-all/spark/dist/jars/
+ '[' -f '/root/tx/spark-all/spark/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar' ']'
+ '[' -d /root/tx/spark-all/spark/resource-managers/kubernetes/core/target/ ']'
+ mkdir -p /root/tx/spark-all/spark/dist/kubernetes/
+ cp -a /root/tx/spark-all/spark/resource-managers/kubernetes/docker/src/main/dockerfiles /root/tx/spark-all/spark/dist/kubernetes/
+ cp -a /root/tx/spark-all/spark/resource-managers/kubernetes/integration-tests/tests /root/tx/spark-all/spark/dist/kubernetes/
+ mkdir -p /root/tx/spark-all/spark/dist/examples/jars
+ cp /root/tx/spark-all/spark/examples/target/scala-2.11/jars/aircompressor-0.10.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/commons-codec-1.10.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/commons-lang-2.6.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-core-1.5.5-nohive.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-mapreduce-1.5.5-nohive.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/orc-shims-1.5.5.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/paranamer-2.8.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/scopt_2.11-3.7.0.jar /root/tx/spark-all/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.4.5.jar /root/tx/spark-all/spark/dist/examples/jars
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/aircompressor-0.10.jar
+ name=aircompressor-0.10.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/aircompressor-0.10.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/aircompressor-0.10.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/commons-codec-1.10.jar
+ name=commons-codec-1.10.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/commons-codec-1.10.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/commons-codec-1.10.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/commons-lang-2.6.jar
+ name=commons-lang-2.6.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/commons-lang-2.6.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/commons-lang-2.6.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-core-1.5.5-nohive.jar
+ name=orc-core-1.5.5-nohive.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-core-1.5.5-nohive.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-core-1.5.5-nohive.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-mapreduce-1.5.5-nohive.jar
+ name=orc-mapreduce-1.5.5-nohive.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-mapreduce-1.5.5-nohive.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-mapreduce-1.5.5-nohive.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/orc-shims-1.5.5.jar
+ name=orc-shims-1.5.5.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/orc-shims-1.5.5.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/orc-shims-1.5.5.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/paranamer-2.8.jar
+ name=paranamer-2.8.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/paranamer-2.8.jar ']'
+ rm /root/tx/spark-all/spark/dist/examples/jars/paranamer-2.8.jar
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/scopt_2.11-3.7.0.jar
+ name=scopt_2.11-3.7.0.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/scopt_2.11-3.7.0.jar ']'
+ for f in '"$DISTDIR"/examples/jars/*'
++ basename /root/tx/spark-all/spark/dist/examples/jars/spark-examples_2.11-2.4.5.jar
+ name=spark-examples_2.11-2.4.5.jar
+ '[' -f /root/tx/spark-all/spark/dist/jars/spark-examples_2.11-2.4.5.jar ']'
+ mkdir -p /root/tx/spark-all/spark/dist/examples/src/main
+ cp -r /root/tx/spark-all/spark/examples/src/main /root/tx/spark-all/spark/dist/examples/src/
+ '[' -e /root/tx/spark-all/spark/LICENSE-binary ']'
+ cp /root/tx/spark-all/spark/LICENSE-binary /root/tx/spark-all/spark/dist/LICENSE
+ cp -r /root/tx/spark-all/spark/licenses-binary /root/tx/spark-all/spark/dist/licenses
+ cp /root/tx/spark-all/spark/NOTICE-binary /root/tx/spark-all/spark/dist/NOTICE
+ '[' -e /root/tx/spark-all/spark/CHANGES.txt ']'
+ cp -r /root/tx/spark-all/spark/data /root/tx/spark-all/spark/dist
+ '[' false == true ']'
+ echo 'Skipping building python distribution package'
Skipping building python distribution package
+ '[' false == true ']'
+ echo 'Skipping building R source package'
Skipping building R source package
+ mkdir /root/tx/spark-all/spark/dist/conf
+ cp /root/tx/spark-all/spark/conf/docker.properties.template /root/tx/spark-all/spark/conf/fairscheduler.xml.template /root/tx/spark-all/spark/conf/log4j.properties.template /root/tx/spark-all/spark/conf/metrics.properties.template /root/tx/spark-all/spark/conf/slaves.template /root/tx/spark-all/spark/conf/spark-defaults.conf.template /root/tx/spark-all/spark/conf/spark-env.sh.template /root/tx/spark-all/spark/dist/conf
+ cp /root/tx/spark-all/spark/README.md /root/tx/spark-all/spark/dist
+ cp -r /root/tx/spark-all/spark/bin /root/tx/spark-all/spark/dist
+ cp -r /root/tx/spark-all/spark/python /root/tx/spark-all/spark/dist
+ '[' false == true ']'
+ cp -r /root/tx/spark-all/spark/sbin /root/tx/spark-all/spark/dist
+ '[' -d /root/tx/spark-all/spark/R/lib/SparkR ']'
+ '[' true == true ']'
+ TARDIR_NAME=spark-2.4.5-bin-2.7.3
+ TARDIR=/root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ rm -rf /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ cp -r /root/tx/spark-all/spark/dist /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
+ tar czf spark-2.4.5-bin-2.7.3.tgz -C /root/tx/spark-all/spark spark-2.4.5-bin-2.7.3
+ rm -rf /root/tx/spark-all/spark/spark-2.4.5-bin-2.7.3
➜ spark git:(v2.4.5) tar zxvf spark-2.4.5-bin-2.7.3.tgz
...省略很多
...省略很多
...省略很多
➜ spark git:(v2.4.5) ✗ cd spark-2.4.5-bin-2.7.3
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ bin/docker-image-tool.sh -r yourhub.com/pulic -t 2.11_2.4.5 build
Sending build context to Docker daemon 201.1MB
Step 1/15 : FROM openjdk:8-jdk-slim
---> a0b0dea106dd
Step 2/15 : ARG spark_jars=jars
---> Running in 788ed25e1956
Removing intermediate container 788ed25e1956
---> db4a6a8ae0d8
Step 3/15 : ARG img_path=kubernetes/dockerfiles
---> Running in fd77ef759bed
Removing intermediate container fd77ef759bed
---> 689207604eb1
Step 4/15 : ARG k8s_tests=kubernetes/tests
---> Running in fe4d6784371b
Removing intermediate container fe4d6784371b
---> 4ac25b12085b
Step 5/15 : RUN set -ex && apt-get update && ln -s /lib /lib64 && apt install -y bash tini libc6 libpam-modules libnss3 && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd && rm -rf /var/cache/apt/*
---> Running in ea60b610c182
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 Packages [213 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
^C
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ vi kubernetes/dockerfiles/spark/Dockerfile
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ bin/docker-image-tool.sh -r yourhub.com/pulic -t 2.11_2.4.5 build
Sending build context to Docker daemon 201.1MB
Step 1/15 : FROM openjdk:8-jdk-slim
---> a0b0dea106dd
Step 2/15 : ARG spark_jars=jars
---> Using cache
---> db4a6a8ae0d8
Step 3/15 : ARG img_path=kubernetes/dockerfiles
---> Using cache
---> 689207604eb1
Step 4/15 : ARG k8s_tests=kubernetes/tests
---> Using cache
---> 4ac25b12085b
Step 5/15 : RUN set -ex && apt-get update && ln -s /lib /lib64 && apt install -y bash tini libc6 libpam-modules libnss3 && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd && rm -rf /var/cache/apt/*
---> Running in 3e9d1d5ac7c1
+ apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://security.debian.org/debian-security buster/updates/main amd64 Packages [213 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7868 B]
Fetched 8365 kB in 3s (3140 kB/s)
Reading package lists...
+ ln -s /lib /lib64
+ apt install -y bash tini libc6 libpam-modules libnss3
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
bash is already the newest version (5.0-4).
bash set to manually installed.
libc6 is already the newest version (2.28-10).
libc6 set to manually installed.
libpam-modules is already the newest version (1.3.1-5).
libpam-modules set to manually installed.
The following package was automatically installed and is no longer required:
lsb-base
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
libnspr4 libsqlite3-0
The following NEW packages will be installed:
libnspr4 libnss3 libsqlite3-0 tini
0 upgraded, 4 newly installed, 0 to remove and 15 not upgraded.
Need to get 2159 kB of archives.
After this operation, 6175 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 libnss3 amd64 2:3.42.1-1+deb10u3 [1159 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libnspr4 amd64 2:4.20-1 [112 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libsqlite3-0 amd64 3.27.2-3 [641 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 tini amd64 0.18.0-1 [247 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2159 kB in 1s (3301 kB/s)
Selecting previously unselected package libnspr4:amd64.
(Reading database ... 6888 files and directories currently installed.)
Preparing to unpack .../libnspr4_2%3a4.20-1_amd64.deb ...
Unpacking libnspr4:amd64 (2:4.20-1) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../libsqlite3-0_3.27.2-3_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.27.2-3) ...
Selecting previously unselected package libnss3:amd64.
Preparing to unpack .../libnss3_2%3a3.42.1-1+deb10u3_amd64.deb ...
Unpacking libnss3:amd64 (2:3.42.1-1+deb10u3) ...
Selecting previously unselected package tini.
Preparing to unpack .../tini_0.18.0-1_amd64.deb ...
Unpacking tini (0.18.0-1) ...
Setting up libsqlite3-0:amd64 (3.27.2-3) ...
Setting up libnspr4:amd64 (2:4.20-1) ...
Setting up tini (0.18.0-1) ...
Setting up libnss3:amd64 (2:3.42.1-1+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
+ mkdir -p /opt/spark
+ mkdir -p /opt/spark/work-dir
+ touch /opt/spark/RELEASE
+ rm /bin/sh
+ ln -sv /bin/bash /bin/sh
'/bin/sh' -> '/bin/bash'
+ echo auth required pam_wheel.so use_uid
+ chgrp root /etc/passwd
+ chmod ug+rw /etc/passwd
+ rm -rf /var/cache/apt/archives
Removing intermediate container 3e9d1d5ac7c1
---> 1ffbc3c2f99f
Step 6/15 : COPY ${spark_jars} /opt/spark/jars
---> 9a4610c4c102
Step 7/15 : COPY bin /opt/spark/bin
---> 06c908997a9f
Step 8/15 : COPY sbin /opt/spark/sbin
---> b32be7e7dda4
Step 9/15 : COPY ${img_path}/spark/entrypoint.sh /opt/
---> 6bee90987e7f
Step 10/15 : COPY examples /opt/spark/examples
---> 65775aca9747
Step 11/15 : COPY ${k8s_tests} /opt/spark/tests
---> 6768a8d5c69a
Step 12/15 : COPY data /opt/spark/data
---> 1520647faf15
Step 13/15 : ENV SPARK_HOME /opt/spark
---> Running in 7688890172f1
Removing intermediate container 7688890172f1
---> bf2dd6442c38
Step 14/15 : WORKDIR /opt/spark/work-dir
---> Running in b42600a07aaf
Removing intermediate container b42600a07aaf
---> 96c18bdcae55
Step 15/15 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
---> Running in 62490f481000
Removing intermediate container 62490f481000
---> c016374ced2b
Successfully built c016374ced2b
Successfully tagged yourhub.com/pulic/spark:2.11_2.4.5
Sending build context to Docker daemon 201.1MB
Step 1/9 : ARG base_img
Step 2/9 : FROM $base_img
---> c016374ced2b
Step 3/9 : WORKDIR /
---> Running in ebdd964393d8
Removing intermediate container ebdd964393d8
---> 76ca1b8a91e3
Step 4/9 : RUN mkdir ${SPARK_HOME}/python
---> Running in 3073d6d8a3f7
Removing intermediate container 3073d6d8a3f7
---> 176a7fb978a3
Step 5/9 : RUN apt install -y python python-pip && apt install -y python3 python3-pip && rm -r /usr/lib/python*/ensurepip && pip install --upgrade pip setuptools && rm -r /root/.cache && rm -rf /var/cache/apt/*
---> Running in 4a5fdb7337fa
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ ls
bin data jars LICENSE NOTICE README.md sbin
conf examples kubernetes licenses python RELEASE
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker images | grep -i "2.11_2.4.5"
yourhub.com/pulic/spark-py 2.11_2.4.5 c7190686b340 About a minute ago 1GB
yourhub.com/pulic/spark 2.11_2.4.5 c016374ced2b 2 minutes ago 504MB
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker push yourhub.com/pulic/spark:2.11_2.4.5
The push refers to repository [yourhub.com/pulic/spark]
7006d88e7365: Pushed
cbcceeb16130: Pushed
d18039eb9e2c: Pushed
2e4b66a0a7ba: Pushed
e94624e3886b: Pushed
ba738a43f5a4: Pushed
2c069a31829d: Pushed
da53a3e76eb8: Pushed
34f9780c8bfb: Mounted from user/dev-spark-operator
e871e75299d2: Mounted from user/dev-spark-operator
109e67eff29c: Mounted from user/dev-spark-operator
556c5fb0d91b: Mounted from user/dev-spark-operator
2.11_2.4.5: digest: sha256:60bccd043dd4d16c7cc2b0879c6de394d1d48561f624082ddd627af2f9c96084 size: 2838
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker push yourhub.com/pulic/spark-py:2.11_2.4.5
The push refers to repository [yourhub.com/pulic/spark-py]
fb728a4376ed: Pushed
19442796254b: Pushed
bf5c82549be3: Pushed
7006d88e7365: Mounted from pulic/spark
cbcceeb16130: Mounted from pulic/spark
d18039eb9e2c: Mounted from pulic/spark
2e4b66a0a7ba: Mounted from pulic/spark
e94624e3886b: Mounted from pulic/spark
ba738a43f5a4: Mounted from pulic/spark
2c069a31829d: Mounted from pulic/spark
da53a3e76eb8: Mounted from pulic/spark
34f9780c8bfb: Mounted from pulic/spark
e871e75299d2: Mounted from pulic/spark
109e67eff29c: Mounted from pulic/spark
556c5fb0d91b: Mounted from pulic/spark
2.11_2.4.5: digest: sha256:49c0b17c8addc21bd30735ccd6cdc88c3c57e826f1b138cbedd1b13fa23258f8 size: 3469
➜ spark-2.4.5-bin-2.7.3 git:(v2.4.5) ✗ docker run -it yourhub.com/pulic/spark:2.11_2.4.5 bash
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_K8S_CMD=bash
+ case "$SPARK_K8S_CMD" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ exec /usr/bin/tini -s -- bash
root@a3f20a0deb94:/opt/spark/work-dir#
root@a3f20a0deb94:/opt/spark/work-dir# ls
root@a3f20a0deb94:/opt/spark/work-dir# ls
root@a3f20a0deb94:/opt/spark/work-dir# cd ..
root@a3f20a0deb94:/opt/spark# ls
RELEASE bin data examples jars sbin tests work-dir
root@a3f20a0deb94:/opt/spark# ls jars/
RoaringBitmap-0.7.45.jar hk2-locator-2.4.0-b34.jar objenesis-2.5.1.jar
activation-1.1.1.jar hk2-utils-2.4.0-b34.jar okhttp-3.12.0.jar
aircompressor-0.10.jar hppc-0.7.2.jar okio-1.15.0.jar
antlr4-runtime-4.7.jar htrace-core-3.1.0-incubating.jar opencsv-2.3.jar
aopalliance-repackaged-2.4.0-b34.jar httpclient-4.5.6.jar orc-core-1.5.5-nohive.jar
apacheds-i18n-2.0.0-M15.jar httpcore-4.4.10.jar orc-mapreduce-1.5.5-nohive.jar
apacheds-kerberos-codec-2.0.0-M15.jar ivy-2.4.0.jar orc-shims-1.5.5.jar
api-asn1-api-1.0.0-M20.jar jackson-annotations-2.6.7.jar oro-2.0.8.jar
api-util-1.0.0-M20.jar jackson-core-2.6.7.jar osgi-resource-locator-1.0.1.jar
arpack_combined_all-0.1.jar jackson-core-asl-1.9.13.jar paranamer-2.8.jar
arrow-format-0.10.0.jar jackson-databind-2.6.7.3.jar parquet-column-1.10.1.jar
arrow-memory-0.10.0.jar jackson-dataformat-yaml-2.6.7.jar parquet-common-1.10.1.jar
arrow-vector-0.10.0.jar jackson-jaxrs-1.9.13.jar parquet-encoding-1.10.1.jar
automaton-1.11-8.jar jackson-mapper-asl-1.9.13.jar parquet-format-2.4.0.jar
avro-1.8.2.jar jackson-module-jaxb-annotations-2.6.7.jar parquet-hadoop-1.10.1.jar
avro-ipc-1.8.2.jar jackson-module-paranamer-2.7.9.jar parquet-jackson-1.10.1.jar
avro-mapred-1.8.2-hadoop2.jar jackson-module-scala_2.11-2.6.7.1.jar protobuf-java-2.5.0.jar
breeze-macros_2.11-0.13.2.jar jackson-xc-1.9.13.jar py4j-0.10.7.jar
breeze_2.11-0.13.2.jar janino-3.0.9.jar pyrolite-4.13.jar
chill-java-0.9.3.jar javassist-3.18.1-GA.jar scala-compiler-2.11.12.jar
chill_2.11-0.9.3.jar javax.annotation-api-1.2.jar scala-library-2.11.12.jar
commons-beanutils-1.9.4.jar javax.inject-2.4.0-b34.jar scala-parser-combinators_2.11-1.1.0.jar
commons-cli-1.2.jar javax.servlet-api-3.1.0.jar scala-reflect-2.11.12.jar
commons-codec-1.10.jar javax.ws.rs-api-2.0.1.jar scala-xml_2.11-1.0.5.jar
commons-collections-3.2.2.jar jaxb-api-2.2.2.jar shapeless_2.11-2.3.2.jar
commons-compiler-3.0.9.jar jcl-over-slf4j-1.7.16.jar shims-0.7.45.jar
commons-compress-1.8.1.jar jersey-client-2.22.2.jar slf4j-api-1.7.16.jar
commons-configuration-1.6.jar jersey-common-2.22.2.jar slf4j-log4j12-1.7.16.jar
commons-crypto-1.0.0.jar jersey-container-servlet-2.22.2.jar snakeyaml-1.15.jar
commons-digester-1.8.jar jersey-container-servlet-core-2.22.2.jar snappy-java-1.1.7.3.jar
commons-httpclient-3.1.jar jersey-guava-2.22.2.jar spark-catalyst_2.11-2.4.5.jar
commons-io-2.4.jar jersey-media-jaxb-2.22.2.jar spark-core_2.11-2.4.5.jar
commons-lang-2.6.jar jersey-server-2.22.2.jar spark-graphx_2.11-2.4.5.jar
commons-lang3-3.5.jar jetty-util-6.1.26.jar spark-kubernetes_2.11-2.4.5.jar
commons-math3-3.4.1.jar joda-time-2.9.3.jar spark-kvstore_2.11-2.4.5.jar
commons-net-3.1.jar json4s-ast_2.11-3.5.3.jar spark-launcher_2.11-2.4.5.jar
compress-lzf-1.0.3.jar json4s-core_2.11-3.5.3.jar spark-mllib-local_2.11-2.4.5.jar
core-1.1.2.jar json4s-jackson_2.11-3.5.3.jar spark-mllib_2.11-2.4.5.jar
curator-client-2.7.1.jar json4s-scalap_2.11-3.5.3.jar spark-network-common_2.11-2.4.5.jar
curator-framework-2.7.1.jar jsp-api-2.1.jar spark-network-shuffle_2.11-2.4.5.jar
curator-recipes-2.7.1.jar jsr305-1.3.9.jar spark-repl_2.11-2.4.5.jar
flatbuffers-1.2.0-3f79e055.jar jtransforms-2.4.0.jar spark-sketch_2.11-2.4.5.jar
generex-1.0.2.jar jul-to-slf4j-1.7.16.jar spark-sql_2.11-2.4.5.jar
gson-2.2.4.jar kryo-shaded-4.0.2.jar spark-streaming_2.11-2.4.5.jar
guava-14.0.1.jar kubernetes-client-4.6.1.jar spark-tags_2.11-2.4.5-tests.jar
hadoop-annotations-2.7.3.jar kubernetes-model-4.6.1.jar spark-tags_2.11-2.4.5.jar
hadoop-auth-2.7.3.jar kubernetes-model-common-4.6.1.jar spark-unsafe_2.11-2.4.5.jar
hadoop-client-2.7.3.jar leveldbjni-all-1.8.jar spire-macros_2.11-0.13.0.jar
hadoop-common-2.7.3.jar log4j-1.2.17.jar spire_2.11-0.13.0.jar
hadoop-hdfs-2.7.3.jar logging-interceptor-3.12.0.jar stax-api-1.0-2.jar
hadoop-mapreduce-client-app-2.7.3.jar lz4-java-1.4.0.jar stream-2.7.0.jar
hadoop-mapreduce-client-common-2.7.3.jar machinist_2.11-0.6.1.jar univocity-parsers-2.7.3.jar
hadoop-mapreduce-client-core-2.7.3.jar macro-compat_2.11-1.1.1.jar validation-api-1.1.0.Final.jar
hadoop-mapreduce-client-jobclient-2.7.3.jar metrics-core-3.1.5.jar xbean-asm6-shaded-4.8.jar
hadoop-mapreduce-client-shuffle-2.7.3.jar metrics-graphite-3.1.5.jar xercesImpl-2.9.1.jar
hadoop-yarn-api-2.7.3.jar metrics-json-3.1.5.jar xmlenc-0.52.jar
hadoop-yarn-client-2.7.3.jar metrics-jvm-3.1.5.jar xz-1.5.jar
hadoop-yarn-common-2.7.3.jar minlog-1.3.0.jar zjsonpatch-0.3.0.jar
hadoop-yarn-server-common-2.7.3.jar netty-3.9.9.Final.jar zookeeper-3.4.6.jar
hk2-api-2.4.0-b34.jar
扫一扫
1246
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
