Apache Spark源码阅读环境搭建

已于 2022-04-29 13:52:31 修改
阅读量660 收藏 1
点赞数 1
分类专栏： # Apache Spark 大数据 文章标签： spark java apache spark 大数据
于 2021-08-26 10:52:08 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_39945938/article/details/119923858
版权

文章目录

版本
win7
jdk 1.8
maven 3.6.3
scala 2.11.8

1 下载源码

# 下载源码
git clone https://github.com/apache/spark.git
# 切换到目标版本
git checkout v2.4.0

在项目根目录打开Git Bash
运行如下命令

export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
./build/mvn -Phadoop-2.6 -Dhadoop.version=2.6.0  -Pyarn -Phive -Phive-thriftserver -DskipTests clean  package

2 导入项目
File
New
Project from existing Sources...
选择clone的项目根目录
点击OK

在这里插入图片描述

3 新建文件

在spark\examples\src\main\resources\目录下创建文件名为spark-version-info.properties的文件
内容如下

λ ~/workspace/big_data/spark/ master* ./build/spark-build-info ./core/target/extra-resources 2.1.1
λ ~/workspace/big_data/spark/ master* cat ./core/target/extra-resources/spark-version-info.properties
# 这里的版本号需要改为对应版本
version=2.4.6
user=test
revision=dec9aa3b37c01454065a4d8899859991f43d4c66
branch=master
date=2017-06-07T15:12:48Z
url=https://github.com/apache/spark

在F:\tmp新建名为words.txt文件，内容如下

java python
java spark
spark java

4 Debug JavaWordCount

4.1 搜索JavaWordCount

双击Shift，输入JavaWordCount，点击回车

4.2 修改参数

右键，Run JavaWordCount.main()，这里会报一个没有master的错
点击Edit Configurations，修改如下参数

  • VM Options=-Dspark.master=local[2]
  • Program arguments=F:\tmp\words.txt
  • Use classpath of module 点击选中 Include dependencies with “Provideed” scope

修改后再次Run JavaWordCount.main()

运行日志：

D:\Java\jdk1.8.0_121\bin\java -Dspark.master=local[2] -javaagent:D:\IntelliJ_IDEA\idea2018\lib\idea_rt.jar=59567:D:\IntelliJ_IDEA\idea2018\bin -Dfile.encoding=UTF-8 -classpath D:\Java\jdk1.8.0_121\jre\lib\charsets.jar;D:\Java\jdk1.8.0_121\jre\lib\deploy.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\access-bridge-64.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\cldrdata.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\dnsns.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\jaccess.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\jfxrt.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\localedata.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\nashorn.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\sunec.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\sunjce_provider.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\sunmscapi.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\sunpkcs11.jar;D:\Java\jdk1.8.0_121\jre\lib\ext\zipfs.jar;D:\Java\jdk1.8.0_121\jre\lib\javaws.jar;D:\Java\jdk1.8.0_121\jre\lib\jce.jar;D:\Java\jdk1.8.0_121\jre\lib\jfr.jar;D:\Java\jdk1.8.0_121\jre\lib\jfxswt.jar;D:\Java\jdk1.8.0_121\jre\lib\jsse.jar;D:\Java\jdk1.8.0_121\jre\lib\management-agent.jar;D:\Java\jdk1.8.0_121\jre\lib\plugin.jar;D:\Java\jdk1.8.0_121\jre\lib\resources.jar;D:\Java\jdk1.8.0_121\jre\lib\rt.jar;E:\os_workspace\spark\examples\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\spark-project\spark\unused\1.0.0\unused-1.0.0.jar;E:\os_workspace\spark\core\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\apache\avro\avro\1.8.2\avro-1.8.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;D:\eclipse_boot\repository-maven\repository-maven\com\thoughtworks\paranamer\paranamer\2.8\paranamer-2.8.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\commons\commons-compress\1.8.1\commons-compress-1.8.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\tukaani\xz\1.5\xz-1.5.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\avro\avro-mapred\1.8.2\avro-mapred-1.8.2-hadoop2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\avro\avro-ipc\1.8.2\avro-ipc-1.8.2.jar;D:\eclipse_boot\repository-maven\repository-maven\com\twitter\chill_2.11\0.9.3\chill_2.11-0.9.3.jar;D:\eclipse_boot\repository-maven\repository-maven\com\esotericsoftware\kryo-shaded\4.0.2\kryo-shaded-4.0.2.jar;D:\eclipse_boot\repository-maven\repository-maven\com\esotericsoftware\minlog\1.3.0\minlog-1.3.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\objenesis\objenesis\2.5.1\objenesis-2.5.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\twitter\chill-java\0.9.3\chill-java-0.9.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\xbean\xbean-asm6-shaded\4.8\xbean-asm6-shaded-4.8.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-client\3.1.0\hadoop-client-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-common\3.1.0\hadoop-common-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-beanutils\commons-beanutils\1.9.3\commons-beanutils-1.9.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\commons\commons-configuration2\2.1.1\commons-configuration2-2.1.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\google\re2j\re2j\1.1\re2j-1.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-auth\3.1.0\hadoop-auth-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\nimbusds\nimbus-jose-jwt\4.41.1\nimbus-jose-jwt-4.41.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;D:\eclipse_boot\repository-maven\repository-maven\net\minidev\json-smart\2.3\json-smart-2.3.jar;D:\eclipse_boot\repository-maven\repository-maven\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\curator\curator-client\2.12.0\curator-client-2.12.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\htrace\htrace-core4\4.1.0-incubating\htrace-core4-4.1.0-incubating.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-simplekdc\1.0.1\kerb-simplekdc-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-client\1.0.1\kerb-client-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerby-config\1.0.1\kerby-config-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-core\1.0.1\kerb-core-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerby-pkix\1.0.1\kerby-pkix-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerby-asn1\1.0.1\kerby-asn1-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerby-util\1.0.1\kerby-util-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-common\1.0.1\kerb-common-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-crypto\1.0.1\kerb-crypto-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-util\1.0.1\kerb-util-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\token-provider\1.0.1\token-provider-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-admin\1.0.1\kerb-admin-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-server\1.0.1\kerb-server-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerb-identity\1.0.1\kerb-identity-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kerby\kerby-xdr\1.0.1\kerby-xdr-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-hdfs-client\3.1.0\hadoop-hdfs-client-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\squareup\okhttp\okhttp\2.7.5\okhttp-2.7.5.jar;D:\eclipse_boot\repository-maven\repository-maven\com\squareup\okio\okio\1.6.0\okio-1.6.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-yarn-api\3.1.0\hadoop-yarn-api-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-yarn-client\3.1.0\hadoop-yarn-client-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-mapreduce-client-core\3.1.0\hadoop-mapreduce-client-core-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-yarn-common\3.1.0\hadoop-yarn-common-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\module\jackson-module-jaxb-annotations\2.6.7\jackson-module-jaxb-annotations-2.6.7.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\jaxrs\jackson-jaxrs-json-provider\2.7.8\jackson-jaxrs-json-provider-2.7.8.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\jaxrs\jackson-jaxrs-base\2.7.8\jackson-jaxrs-base-2.7.8.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-mapreduce-client-jobclient\3.1.0\hadoop-mapreduce-client-jobclient-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-mapreduce-client-common\3.1.0\hadoop-mapreduce-client-common-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\hadoop\hadoop-annotations\3.1.0\hadoop-annotations-3.1.0.jar;E:\os_workspace\spark\launcher\target\scala-2.11\classes;E:\os_workspace\spark\common\kvstore\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\core\jackson-core\2.6.7\jackson-core-2.6.7.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\core\jackson-annotations\2.6.7\jackson-annotations-2.6.7.jar;E:\os_workspace\spark\common\network-common\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\com\google\guava\guava\14.0.1\guava-14.0.1.jar;E:\os_workspace\spark\common\network-shuffle\target\scala-2.11\classes;E:\os_workspace\spark\common\unsafe\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\javax\activation\activation\1.1.1\activation-1.1.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\curator\curator-recipes\2.12.0\curator-recipes-2.12.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\curator\curator-framework\2.12.0\curator-framework-2.12.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\zookeeper\zookeeper\3.4.9\zookeeper-3.4.9.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-plus\9.3.24.v20180605\jetty-plus-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-webapp\9.3.24.v20180605\jetty-webapp-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-xml\9.3.24.v20180605\jetty-xml-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-jndi\9.3.24.v20180605\jetty-jndi-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-security\9.3.24.v20180605\jetty-security-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-util\9.3.24.v20180605\jetty-util-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-server\9.3.24.v20180605\jetty-server-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-io\9.3.24.v20180605\jetty-io-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-http\9.3.24.v20180605\jetty-http-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-continuation\9.3.24.v20180605\jetty-continuation-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-servlet\9.3.24.v20180605\jetty-servlet-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-proxy\9.3.24.v20180605\jetty-proxy-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-client\9.3.24.v20180605\jetty-client-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\org\eclipse\jetty\jetty-servlets\9.3.24.v20180605\jetty-servlets-9.3.24.v20180605.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\commons\commons-lang3\3.5\commons-lang3-3.5.jar;D:\eclipse_boot\repository-maven\repository-maven\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;D:\eclipse_boot\repository-maven\repository-maven\org\slf4j\slf4j-api\1.7.16\slf4j-api-1.7.16.jar;D:\eclipse_boot\repository-maven\repository-maven\org\slf4j\jul-to-slf4j\1.7.16\jul-to-slf4j-1.7.16.jar;D:\eclipse_boot\repository-maven\repository-maven\org\slf4j\jcl-over-slf4j\1.7.16\jcl-over-slf4j-1.7.16.jar;D:\eclipse_boot\repository-maven\repository-maven\log4j\log4j\1.2.17\log4j-1.2.17.jar;D:\eclipse_boot\repository-maven\repository-maven\org\slf4j\slf4j-log4j12\1.7.16\slf4j-log4j12-1.7.16.jar;D:\eclipse_boot\repository-maven\repository-maven\com\ning\compress-lzf\1.0.3\compress-lzf-1.0.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\xerial\snappy\snappy-java\1.1.7.1\snappy-java-1.1.7.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\lz4\lz4-java\1.4.0\lz4-java-1.4.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\github\luben\zstd-jni\1.3.2-2\zstd-jni-1.3.2-2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\roaringbitmap\RoaringBitmap\0.5.11\RoaringBitmap-0.5.11.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-net\commons-net\3.1\commons-net-3.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\json4s\json4s-jackson_2.11\3.5.3\json4s-jackson_2.11-3.5.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\json4s\json4s-core_2.11\3.5.3\json4s-core_2.11-3.5.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\json4s\json4s-ast_2.11\3.5.3\json4s-ast_2.11-3.5.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\json4s\json4s-scalap_2.11\3.5.3\json4s-scalap_2.11-3.5.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\core\jersey-client\2.22.2\jersey-client-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\ws\rs\javax.ws.rs-api\2.0.1\javax.ws.rs-api-2.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\hk2\hk2-api\2.4.0-b34\hk2-api-2.4.0-b34.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\hk2\hk2-utils\2.4.0-b34\hk2-utils-2.4.0-b34.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\hk2\external\aopalliance-repackaged\2.4.0-b34\aopalliance-repackaged-2.4.0-b34.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\hk2\external\javax.inject\2.4.0-b34\javax.inject-2.4.0-b34.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\hk2\hk2-locator\2.4.0-b34\hk2-locator-2.4.0-b34.jar;D:\eclipse_boot\repository-maven\repository-maven\org\javassist\javassist\3.18.1-GA\javassist-3.18.1-GA.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\core\jersey-common\2.22.2\jersey-common-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\annotation\javax.annotation-api\1.2\javax.annotation-api-1.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\bundles\repackaged\jersey-guava\2.22.2\jersey-guava-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\hk2\osgi-resource-locator\1.0.1\osgi-resource-locator-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\core\jersey-server\2.22.2\jersey-server-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\media\jersey-media-jaxb\2.22.2\jersey-media-jaxb-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\validation\validation-api\1.1.0.Final\validation-api-1.1.0.Final.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\containers\jersey-container-servlet\2.22.2\jersey-container-servlet-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\glassfish\jersey\containers\jersey-container-servlet-core\2.22.2\jersey-container-servlet-core-2.22.2.jar;D:\eclipse_boot\repository-maven\repository-maven\io\netty\netty-all\4.1.17.Final\netty-all-4.1.17.Final.jar;D:\eclipse_boot\repository-maven\repository-maven\io\netty\netty\3.9.9.Final\netty-3.9.9.Final.jar;D:\eclipse_boot\repository-maven\repository-maven\com\clearspring\analytics\stream\2.7.0\stream-2.7.0.jar;D:\eclipse_boot\repository-maven\repository-maven\io\dropwizard\metrics\metrics-core\3.1.5\metrics-core-3.1.5.jar;D:\eclipse_boot\repository-maven\repository-maven\io\dropwizard\metrics\metrics-jvm\3.1.5\metrics-jvm-3.1.5.jar;D:\eclipse_boot\repository-maven\repository-maven\io\dropwizard\metrics\metrics-json\3.1.5\metrics-json-3.1.5.jar;D:\eclipse_boot\repository-maven\repository-maven\io\dropwizard\metrics\metrics-graphite\3.1.5\metrics-graphite-3.1.5.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\core\jackson-databind\2.6.7.1\jackson-databind-2.6.7.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\module\jackson-module-scala_2.11\2.6.7.1\jackson-module-scala_2.11-2.6.7.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\fasterxml\jackson\module\jackson-module-paranamer\2.7.9\jackson-module-paranamer-2.7.9.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\ivy\ivy\2.4.0\ivy-2.4.0.jar;D:\eclipse_boot\repository-maven\repository-maven\oro\oro\2.0.8\oro-2.0.8.jar;D:\eclipse_boot\repository-maven\repository-maven\net\razorvine\pyrolite\4.13\pyrolite-4.13.jar;D:\eclipse_boot\repository-maven\repository-maven\net\sf\py4j\py4j\0.10.7\py4j-0.10.7.jar;E:\os_workspace\spark\common\tags\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\apache\commons\commons-crypto\1.0.0\commons-crypto-1.0.0.jar;E:\os_workspace\spark\streaming\target\scala-2.11\classes;E:\os_workspace\spark\mllib\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\scala-lang\modules\scala-parser-combinators_2.11\1.1.0\scala-parser-combinators_2.11-1.1.0.jar;E:\os_workspace\spark\sql\core\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\com\univocity\univocity-parsers\2.7.3\univocity-parsers-2.7.3.jar;E:\os_workspace\spark\common\sketch\target\scala-2.11\classes;E:\os_workspace\spark\sql\catalyst\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\codehaus\janino\janino\3.0.9\janino-3.0.9.jar;D:\eclipse_boot\repository-maven\repository-maven\org\codehaus\janino\commons-compiler\3.0.9\commons-compiler-3.0.9.jar;D:\eclipse_boot\repository-maven\repository-maven\org\antlr\antlr4-runtime\4.7\antlr4-runtime-4.7.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\orc\orc-core\1.5.2\orc-core-1.5.2-nohive.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\orc\orc-shims\1.5.2\orc-shims-1.5.2.jar;D:\eclipse_boot\repository-maven\repository-maven\io\airlift\aircompressor\0.10\aircompressor-0.10.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\orc\orc-mapreduce\1.5.2\orc-mapreduce-1.5.2-nohive.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\parquet\parquet-column\1.10.0\parquet-column-1.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\parquet\parquet-common\1.10.0\parquet-common-1.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\parquet\parquet-encoding\1.10.0\parquet-encoding-1.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\parquet\parquet-hadoop\1.10.0\parquet-hadoop-1.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\parquet\parquet-format\2.4.0\parquet-format-2.4.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\parquet\parquet-jackson\1.10.0\parquet-jackson-1.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\arrow\arrow-vector\0.10.0\arrow-vector-0.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\arrow\arrow-format\0.10.0\arrow-format-0.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\arrow\arrow-memory\0.10.0\arrow-memory-0.10.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\carrotsearch\hppc\0.7.2\hppc-0.7.2.jar;D:\eclipse_boot\repository-maven\repository-maven\com\vlkan\flatbuffers\1.2.0-3f79e055\flatbuffers-1.2.0-3f79e055.jar;E:\os_workspace\spark\mllib-local\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\scalanlp\breeze_2.11\0.13.2\breeze_2.11-0.13.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\scalanlp\breeze-macros_2.11\0.13.2\breeze-macros_2.11-0.13.2.jar;D:\eclipse_boot\repository-maven\repository-maven\net\sf\opencsv\opencsv\2.3\opencsv-2.3.jar;D:\eclipse_boot\repository-maven\repository-maven\com\github\rwl\jtransforms\2.4.0\jtransforms-2.4.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\spire-math\spire_2.11\0.13.0\spire_2.11-0.13.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\spire-math\spire-macros_2.11\0.13.0\spire-macros_2.11-0.13.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\typelevel\machinist_2.11\0.6.1\machinist_2.11-0.6.1.jar;D:\eclipse_boot\repository-maven\repository-maven\com\chuusai\shapeless_2.11\2.3.2\shapeless_2.11-2.3.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\typelevel\macro-compat_2.11\1.1.1\macro-compat_2.11-1.1.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\jpmml\pmml-model\1.2.15\pmml-model-1.2.15.jar;D:\eclipse_boot\repository-maven\repository-maven\org\jpmml\pmml-schema\1.2.15\pmml-schema-1.2.15.jar;E:\os_workspace\spark\sql\hive\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\spark-project\hive\hive-exec\1.2.1.spark2\hive-exec-1.2.1.spark2.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-io\commons-io\2.4\commons-io-2.4.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;D:\eclipse_boot\repository-maven\repository-maven\javolution\javolution\5.5.1\javolution-5.5.1.jar;D:\eclipse_boot\repository-maven\repository-maven\log4j\apache-log4j-extras\1.2.17\apache-log4j-extras-1.2.17.jar;D:\eclipse_boot\repository-maven\repository-maven\org\antlr\antlr-runtime\3.4\antlr-runtime-3.4.jar;D:\eclipse_boot\repository-maven\repository-maven\org\antlr\stringtemplate\3.2.1\stringtemplate-3.2.1.jar;D:\eclipse_boot\repository-maven\repository-maven\antlr\antlr\2.7.7\antlr-2.7.7.jar;D:\eclipse_boot\repository-maven\repository-maven\org\antlr\ST4\4.0.4\ST4-4.0.4.jar;D:\eclipse_boot\repository-maven\repository-maven\com\googlecode\javaewah\JavaEWAH\0.3.2\JavaEWAH-0.3.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\iq80\snappy\snappy\0.2\snappy-0.2.jar;D:\eclipse_boot\repository-maven\repository-maven\stax\stax-api\1.0.1\stax-api-1.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\spark-project\hive\hive-metastore\1.2.1.spark2\hive-metastore-1.2.1.spark2.jar;D:\eclipse_boot\repository-maven\repository-maven\com\jolbox\bonecp\0.8.0.RELEASE\bonecp-0.8.0.RELEASE.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\datanucleus\datanucleus-api-jdo\3.2.6\datanucleus-api-jdo-3.2.6.jar;D:\eclipse_boot\repository-maven\repository-maven\org\datanucleus\datanucleus-rdbms\3.2.9\datanucleus-rdbms-3.2.9.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-pool\commons-pool\1.5.4\commons-pool-1.5.4.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-dbcp\commons-dbcp\1.4\commons-dbcp-1.4.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\jdo\jdo-api\3.0.1\jdo-api-3.0.1.jar;D:\eclipse_boot\repository-maven\repository-maven\javax\transaction\jta\1.1\jta-1.1.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\calcite\calcite-avatica\1.2.0-incubating\calcite-avatica-1.2.0-incubating.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\calcite\calcite-core\1.2.0-incubating\calcite-core-1.2.0-incubating.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\calcite\calcite-linq4j\1.2.0-incubating\calcite-linq4j-1.2.0-incubating.jar;D:\eclipse_boot\repository-maven\repository-maven\net\hydromatic\eigenbase-properties\1.1.5\eigenbase-properties-1.1.5.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\httpcomponents\httpclient\4.5.6\httpclient-4.5.6.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\httpcomponents\httpcore\4.4.10\httpcore-4.4.10.jar;D:\eclipse_boot\repository-maven\repository-maven\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;D:\eclipse_boot\repository-maven\repository-maven\commons-codec\commons-codec\1.10\commons-codec-1.10.jar;D:\eclipse_boot\repository-maven\repository-maven\joda-time\joda-time\2.9.3\joda-time-2.9.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\jodd\jodd-core\3.5.2\jodd-core-3.5.2.jar;D:\eclipse_boot\repository-maven\repository-maven\org\datanucleus\datanucleus-core\3.2.10\datanucleus-core-3.2.10.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\thrift\libthrift\0.9.3\libthrift-0.9.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\thrift\libfb303\0.9.3\libfb303-0.9.3.jar;D:\eclipse_boot\repository-maven\repository-maven\org\apache\derby\derby\10.12.1.1\derby-10.12.1.1.jar;E:\os_workspace\spark\graphx\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\com\github\fommil\netlib\core\1.1.2\core-1.1.2.jar;D:\eclipse_boot\repository-maven\repository-maven\net\sourceforge\f2j\arpack_combined_all\0.1\arpack_combined_all-0.1.jar;E:\os_workspace\spark\external\kafka-0-10\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\apache\kafka\kafka-clients\2.0.0\kafka-clients-2.0.0.jar;E:\os_workspace\spark\external\kafka-0-10-sql\target\scala-2.11\classes;D:\eclipse_boot\repository-maven\repository-maven\org\apache\commons\commons-math3\3.4.1\commons-math3-3.4.1.jar;D:\eclipse_boot\repository-maven\repository-maven\org\scala-lang\scala-library\2.11.12\scala-library-2.11.12.jar;D:\eclipse_boot\repository-maven\repository-maven\com\github\scopt\scopt_2.11\3.7.0\scopt_2.11-3.7.0.jar;D:\eclipse_boot\repository-maven\repository-maven\com\twitter\parquet-hadoop-bundle\1.6.0\parquet-hadoop-bundle-1.6.0.jar;D:\eclipse_boot\repository-maven\repository-maven\org\scala-lang\scala-reflect\2.11.12\scala-reflect-2.11.12.jar;D:\eclipse_boot\repository-maven\repository-maven\org\scala-lang\modules\scala-xml_2.11\1.0.5\scala-xml_2.11-1.0.5.jar;D:\scala\scala-2.11.8\lib\scala-actors-2.11.0.jar;D:\scala\scala-2.11.8\lib\scala-actors-migration_2.11-1.1.0.jar;D:\scala\scala-2.11.8\lib\scala-library.jar;D:\scala\scala-2.11.8\lib\scala-parser-combinators_2.11-1.0.4.jar;D:\scala\scala-2.11.8\lib\scala-reflect.jar;D:\scala\scala-2.11.8\lib\scala-swing_2.11-1.0.2.jar;D:\scala\scala-2.11.8\lib\scala-xml_2.11-1.0.4.jar org.apache.spark.examples.JavaWordCount F:\tmp\words.txt
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/08/26 10:01:38 INFO SparkContext: Running Spark version 2.4.0
21/08/26 10:01:39 INFO SparkContext: Submitted application: JavaWordCount
21/08/26 10:01:39 INFO SecurityManager: Changing view acls to: my,root
21/08/26 10:01:39 INFO SecurityManager: Changing modify acls to: my,root
21/08/26 10:01:39 INFO SecurityManager: Changing view acls groups to: 
21/08/26 10:01:39 INFO SecurityManager: Changing modify acls groups to: 
21/08/26 10:01:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(my, root); groups with view permissions: Set(); users  with modify permissions: Set(my, root); groups with modify permissions: Set()
21/08/26 10:01:41 INFO Utils: Successfully started service 'sparkDriver' on port 59582.
21/08/26 10:01:41 INFO SparkEnv: Registering MapOutputTracker
21/08/26 10:01:41 INFO SparkEnv: Registering BlockManagerMaster
21/08/26 10:01:41 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/08/26 10:01:41 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/08/26 10:01:41 INFO DiskBlockManager: Created local directory at C:\Users\my\AppData\Local\Temp\blockmgr-4b76788b-ab68-4bca-a6f6-fe52e1c7d4e6
21/08/26 10:01:41 INFO MemoryStore: MemoryStore started with capacity 869.7 MB
21/08/26 10:01:41 INFO SparkEnv: Registering OutputCommitCoordinator
21/08/26 10:01:41 INFO log: Logging initialized @5555ms
21/08/26 10:01:41 INFO Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
21/08/26 10:01:41 INFO Server: Started @5667ms
21/08/26 10:01:41 INFO AbstractConnector: Started ServerConnector@7d1cfb8b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
21/08/26 10:01:41 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6436a7db{/jobs,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4c51cf28{/jobs/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6995bf68{/jobs/job,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3568f9d2{/jobs/job/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@71c27ee8{/stages,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3e7dd664{/stages/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5b1ebf56{/stages/stage,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@1b835480{/stages/stage/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3549bca9{/stages/pool,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4f25b795{/stages/pool/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6fb365ed{/storage,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6e950bcf{/storage/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@16414e40{/storage/rdd,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@74bada02{/storage/rdd/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@525575{/environment,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@46dffdc3{/environment/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5a709816{/executors,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@78383390{/executors/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6ad3381f{/executors/threadDump,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@53dbe163{/executors/threadDump/json,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@db57326{/static,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@22d7b4f8{/,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@149dd36b{/api,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4982cc36{/jobs/job/kill,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@691939c9{/stages/stage/kill,null,AVAILABLE,@Spark}
21/08/26 10:01:41 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://my-THINK:4040
21/08/26 10:01:41 INFO Executor: Starting executor ID driver on host localhost
21/08/26 10:01:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59595.
21/08/26 10:01:42 INFO NettyBlockTransferService: Server created on my-THINK:59595
21/08/26 10:01:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/08/26 10:01:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, my-THINK, 59595, None)
21/08/26 10:01:42 INFO BlockManagerMasterEndpoint: Registering block manager my-THINK:59595 with 869.7 MB RAM, BlockManagerId(driver, my-THINK, 59595, None)
21/08/26 10:01:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, my-THINK, 59595, None)
21/08/26 10:01:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, my-THINK, 59595, None)
21/08/26 10:01:42 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@1e63ec0b{/metrics/json,null,AVAILABLE,@Spark}
21/08/26 10:01:42 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/E:/os_workspace/spark/spark-warehouse').
21/08/26 10:01:42 INFO SharedState: Warehouse path is 'file:/E:/os_workspace/spark/spark-warehouse'.
21/08/26 10:01:42 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4c37b5b{/SQL,null,AVAILABLE,@Spark}
21/08/26 10:01:42 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@73db4768{/SQL/json,null,AVAILABLE,@Spark}
21/08/26 10:01:42 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3c435123{/SQL/execution,null,AVAILABLE,@Spark}
21/08/26 10:01:42 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@50fe837a{/SQL/execution/json,null,AVAILABLE,@Spark}
21/08/26 10:01:42 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5ce33a58{/static/sql,null,AVAILABLE,@Spark}
21/08/26 10:01:44 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/08/26 10:01:51 INFO FileSourceStrategy: Pruning directories with: 
21/08/26 10:01:51 INFO FileSourceStrategy: Post-Scan Filters: 
21/08/26 10:01:51 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/08/26 10:01:51 INFO FileSourceScanExec: Pushed Filters: 
21/08/26 10:01:53 INFO CodeGenerator: Code generated in 433.874512 ms
21/08/26 10:01:53 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 309.2 KB, free 869.4 MB)
21/08/26 10:01:53 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 26.2 KB, free 869.4 MB)
21/08/26 10:01:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on my-THINK:59595 (size: 26.2 KB, free: 869.7 MB)
21/08/26 10:01:53 INFO SparkContext: Created broadcast 0 from javaRDD at JavaWordCount.java:45
21/08/26 10:01:53 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/08/26 10:01:54 INFO SparkContext: Starting job: collect at JavaWordCount.java:53
21/08/26 10:01:54 INFO DAGScheduler: Registering RDD 5 (mapToPair at JavaWordCount.java:49)
21/08/26 10:01:54 INFO DAGScheduler: Got job 0 (collect at JavaWordCount.java:53) with 1 output partitions
21/08/26 10:01:54 INFO DAGScheduler: Final stage: ResultStage 1 (collect at JavaWordCount.java:53)
21/08/26 10:01:54 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
21/08/26 10:01:54 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
21/08/26 10:01:54 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[5] at mapToPair at JavaWordCount.java:49), which has no missing parents
21/08/26 10:01:54 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.8 KB, free 869.4 MB)
21/08/26 10:01:54 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.6 KB, free 869.4 MB)
21/08/26 10:01:54 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on my-THINK:59595 (size: 6.6 KB, free: 869.7 MB)
21/08/26 10:01:54 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/08/26 10:01:54 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[5] at mapToPair at JavaWordCount.java:49) (first 15 tasks are for partitions Vector(0))
21/08/26 10:01:54 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/08/26 10:01:54 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8291 bytes)
21/08/26 10:01:54 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/08/26 10:01:55 INFO CodeGenerator: Code generated in 49.56175 ms
21/08/26 10:01:55 INFO FileScanRDD: Reading File path: file:///F:/tmp/words.txt, range: 0-35, partition values: [empty row]
21/08/26 10:01:55 INFO CodeGenerator: Code generated in 15.705632 ms
21/08/26 10:01:55 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1749 bytes result sent to driver
21/08/26 10:01:55 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 952 ms on localhost (executor driver) (1/1)
21/08/26 10:01:55 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/08/26 10:01:55 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at JavaWordCount.java:49) finished in 1.200 s
21/08/26 10:01:55 INFO DAGScheduler: looking for newly runnable stages
21/08/26 10:01:55 INFO DAGScheduler: running: Set()
21/08/26 10:01:55 INFO DAGScheduler: waiting: Set(ResultStage 1)
21/08/26 10:01:55 INFO DAGScheduler: failed: Set()
21/08/26 10:01:55 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[6] at reduceByKey at JavaWordCount.java:51), which has no missing parents
21/08/26 10:01:55 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.8 KB, free 869.3 MB)
21/08/26 10:01:55 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.2 KB, free 869.3 MB)
21/08/26 10:01:55 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on my-THINK:59595 (size: 2.2 KB, free: 869.7 MB)
21/08/26 10:01:55 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
21/08/26 10:01:55 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ShuffledRDD[6] at reduceByKey at JavaWordCount.java:51) (first 15 tasks are for partitions Vector(0))
21/08/26 10:01:55 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/08/26 10:01:55 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, ANY, 7662 bytes)
21/08/26 10:01:55 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/08/26 10:01:55 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/08/26 10:01:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 68 ms
21/08/26 10:01:56 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1332 bytes result sent to driver
21/08/26 10:01:56 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 321 ms on localhost (executor driver) (1/1)
21/08/26 10:01:56 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/08/26 10:01:56 INFO DAGScheduler: ResultStage 1 (collect at JavaWordCount.java:53) finished in 0.389 s
21/08/26 10:01:56 INFO DAGScheduler: Job 0 finished: collect at JavaWordCount.java:53, took 1.859355 s
spark: 2
python: 1
java: 3
21/08/26 10:01:56 INFO AbstractConnector: Stopped Spark@7d1cfb8b{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
21/08/26 10:01:56 INFO SparkUI: Stopped Spark web UI at http://my-THINK:4040
21/08/26 10:01:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/08/26 10:01:56 INFO MemoryStore: MemoryStore cleared
21/08/26 10:01:56 INFO BlockManager: BlockManager stopped
21/08/26 10:01:56 INFO BlockManagerMaster: BlockManagerMaster stopped
21/08/26 10:01:56 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/08/26 10:01:56 INFO SparkContext: Successfully stopped SparkContext
21/08/26 10:01:56 INFO ShutdownHookManager: Shutdown hook called
21/08/26 10:01:56 INFO ShutdownHookManager: Deleting directory C:\Users\my\AppData\Local\Temp\spark-7eb34d28-8052-4049-b519-aae09d2950ad

Process finished with exit code 0

4.3 Debug

在SparkSession.getOrCreate()方法中打个断点
在JavaWordCount里右键，点击Debug JavaWordCount.main()
在这里插入图片描述

到这里就可以开始愉快的阅读源码了

遇到的报错

1 未设置Master

org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
	at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:43)

解决：
点击Edit Configurations，VM Options 添加参数 -Dspark.master=local[2]

2 模块编译输出路径冲突

日志

Error:scalac: Output path E:\os_workspace\spark\target\classes is shared between: Module 'spark-assembly_2.11' production, Module 'spark-parent_2.11' production
Please configure separate output paths to proceed with the compilation.
TIP: you can use Project Artifacts to combine compiled classes if needed.

解决：
File
Project Structure
点击Models
点击出现冲突的Model
点击Path
点击Inherit project compile output path

在这里插入图片描述

终回首
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark搭建源码阅读环境
斌躏天下的博客
04-19 611
如题： 怎样编译spark源码方便读呢？首先当然是先去官网或者github上下载spark源码包，这里附上官网地址spark-2.4.5下载 下载好spark源码包以后解压，然后编译一下导入IDEA就可以愉快的看源码了，怎样编译呢？ 这里以windows 10举例。 环境准备 首先要搭建好需要的环境，比如scala，maven和java，我的环境是1.8的java，2.11.11的scala以及3...
分析Spark源码第一步——搭建源码阅读环境
midsummer
05-29 6361
为了深入理解spark，想了解一下源码，故搭建了一下源码阅读环境。我是在windows环境下进行的。 step1:工具的获取 我安装的软件及版本如下，搭建之前请安装好这些： (1)java1.8 (2)scala 安装可参考http://www.runoob.com/scala/scala-install.html (3)maven 参考 https://www.yiibai.com...
spark源码阅读环境搭建
天龙八部
11-30 447
目录 1.环境介绍 2.源码编译 2.1编译成功的流程 3.编译中遇到的问题 3.1 在命令行直接执行编译命令 3.2 未知失败问题（重新执行后成功） 3.3编译成功后本地启动spark-shell 4.导入idear 5.执行测试案例遇到的问题和解决方法 5.1添加依赖 5.2 idear执行测试命令 1.环境介绍 用spark源码2.4.3为例 源码下载...
Spark源码阅读环境搭建
SunnyRivers
08-28 1026
前言 对于Spark的应用，仅仅会使用期API来编程只能达到初级（助理）工程师或中级（熟练）工程师的水平，而学会调优则可以让你进阶为高级工程师。那么怎么才能成为顶尖的工程师呢？源码！ 准备工作 （1）下载IDEA （2）配置Scala2.11 （3）配置Maven （4）下载Spark源码 详细步骤 （1）官网下载 （2）打开IDEA，单击Import Project，并选择...
Spark源码阅读01：环境搭建
weixin_43851569的博客
08-29 220
源码阅读环境搭建 我这周都在抽时间尝试把Spark源码编译成功。刚开始我选择的版本是 2.3+ 发现怎么都编译不成功。遇到的一个问题是 scala-maven-plugin compile failed, 查版本号，修改版本后尝试未果。选早一点的版本的想法是说，阅读源码可能会少一些，毕竟版本越高，代码越庞大和复杂。 但是早期的版本已经是过去式了，依赖的其他包库也会修改和引入新的特性，编译失败卡的时间太久了，不值得投入，遂直接拥抱 Spark 3.0 编译源码 去 github 上克隆一份代码，切换到你想看的
spark 源码阅读环境
weixin_43887873的博客
03-22 198
windows + Intellij IDEA打造Spark源码阅读环境 1.Spark源码阅读环境的准备 Spark源码是有Scala语言写成的，目前，IDEA对Scala的支持要比eclipse要好，大多数人会选在在IDEA上完成Spark平台应用的开发。因此，Spark源码阅读的IDE理所当然的选择了IDEA。 本文介绍的是Windows下的各项配置方法（默认...
Spark源码学习之IDEA源码阅读环境搭建
最新发布
2401_84048554的博客
05-02 325
由于文案过于长，在此就不一一介绍了，这份Java后端架构进阶笔记内容包括：Java集合，JVM、Java并发、微服务、SpringNetty与 RPC 、网络、日志 、Zookeeper 、Kafka 、RabbitMQ 、Hbase 、MongoDB、Cassandra 、Java基础、负载均衡、数据库、一致性算法、Java算法、数据结构、分布式缓存等等知识详解。本知识体系适合于所有Java程序员学习，关于以上目录中的知识点都有详细的讲解及介绍，掌握该知识点的所有内容对你会有一个质的提升，
apache spark源码阅读环境搭建
03-16
要搭建Apache Spark源码阅读环境，需要以下步骤： 1. 安装Java开发环境：首先需要安装Java开发环境，建议使用JDK 8或以上版本。 2. 下载Spark源码：从官网下载Spark源码，解压到本地目录。 3. 安装Scala：Spark...
Spark源码阅读环境搭建详解
11-09
### Spark源码阅读环境搭建详解 #### 一、引言 随着大数据技术的快速发展，Apache Spark作为一款重要的分布式计算框架，被广泛应用于数据处理、机器学习等领域。深入了解Spark的内部工作机制，不仅能够帮助开发者...
spark环境搭建java_Spark环境搭建
weixin_36316642的博客
02-16 850
一、Java安装1、安装目录准备在安装这些包之前在家目录下创建一个app的目录，app目录中创建一个tmp目录：[root@hadoop-master ~]#mkdir -p app/tmp2、解压JDK上传JDK1.8版本jdk-8u74-linux-x64.tar.gz到linux上，并且将其解压到app目录下：[root@hadoop-master ~]#tar -xzvf jdk-8u7...
Spark环境搭建教程
xgb2018的博客
03-27 4253
Spark环境搭建 Spark Local本地模式 Standalone-独立集群 Standalone-HA Spark-On-Yarn Spark 是一种基于内存的快速、通用、可扩展的大数据分析计算引擎
01-spark源码走读本地环境搭建
qq_31454379的博客
03-19 2288
本地环境准备必备环境spark源码导入IDE安装必备ide插件IDE配置默认maven环境项目配置jdk项目增加scala支持项目切换3.2分支编译源码运行测试类 必备环境 IDE ideaIU-2021.3.2 安装jdk并配置环境变量 maven安装，并配置环境变量 spark源码导入IDE https://github.com/apache/spark git clone https://github.com/apache/spark cd spark ## 这里主要以3.2分支为主，比较稳
Spark源码阅读之环境配置（Windows）
LMR的博客
04-05 6708
Spark源码阅读是深入理解大数据框架和性能调优的必要过程，本文介绍在Windows环境下配置Spark源码阅读环境。 相关工具 本文是在windows下的IDEA上搭建源码阅读环境，下列配置仅供参考，可以根据实际情况更改，不需要完全一致。 -Java1.8 -Scala 2.12.8 -Maven 3.5.4 -Git 2.7.2 -Intellij IDEA 2019.1.1 ...
spark2.2.0源码环境搭建
lixiaoksi的博客
09-29 341
概述 本文主要记录在win10环境下编译spark源码过程中的主要步骤，中间涉及一些问题导致运行example例子未成功，因此记录下来加深印象。话不多说，开始记录。 环境准备 jdk version ：1.8 scala version ：2.10.0 maven version ：3.3.9 spark version ： 2.2.0 step1 在一切准备妥当之后，就可以动手了，首先下载spark源码，可以去官网下载源码包，也可以去github上下载源码包，我使用命令方式来下载。 进入你需要存放源
spark源码 环境 配置
天下月色共三分的博客
09-08 431
spark 源码 环境配置
使用Intellij构建spark源码阅读环境
lisongfeng的专栏
06-20 2405
步骤1： 下载spark-源码，这里以spark1.3为例。 http://spark.apache.org/downloads.html 2.下载编译后的spark源码，这里下载根据Hadoop2.3版本编译的。 3.使用Intellij新建scala工程（使用scala编写spark application，也可以新建java工程） 5.配置项目scala
Windows下spark2.3.3源码阅读环境搭建
qq_39124136的博客
09-30 219
1.软件环境 本人系统环境是windows7,IDE 为idea2016 本次编译是通过maven编译，Maven版本为3.3.9，不用SBT编译 自Spark 2.2.0起，对Java 7的支持已被删除！ 给Maven多配置一点内存，防止编译期间OOM异常，官网推荐 export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" Git Jdk...
Intellij Idea环境下Spark源码阅读环境
longyi_java的专栏
10-01 560
转自：http://m.blog.csdn.net/article/details?id=51036300 1. 在windows下搭建 Spark源码阅读环境的准备 Spark源码是有Scala语言写成的，目前，IDEA对Scala的支持要比eclipse要好，大多数人会选在在IDEA上完成Spark平台应用的开发。因此，Spark源
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值