Spark2学习2之window下编译spark-2.0.0

最新推荐文章于 2021-08-10 22:43:45 发布
最新推荐文章于 2021-08-10 22:43:45 发布
阅读量3.7k 收藏
点赞数
分类专栏： spark 文章标签： spark 2-0-0
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xubo245/article/details/52073819
版权 
更多代码请见：https://github.com/xubo245/SparkLearning  
Spark中组件Ml的学习

1.解释

(1)基本配置：
scala-2.11.8
java1.7
maven3.3.9

(2) 下载：github

(3) 设定jvm参数：

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

(4)编译：mvn clean package -DskipTests

2.代码：

mvn clean package -DskipTests

在window下使用build/mvn -DskipTests clean package，git下使用这个也不能成功，会导致各种问题，bash起了好多线程，最后老报内存不足

3.结果：

编译成功

[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-strea
ming-kafka-0-10-assembly_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10-assembly\target\spa
rk-streaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-
streaming-kafka-0-10-assembly_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10-assembly\target\spa
rk-streaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  4.060 s]
[INFO] Spark Project Tags ................................. SUCCESS [  6.272 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 12.164 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.965 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  8.533 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 15.566 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 11.382 s]
[INFO] Spark Project Core ................................. SUCCESS [04:00 min]
[INFO] Spark Project GraphX ............................... SUCCESS [ 46.278 s]
[INFO] Spark Project Streaming ............................ SUCCESS [01:30 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:04 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:45 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 32.449 s]
[INFO] Spark Project ML Library ........................... SUCCESS [02:56 min]
[INFO] Spark Project Tools ................................ SUCCESS [  6.504 s]
[INFO] Spark Project Hive ................................. SUCCESS [03:00 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 19.472 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  4.420 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 17.863 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 26.544 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [  3.423 s]
[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [ 32.688 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 48.515 s]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  5.093 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 30.390 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  4.189 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:07 min
[INFO] Finished at: 2016-07-30T16:49:32+08:00
[INFO] Final Memory: 95M/994M
[INFO] ------------------------------------------------------------------------

为什么没有看到完整的jar包？？

参考
【1】http://spark.apache.org/docs/2.0.0/
【2】https://github.com/apache/spark
【3】https://github.com/xubo245/SparkLearning

部分记录

ar.
[INFO] Excluding org.tukaani:xz:jar:1.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.2.0 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0 from th
e shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0 from
 the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.2.0 from the shaded
jar.
[INFO] Excluding com.google.inject:guice:jar:3.0 from the shaded jar.
[INFO] Excluding javax.inject:javax.inject:jar:1 from the shaded jar.
[INFO] Excluding aopalliance:aopalliance:jar:1.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0 from the
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0 fro
m the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.2.0 from the shaded jar
.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0 from t
he shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.2.0 from the shaded
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0 f
rom the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-annotations:jar:2.2.0 from the shaded
jar.
[INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.7.1 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.10 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded j
ar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.4.0 from the shaded ja
r.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.4.0 from the shaded
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.4.0 from the shaded jar
.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.5 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.16 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.16 from the shaded jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.1.2.4 from the shaded jar.
[INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the shaded jar.
[INFO] Excluding commons-net:commons-net:jar:2.2 from the shaded jar.
[INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.6 from the shaded ja
r.
[INFO] Excluding org.apache.kafka:kafka_2.11:jar:0.10.0.0 from the shaded jar.
[INFO] Excluding com.101tec:zkclient:jar:0.8 from the shaded jar.
[INFO] Excluding com.yammer.metrics:metrics-core:jar:2.2.0 from the shaded jar.
[INFO] Excluding org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4
from the shaded jar.
[INFO] Excluding org.apache.kafka:kafka-clients:jar:0.10.0.0 from the shaded jar
.
[INFO] Excluding org.apache.spark:spark-tags_2.11:jar:2.1.0-SNAPSHOT from the sh
aded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing C:\soft\spark\spark\external\kafka-0-10\target\spark-streaming-
kafka-0-10_2.11-2.1.0-SNAPSHOT.jar with C:\soft\spark\spark\external\kafka-0-10\
target\spark-streaming-kafka-0-10_2.11-2.1.0-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at: C:\soft\spark\spark\external\kafka-0-1
0\dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: C:\soft\spark\spark\external\kafka-0-1
0\dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: C:\soft\spark\spark\external\kafka-0-1
0\dependency-reduced-pom.xml
[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-strea
ming-kafka-0-10_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10\target\spark-stream
ing-kafka-0-10_2.11-2.1.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-
streaming-kafka-0-10_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10\target\spark-stream
ing-kafka-0-10_2.11-2.1.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Integration for Kafka 0.10 Assembly 2.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ spark-streaming-kafk
a-0-10-assembly_2.11 ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ spark-stream
ing-kafka-0-10-assembly_2.11 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-stre
aming-kafka-0-10-assembly_2.11 ---
[INFO] Add Source directory: C:\soft\spark\spark\external\kafka-0-10-assembly\sr
c\main\scala
[INFO] Add Test Source directory: C:\soft\spark\spark\external\kafka-0-10-assemb
ly\src\test\scala
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) @ spark-st
reaming-kafka-0-10-assembly_2.11 ---
[INFO] Dependencies classpath:
C:\Users\xingyun.xb\.m2\repository\org\apache\avro\avro\1.7.7\avro-1.7.7.jar;C:\
Users\xingyun.xb\.m2\repository\com\101tec\zkclient\0.8\zkclient-0.8.jar;C:\User
s\xingyun.xb\.m2\repository\javax\inject\javax.inject\1\javax.inject-1.jar;C:\so
ft\spark\spark\external\kafka-0-10\target\spark-streaming-kafka-0-10_2.11-2.1.0-
SNAPSHOT.jar;C:\Users\xingyun.xb\.m2\repository\org\codehaus\jackson\jackson-cor
e-asl\1.9.13\jackson-core-asl-1.9.13.jar;C:\Users\xingyun.xb\.m2\repository\com\
thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\xingyun.xb\.m2\r
epository\com\google\inject\guice\3.0\guice-3.0.jar;C:\Users\xingyun.xb\.m2\repo
sitory\commons-io\commons-io\2.4\commons-io-2.4.jar;C:\Users\xingyun.xb\.m2\repo
sitory\org\scala-lang\modules\scala-parser-combinators_2.11\1.0.4\scala-parser-c
ombinators_2.11-1.0.4.jar;C:\Users\xingyun.xb\.m2\repository\org\apache\kafka\ka
fka-clients\0.10.0.0\kafka-clients-0.10.0.0.jar;C:\Users\xingyun.xb\.m2\reposito
ry\org\apache\hadoop\hadoop-yarn-api\2.2.0\hadoop-yarn-api-2.2.0.jar;C:\Users\xi
ngyun.xb\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar;C:\Users\xingyun.xb\.m2\re
pository\org\apache\avro\avro-ipc\1.7.7\avro-ipc-1.7.7.jar;C:\Users\xingyun.xb\.
m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.j
ar;C:\Users\xingyun.xb\.m2\repository\org\apache\hadoop\hadoop-yarn-client\2.2.0
\hadoop-yarn-client-2.2.0.jar;C:\Users\xingyun.xb\.m2\repository\org\apache\hado
op\hadoop-yarn-common\2.2.0\hadoop-yarn-common-2.2.0.jar;C:\Users\xingyun.xb\.m2
\repository\org\apache\kafka\kafka_2.11\0.10.0.0\kafka_2.11-0.10.0.0.jar;C:\User
s\xingyun.xb\.m2\repository\aopalliance\aopalliance\1.0\aopalliance-1.0.jar;C:\U
sers\xingyun.xb\.m2\repository\com\yammer\metrics\metrics-core\2.2.0\metrics-cor
e-2.2.0.jar;C:\Users\xingyun.xb\.m2\repository\org\codehaus\jackson\jackson-mapp
er-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;C:\soft\spark\spark\common\tags\targ
et\spark-tags_2.11-2.1.0-SNAPSHOT.jar;C:\Users\xingyun.xb\.m2\repository\org\spa
rk-project\spark\unused\1.0.0\unused-1.0.0.jar
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-streaming
-kafka-0-10-assembly_2.11 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-stre
aming-kafka-0-10-assembly_2.11 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\soft\spark\spark\external\kafka-0-
10-assembly\src\main\resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-stream
ing-kafka-0-10-assembly_2.11 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ spark-streami
ng-kafka-0-10-assembly_2.11 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) @ spark-streaming-kafka-
0-10-assembly_2.11 ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: C:\soft\spark\spark\external\kafka-0-10-assembly\target
\tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sp
ark-streaming-kafka-0-10-assembly_2.11 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\soft\spark\spark\external\kafka-0-
10-assembly\src\test\resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spa
rk-streaming-kafka-0-10-assembly_2.11 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ spark
-streaming-kafka-0-10-assembly_2.11 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (generate-test-classpath
) @ spark-streaming-kafka-0-10-assembly_2.11 ---
[INFO]
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ spark-streaming-ka
fka-0-10-assembly_2.11 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-surefire-plugin:2.19.1:test (test) @ spark-streaming-kafka-0-10
-assembly_2.11 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-streaming-kafka-0-10-a
ssembly_2.11 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.6:test-jar (prepare-test-jar) @ spark-streaming-ka
fka-0-10-assembly_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10-assembly\target\spa
rk-streaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ spark-streaming-kafka-0-10-a
ssembly_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10-assembly\target\spa
rk-streaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-s
treaming-kafka-0-10-assembly_2.11 ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.3:shade (default) @ spark-streaming-kafka-0-10
-assembly_2.11 ---
[INFO] Including org.apache.spark:spark-streaming-kafka-0-10_2.11:jar:2.1.0-SNAP
SHOT in the shaded jar.
[INFO] Including org.apache.kafka:kafka_2.11:jar:0.10.0.0 in the shaded jar.
[INFO] Including com.101tec:zkclient:jar:0.8 in the shaded jar.
[INFO] Including com.yammer.metrics:metrics-core:jar:2.2.0 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4
in the shaded jar.
[INFO] Including org.apache.kafka:kafka-clients:jar:0.10.0.0 in the shaded jar.
[INFO] Including org.apache.spark:spark-tags_2.11:jar:2.1.0-SNAPSHOT in the shad
ed jar.
[INFO] Including commons-io:commons-io:jar:2.4 in the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.7.7 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.3 in the shaded jar.

[INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded jar
.
[INFO] Including org.tukaani:xz:jar:1.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.2.0 in the shaded ja
r.
[INFO] Including com.google.inject:guice:jar:3.0 in the shaded jar.
[INFO] Including javax.inject:javax.inject:jar:1 in the shaded jar.
[INFO] Including aopalliance:aopalliance:jar:1.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.2.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.2.0 in the shaded ja
r.
[INFO] Including org.apache.avro:avro-ipc:jar:1.7.7 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shade
d jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[WARNING] spark-tags_2.11-2.1.0-SNAPSHOT.jar, spark-streaming-kafka-0-10_2.11-2.
1.0-SNAPSHOT.jar, unused-1.0.0.jar define 1 overlapping classes:
[WARNING]   - org.apache.spark.unused.UnusedStubClass
[WARNING] hadoop-yarn-common-2.2.0.jar, hadoop-yarn-api-2.2.0.jar define 3 overl
apping classes:
[WARNING]   - org.apache.hadoop.yarn.factories.package-info
[WARNING]   - org.apache.hadoop.yarn.util.package-info
[WARNING]   - org.apache.hadoop.yarn.factory.providers.package-info
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing C:\soft\spark\spark\external\kafka-0-10-assembly\target\spark-s
treaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT.jar with C:\soft\spark\spark\ex
ternal\kafka-0-10-assembly\target\spark-streaming-kafka-0-10-assembly_2.11-2.1.0
-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at: C:\soft\spark\spark\external\kafka-0-1
0-assembly\dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: C:\soft\spark\spark\external\kafka-0-1
0-assembly\dependency-reduced-pom.xml
[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-strea
ming-kafka-0-10-assembly_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10-assembly\target\spa
rk-streaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-
streaming-kafka-0-10-assembly_2.11 ---
[INFO] Building jar: C:\soft\spark\spark\external\kafka-0-10-assembly\target\spa
rk-streaming-kafka-0-10-assembly_2.11-2.1.0-SNAPSHOT-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  4.060 s]
[INFO] Spark Project Tags ................................. SUCCESS [  6.272 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 12.164 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.965 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  8.533 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 15.566 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 11.382 s]
[INFO] Spark Project Core ................................. SUCCESS [04:00 min]
[INFO] Spark Project GraphX ............................... SUCCESS [ 46.278 s]
[INFO] Spark Project Streaming ............................ SUCCESS [01:30 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:04 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:45 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 32.449 s]
[INFO] Spark Project ML Library ........................... SUCCESS [02:56 min]
[INFO] Spark Project Tools ................................ SUCCESS [  6.504 s]
[INFO] Spark Project Hive ................................. SUCCESS [03:00 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 19.472 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  4.420 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 17.863 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 26.544 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [  3.423 s]
[INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [ 32.688 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 48.515 s]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [  5.093 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 30.390 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  4.189 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:07 min
[INFO] Finished at: 2016-07-30T16:49:32+08:00
[INFO] Final Memory: 95M/994M
[INFO] ------------------------------------------------------------------------

C:\soft\spark\spark>
KeepLearningBigData
关注
专栏目录
【大数据之路5-1】数据仓库工具 Hive
程序员五哥
06-09 1129
大数据：历史数据，预测未来Hive 由 FaceBook 实现并开源，是基于 Hadoop 的一个数据仓库工具，可以将结构化的数据映射为一张数据库表，并提供 HQL（Hive SQL）查询 功能，底层数据存储在 HDFS 上。Hive 的本质是将 HQL 语句转换为 MapReduce 任务运行，使不熟悉 MapReduce 的用户很方便的使用 HQL 处理和计算 HDFS 上结构化的数据，适用于离线的批量数据计算。分条：基于 Hadoop：Hive 依赖于 HDFS 存储数据，Hive 将 HQL 转换成
Hadoop生态圈 大数据文档
BigData_XiaoBai的博客
12-01 800
文档基于介绍基于Hadoop的大数据生态圈。介绍下图每一个组件的使用场景及使用方法，同时还对每一个组件有更深入的介绍。 ..
spark2.0编译版-适用于hive2.3的hive on spark
09-26
hive on spark要求spark中不包含hive的jar包，必须重新编译，历尽艰辛终于搞好了，适配hive2.3、hadoop2.7.6环境。
Spark 2.0.0正式版编译及问题分析
大数据挖掘SparkExpert的博客
08-31 1453
从上次编译了技术预览版2.0.1之后，官网终于放出了正式版本的2.0.0版本。 Spark Release 2.0.0（见http://spark.apache.org/releases/spark-release-2-0-0.html） Apache Spark 2.0.0 is the first release on the 2.x line. The major
Spark-2.0.2源码编译
learn_8的专栏
04-10 128
注：图片如果损坏，点击文章链接：https://www.toutiao.com/i6813925210731840013/ Spark官网下载地址： http://spark.apache.org/downloads.html 有两种一种是编译好的 一种是没编译的（需要自己编译） 看自己的选择 下载好的文件 解压之后可以看源码 直接用IDE工具打开就可以 ...
编译安装spark2.2.0
leofionn的博客
03-06 492
环境：centos6.8软件准备：spark-2.2.0.tgz源码                   jdk-8u144-linux-x64.tar.gz   JDK1.8以上                     apache-maven-3.3.9-bin.tar.gz                   scala-2.11.8.tgz                   hadoop...
windows下编译spark的源码
funny75的专栏
12-24 3007
1.maven的安装 到maven官网下载maven,maven依赖于jdk，所以先确保以安装jdk( http://maven.apache.org/download.cgi ) 解压到一个目录下，我选择的是D:\learning_soft\maven 然后配置环境变量 %M2_HOME%=D:\learning_soft\maven\apache-maven-3.3.9 添加到path
Linux搭建Hive On Spark环境(spark-1.6.3-without-hive+hadoop2.8.0+hive2.1.1)
pucao_cug的专栏
05-26 1万+
不需要自己手动编译Spark源码，在Linux(CentOS7或者Ubuntu)上搭建Hive On Spark环境 版本号是spark-1.6.3-without-hive hadoop2.8.0 hive2.1.1
EBT 道客巴巴的加密与破解 - 实用组合工具箱
08-04 7963
本文系列共有三篇，前两篇链接在此：  《 EBT 道客巴巴的加密与破解 －免费下载器的基础》  《EBT 道客巴巴的加密与破解 序章》 写到这里有点忧伤，有两点，一是CSDN在删除我之前上传的附件，二是doc88.com服务器端没有完成破解，也就是说我只解开了doc88客户端。不想说什么，纯贴个代码就走。 AIR 项目主文件： <s:Windowe
手动搭建Hadoop环境
casepk的专栏
02-02 980
  环境 namenode有2个，datanode有3个。   基础配置 环境变量 .bashrc(怕配覆盖就放.bash_profile中) ============= export PS1="[\u@\h|\w]\\$ "   #alias alias l='ls --group-directories-first --time-style=long-iso -l' a...
Spark2.2.0源码编译打包
就问你吃不吃药
03-22 937
下载Spark2.2.0源码 在Spark官网下载源码，选着Source Code下载即可（也可以git clone spark源码），下载之后的tar包传入机器对应位置解压即可。 安装Maven并配置 在环境变量里设置maven内存大小 开始编译 由于Hadoop版本使用的是2.7.3，并且需要hive支持，所以编译时需要指定版本 不需要打安装包编译命令 ./build...
01-安装Spark-2.0.2--配置支持pytho3、ipython3
GYT0313
12-26 720
前言 本篇博文为博主开始学习Spark技术的第一篇博客。 1. 下载安装Spark-2.0.2 因为参考书籍：Spark快速大数据分析使用的是Spark-1.0不支持python3，且官网下载时没有以前版本的链接，并且博主所安装的Hadoop集群为2.7.3，因此下载Spark-2.0版本。 下载 Spark官网链接：http://spark.apache.org/downloads.html...
Spark实战（1）_Spark2.0环境搭建
分享数据科学家的自我修养
01-07 669
环境安装软件准备 CentOS-7-x86_64-Everything-1611.iso spark-2.0.1-bin-hadoop2.7.tgz hadoop-2.7.3.tar.gz scala-2.11.8.tgz jdk-8u91-linux-x64.tar.gz 建立Linux虚拟机(全节点) 客户机操作系统：CentOS-7-x86_64。 网络和主机名设置：
Java实现不同方式的压缩
calm_encode的博客
08-10 495
一 概述 实现压缩与解压导的包 解压： <!-- https://mvnrepository.com/artifact/org.tukaani/xz --> <dependency> <groupId>org.tukaani</groupId> <artifactId>xz</artifactId> <version>1.5</version> &l...
Spark伪分布安装（依赖Hadoop）
marho
08-24 6771
一、伪分布安装Spark 安装环境： Ubuntu 14.04 LTS 64位+Hadoop2.7.2+Spark2.0.0+jdk1.7.0_76 1、安装jdk1.7 （1）下载jdk-7u76-linux-x64.tar.gz； （2）解压jdk-7u76-linux-x64.tar.gz，并将其移动到/opt/java/jdk路径下（自建）； 命令：tar -zxvf jdk-
windows下spark2.1源码编译及修改
Dreamershi的专栏
05-05 4188
Windows编译spark源码过程 对spark源码修改后需要重新编译spark源码，由于当前linux虚拟机上无法通过代理联网，公司提供的maven仓库也ping不通，只能在windows上编译spark源码。 编译过程如下： 1. 在spark官网下载spark源码http://spark.apache.org/downloads.html   选择2.1.0源码下载。 2.
Spark Streaming集成Kafka的两种方式
不清不慎的博客
05-13 3174
Spark Streaming集成Kafka 转载请标明原文链接： 在实际开发中Spark Streaming经常会结合Flume以及Kafka来实时计算数据。这篇文章主要讲述如何集成Kafka，对于Kafka的集成有两种方式： Receiver-based Approach Direct Approach (No Receivers) 在集成Kafka之前，必须先安装kafka并且启...
Hive On Spark2.0.0 搭建
Dzhantao的博客
07-17 2879
      Hive作为Hadoop家族成员中一份子，在现在的离线数据分析中的重要性已经不用多说。但是众所周知，hive是以MR程序跑在YARN集群上的，这其中的酸爽，想必用过的都说好了。随着技术的进步，Hive的开发者也已经意识到了MR的效率已经不能满足很多的日常需求，所以从hive二代开始，尝试着用spark引擎来替换掉MR，到现在为止，已经hive已经能支持几个版本的spark了。好了，下面...
Spark2.0.0集群环境部署（Spark On Yarn）
热门推荐
DataFlow范式
08-08 1万+
Spark 2.0.0发布已经有一段时间了，目前公司生产环境还是使用1.6系列版本。为了测试Spark 2.0.0各方面的稳定性和计算性能，我基于CDH集群环境，搭建了Spark On Yarn集群环境。         环境信息：         CDH版本：CDH-5.7.0                   其中，Hadoop版本：2.6.0         Java版本：1.7.0_80
d2l-zh-pytorch-2.0.0.pdf
最新发布
01-07
d2l-zh-pytorch-2.0.0.pdf是一份关于使用PyTorch 2.0.0进行深度学习的中文指南。PyTorch是一个开源的深度学习框架，具有灵活性和易用性，能够帮助开发者快速构建和训练深度学习模型。 这份指南从基础概念开始介绍...
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值