spark-cdh版本编译过程中的各种坑

最新推荐文章于 2023-08-18 10:31:25 发布

Jussi_Lee

最新推荐文章于 2023-08-18 10:31:25 发布

阅读量1.8k

点赞数 2

CC 4.0 BY-SA版权

分类专栏：大数据文章标签： spark

本文链接：https://blog.csdn.net/qq_29269907/article/details/84587280

本文详细记录了在编译Spark-CDH版本过程中遇到的三大问题及解决方案：1) Maven settings.xml配置问题，需去除mirrorOf * 以使pom配置生效；2) 阿里云服务器上仓库连接失败，可能由http与https差异引起，改用https后问题解决；3) pom文件中需要特定的配置项。通过这些解决方法，成功克服了编译过程中的障碍。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark-cdh版本编译过程中的各种坑

bug1

[ERROR] Failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.1.0: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 in nexus-aliyun (http://maven.aliyun.com/nexus/content/groups/public) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.1.0: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 in nexus-aliyun (http://maven.aliyun.com/nexus/content/groups/public)
        at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
        at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
        at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:245)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
        at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java: