Spark：基础知识02

最新推荐文章于 2024-11-07 19:20:36 发布

Xiao Miao

最新推荐文章于 2024-11-07 19:20:36 发布

阅读量404

点赞数 1

分类专栏： Spark 文章标签： spark

本文链接：https://blog.csdn.net/qq_43713049/article/details/117917909

版权

本文介绍了Spark的基础知识，包括在IDEA中开发Spark应用，如WordCount和TopKey的编程实现，Spark YARN的使用，以及不同部署模式的解析。详细讲解了SparkContext的创建，Spark应用的提交过程，以及Client和Cluster模式的区别。

摘要由CSDN通过智能技术生成

文章目录

Spark：基础知识

Spark：基础知识

一、IDEA应用开发

1.依赖

<!-- 指定仓库位置，依次为aliyun、cloudera和jboss仓库 -->
<repositories>
<repository>
<id>aliyun</id>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
</repository>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>jboss</id>
<url>http://repository.jboss.com/nexus/content/groups/public</url>
</repository>
</repositories>
<properties>
<scala.version>2.12.10</scala.version>
<scala.binary.version>2.12</scala.binary.version>
<spark.version>2.4.5</spark.version>
<hadoop.version>2.6.0-cdh5.16.2</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
<build>
<outputDirectory>target/classes</outputDirectory>
<testOutputDirectory>target/test-classes</testOutputDirectory>
<resources>
<resource>
<directory>${project.basedir}/src/main/resources</directory>
</resource>
</resources>
<!-- Maven 编译的插件 -->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

2.应用入口：SparkContext

Spark Application程序入口为：SparkContext，任何一个应用首先需要构建SparkContext对象，如下两步构建：

1.创建SparkConf对象
- 设置Spark Application基本信息，比如应用的名称AppName和应用运行Master
2.传递SparkConf对象，创建SparkContext对象

package com.miao

import org.apache.spark.{SparkConf, SparkContext}


object Spark_ConText {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf()
      .setAppName("appname")
      .setMaster("master")