spark java api 入门

最新推荐文章于 2021-06-28 16:21:23 发布

置顶一个懒人的故事

最新推荐文章于 2021-06-28 16:21:23 发布

阅读量442

点赞数 2

分类专栏：大数据文章标签： java spark maven 大数据 linux

本文链接：https://blog.csdn.net/carl918/article/details/105664153

版权

大数据专栏收录该内容

5 篇文章 0 订阅

订阅专栏

前提

具备java,spark基本知识

安装,配置要自己能搞定

第一步，加入依赖

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com</groupId>
	<artifactId>spark</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-core_2.11</artifactId>
			<version>2.4.5</version>
           <!-- 开发编译期间提供,不会打进jar包,节省空间和带宽,运行时环境提供-->
                   <scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>fastjson</artifactId>
			<version>1.2.59</version>
		</dependency>
	</dependencies>

	<!--打包,把依赖包打在一起,俗称胖jar包 -->
	<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <configuration>
                <!-- put your configurations here -->
                <mainClass>com.spark.JavaWordCount</mainClass>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
</project>

第二步，统计单词功能实现

可以本地直接调试,注意,conf.setMaster("local[2]"); //本地2个工作线程

package com.spark;

import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;

import com.alibaba.fastjson.JSON;

import scala.Tuple2;

public class JavaWordCount {

// 统计输入文件单词,并按大小排序
	public static void main(String[] args) {
		SparkConf conf = new SparkConf();
		conf.setAppName("JavaWordCount");
           //idea中调试 local 运行在本地计算机
		conf.setMaster("local[2]");		
		JavaSparkContext jsc = new JavaSparkContext(conf);
          // 输入文件的路径
		JavaRDD<String> lines = jsc.textFile(args[0]);  
		JavaRDD<String> words = lines.flatMap(line -> Arrays.asList(line.split(" ")).iterator());
		JavaPairRDD<String, Integer> tup = words.mapToPair(word -> new Tuple2<>(word, 1));
		JavaPairRDD<String, Integer> reduced = tup.reduceByKey((x, y) -> x + y);
		JavaPairRDD<Integer, String> swap = reduced.mapToPair(tup2 -> tup2.swap());
		JavaPairRDD<Integer, String> sort = swap.sortByKey();
		JavaPairRDD<String, Integer> res = sort.mapToPair(tup3 -> tup3.swap());
		String json = JSON.toJSONString(res.collect());
		System.out.println(json);
           // 输出数据的目录,必须是不存在的,程序自动创建
		res.saveAsTextFile(args[1]);
		jsc.close();
	}
}

第三步打包,集群运行

首先注释掉conf.setMaster("local[2]"); 这一行,准备在集群上运行

以下是运行命令--deploy-mode cluster 不加,就是client模式,

[root@ spark-2.4.5-bin-hadoop2.7]# ./bin/spark-submit \
  --class com.spark.JavaWordCount \
  --master spark://192.168.8.3:7077 \  
  --deploy-mode cluster \  
  --verbose \
  /home/app/spark-1.0-SNAPSHOT.jar \
  /home/aaa.log \
  /home/bbb