Spark用Java编写WordCount

最新推荐文章于 2023-09-01 19:56:47 发布
最新推荐文章于 2023-09-01 19:56:47 发布
阅读量450 收藏 1
点赞数
分类专栏： # 2.Spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Jackson_mvp/article/details/104055579
版权

 一.Spark用Java编写WordCount



import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;
import scala.Tuple2;

import java.util.Arrays;
import java.util.Iterator;

import static java.lang.System.*;

/**
 * Author : Jackson
 * Version : 2020/1/20 & 1.0
 */

public class WordCountJava {
    public static void main(String[] args) {
        //获取spark的环境
        SparkConf conf = new SparkConf().setAppName("Java").setMaster("local");
        JavaSparkContext jsc = new JavaSparkContext(conf);

        //读取数据
        JavaRDD<String> javardd = jsc.textFile("D:\\input\\test\\李庆\\wordcount.txt");

        //数据的切分
        JavaRDD word = javardd.flatMap(new FlatMapFunction<String, String>() {
            @Override
            public Iterator<String> call(String s) throws Exception {
                String[] s1 = s.split(" ");
                return Arrays.asList(s1).iterator();
            }
        });

        //数据拼接
        JavaPairRDD<Object, Integer> javaPairRDD = word.mapToPair(new PairFunction<String, String, Integer>() {
            @Override
            public Tuple2 call(String s) throws Exception {
                return new Tuple2<>(s, 1);
            }
        });


        //reduce合并
        JavaPairRDD<Object, Integer> reduce = javaPairRDD.reduceByKey(new Function2<Integer, Integer, Integer>() {
            @Override
            public Integer call(Integer s, Integer s2) throws Exception {
                return s + s2;
            }
        });
        //打印
        reduce.foreach(new VoidFunction<Tuple2<Object, Integer>>() {
            @Override
            public void call(Tuple2<Object, Integer> objectIntegerTuple2) throws Exception {
                out.println(objectIntegerTuple2);
            }
        });


    }
}

 

结果：


Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/01/20 19:19:57 INFO SparkContext: Running Spark version 2.1.0
20/01/20 19:19:58 INFO SecurityManager: Changing view acls to: Jackson,ÀîÇì
20/01/20 19:19:58 INFO SecurityManager: Changing modify acls to: Jackson,ÀîÇì
20/01/20 19:19:58 INFO SecurityManager: Changing view acls groups to: 
20/01/20 19:19:58 INFO SecurityManager: Changing modify acls groups to: 
20/01/20 19:19:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(Jackson, ÀîÇì); groups with view permissions: Set(); users  with modify permissions: Set(Jackson, ÀîÇì); groups with modify permissions: Set()
20/01/20 19:20:01 INFO Utils: Successfully started service 'sparkDriver' on port 56834.
20/01/20 19:20:01 INFO SparkEnv: Registering MapOutputTracker
20/01/20 19:20:01 INFO SparkEnv: Registering BlockManagerMaster
20/01/20 19:20:01 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/01/20 19:20:01 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/01/20 19:20:01 INFO DiskBlockManager: Created local directory at C:\Users\李庆\AppData\Local\Temp\blockmgr-ec822fe3-38c5-4047-9017-b8ad42757d63
20/01/20 19:20:01 INFO MemoryStore: MemoryStore started with capacity 1989.6 MB
20/01/20 19:20:01 INFO SparkEnv: Registering OutputCommitCoordinator
20/01/20 19:20:01 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/01/20 19:20:01 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.2.1:4040
20/01/20 19:20:01 INFO Executor: Starting executor ID driver on host localhost
20/01/20 19:20:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56848.
20/01/20 19:20:02 INFO NettyBlockTransferService: Server created on 192.168.2.1:56848
20/01/20 19:20:02 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/01/20 19:20:02 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.2.1, 56848, None)
20/01/20 19:20:02 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.2.1:56848 with 1989.6 MB RAM, BlockManagerId(driver, 192.168.2.1, 56848, None)
20/01/20 19:20:02 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.2.1, 56848, None)
20/01/20 19:20:02 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.2.1, 56848, None)
20/01/20 19:20:02 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 127.1 KB, free 1989.5 MB)
20/01/20 19:20:02 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 14.3 KB, free 1989.5 MB)
20/01/20 19:20:02 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.2.1:56848 (size: 14.3 KB, free: 1989.6 MB)
20/01/20 19:20:02 INFO SparkContext: Created broadcast 0 from textFile at WordCountJava.java:31
20/01/20 19:20:03 INFO FileInputFormat: Total input paths to process : 1
20/01/20 19:20:03 INFO SparkContext: Starting job: foreach at WordCountJava.java:59
20/01/20 19:20:03 INFO DAGScheduler: Registering RDD 3 (mapToPair at WordCountJava.java:43)
20/01/20 19:20:03 INFO DAGScheduler: Got job 0 (foreach at WordCountJava.java:59) with 1 output partitions
20/01/20 19:20:03 INFO DAGScheduler: Final stage: ResultStage 1 (foreach at WordCountJava.java:59)
20/01/20 19:20:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
20/01/20 19:20:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
20/01/20 19:20:03 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at WordCountJava.java:43), which has no missing parents
20/01/20 19:20:03 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.0 KB, free 1989.5 MB)
20/01/20 19:20:03 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.8 KB, free 1989.5 MB)
20/01/20 19:20:03 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.2.1:56848 (size: 2.8 KB, free: 1989.6 MB)
20/01/20 19:20:03 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:996
20/01/20 19:20:03 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at mapToPair at WordCountJava.java:43)
20/01/20 19:20:03 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
20/01/20 19:20:03 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 5977 bytes)
20/01/20 19:20:03 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
20/01/20 19:20:03 INFO HadoopRDD: Input split: file:/D:/input/test/李庆/wordcount.txt:0+96
20/01/20 19:20:03 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
20/01/20 19:20:03 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
20/01/20 19:20:03 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
20/01/20 19:20:03 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
20/01/20 19:20:03 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
20/01/20 19:20:03 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1816 bytes result sent to driver
20/01/20 19:20:03 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 210 ms on localhost (executor driver) (1/1)
20/01/20 19:20:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
20/01/20 19:20:03 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at WordCountJava.java:43) finished in 0.227 s
20/01/20 19:20:03 INFO DAGScheduler: looking for newly runnable stages
20/01/20 19:20:03 INFO DAGScheduler: running: Set()
20/01/20 19:20:03 INFO DAGScheduler: waiting: Set(ResultStage 1)
20/01/20 19:20:03 INFO DAGScheduler: failed: Set()
20/01/20 19:20:03 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCountJava.java:52), which has no missing parents
20/01/20 19:20:03 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 1989.5 MB)
20/01/20 19:20:03 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1870.0 B, free 1989.4 MB)
20/01/20 19:20:03 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.2.1:56848 (size: 1870.0 B, free: 1989.6 MB)
20/01/20 19:20:03 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:996
20/01/20 19:20:03 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ShuffledRDD[4] at reduceByKey at WordCountJava.java:52)
20/01/20 19:20:03 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
20/01/20 19:20:03 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, ANY, 5750 bytes)
20/01/20 19:20:03 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
20/01/20 19:20:03 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
20/01/20 19:20:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms
(aa,2)
(gg,2)
(spark,2)
(hadoop,2)
(dd,2)
(hadopp,2)
(linux,2)
(ff,2)
(bb,2)
(cc,2)
20/01/20 19:20:03 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1632 bytes result sent to driver
20/01/20 19:20:03 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 38 ms on localhost (executor driver) (1/1)
20/01/20 19:20:03 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
20/01/20 19:20:03 INFO DAGScheduler: ResultStage 1 (foreach at WordCountJava.java:59) finished in 0.038 s
20/01/20 19:20:03 INFO DAGScheduler: Job 0 finished: foreach at WordCountJava.java:59, took 0.403764 s
20/01/20 19:20:03 INFO SparkContext: Invoking stop() from shutdown hook
20/01/20 19:20:03 INFO SparkUI: Stopped Spark web UI at http://192.168.2.1:4040
20/01/20 19:20:03 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/01/20 19:20:03 INFO MemoryStore: MemoryStore cleared
20/01/20 19:20:03 INFO BlockManager: BlockManager stopped
20/01/20 19:20:03 INFO BlockManagerMaster: BlockManagerMaster stopped
20/01/20 19:20:03 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/01/20 19:20:03 INFO SparkContext: Successfully stopped SparkContext
20/01/20 19:20:03 INFO ShutdownHookManager: Shutdown hook called
20/01/20 19:20:03 INFO ShutdownHookManager: Deleting directory C:\Users\李庆\AppData\Local\Temp\spark-d674268f-2495-4339-a2ee-543d2285308f

Process finished with exit code 0

 

                                                                                                                                           ————保持饥饿，保持学习

                                                                                                                                                                  Jackson_MVP     

 

 

 

SuperBigData~
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
用Spark写一个简单的wordcount词频统计程序
weixin_33805743的博客
06-08 872
public class WordCountLocal {public static void main(String[] args) {SparkConf conf = new SparkConf().setAppName("WordCountLocal").setMaster("local[2]");JavaSparkContext sc = new Jav...
Spark 的JavaWordCount分步详解
晓晓的天空
03-16 1111
Spark 的JavaWordCount分步详解
Spark入门-Java实现WordCount并打包测试
Horsleyli的博客
07-13 755
一、Java代码实现WordCount import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.FlatMapFunction; imp
Spark入门第一步：WordCount之java版、Scala版
趣学程序
10-27 394
Spark入门第一步：WordCount之java版、Scala版 Spark入门系列，第一步，编写WordCount程序。 我们分别使用java和scala进行编写，从而比较二者的代码量 数据文件 通过读取下面的文件内容，统计每个单词出现的次数 java scala python android spark storm spout bolt kafka MQ elasticsearch lo...
Spark用Java实现的WordCount
大数据
12-04 1056
java版本的sparkWordCount /** * Java版本的spark WordCount */ public class JavaWordCount { public static void main(String[] args) { //1.先创建conf对象进行配置，主要是设置名称，为了设置运行模式 SparkConf conf = ne...
Spark：用Scala和Java实现WordCount
03-03
用Scala和Java实现WordCount，其中Java实现的JavaWordCount是spark自带的例子（$SPARK_HOME/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java）OS:RedHatEnterpriseLinuxServerrelease6.4...
WordCount_Spark!_spark_wordcount_java_
10-01
在spark上运行wordcount程序，可以进行单词计数
java实现的spark的wordcount单词技术程序
12-06
采用java实现的spark的wordcount单词技术的程序.可以跑通。
Java实现Spark词配对Wordcount计数代码实现
05-24
使用Spark实现对一个文档中的每一行的单词进行词配对计数，要求去标点符号，将大写符号统一转化成为小写单词。
spark概述及Java wordcount、Scala wordcount
w4xj的博客
06-09 227
1.spark是什么 [1].官网 http://spark.apache.org/ [2].是什么 Apache Spark™ is a fast and general engine for large-scale data processing. Apache Spark is an open source cluster computing system that aims to ...
Spark入门程序WordCount实现（Java）
机器学习，大数据
11-29 2396
文章目录1. 环境与数据准备1.1 运行环境1.2 数据准备2. 代码实现2.1 maven工程创建2.2 代码逻辑2.3 编译、打包、运行Reference 为了快速入门java与Spark工程的构建与开发，本次使用java实现Spark经典程序WordCount，过程记录如下，如有错误，请指正。 1. 环境与数据准备 1.1 运行环境 Hadoop：2.7.7 Spark：2.4.7 java：1.8.0_211 Maven：3.6.3 1.2 数据准备 使用如下命令启动Hadoo
Spark—通过Java、Scala API实现WordCount案例的基本操作
piaow_的博客
07-07 891
通过WordCount案例，体会编写spark程序时用scala语言相比较于java，作为原生语言的优越性。
Java版spark入门WordCount
qq_44962075的博客
04-19 535
最近新入职了一家公司，这家公司主要以开发大数据平台为主，主流语言用的是java语言，涉及到的组件有很多很多 kafaka、pg数据库、hbase、hudi、atlas、flink、spark、hive、hadoop、mongdb、presto、hbase、dolphinscheduler、alluxio等等，以前很少用java去写spark代码，如此一来感觉十分懵逼，以后准备更新以Java语言方向为主的大数据开发。这里先从基础开发，我这里先整理的是Java，JavaLambda，Scala三种方式的word
4 | Java Spark实现 WordCount
最新发布
微信号：RunsenLiu
09-01 1317
简单的 Java Spark 实现 WordCount 的教程，它将教您如何使用 Apache Spark 来统计文本文件中每个单词的出现次数。首先，确保您已经安装了 Apache Spark 并设置了运行环境。您需要准备一个包含文本内容的文本文件，以便对其进行 WordCount 分析。
Spark 安装及WordCount编写（Spark、Scala、java三种方法）
Mogeko1的博客
04-12 624
Spark 安装及WordCount编写（Spark、Scala、java三种方法）
spark wordcount java_spark - java 编程实现Word count
weixin_31310913的博客
02-12 268
本文通过一个demon向读者展示，如何用spark 实现word count 功能。创建项目创建maven项目，添加spark核心依赖org.apache.sparkspark-core_2.122.4.0如果使用java8版本还需要加入com.thoughtworks.paranamerparanamer2.8否则读取文件时候出现异常JavaRDD rdd1 = sc.textFile("/Us...
Spark开发工程师（含项目）
02-21
本课程为大数据金融信贷项目实战课，着重讲解企业中常用的大数据技术理论与实战，如Hadoop、Hive、HBase、Sqoop、Flume、Kafka、Spark Streaming、Spark SQL、Spark Structured Streaming等。课程包含离线项目和实时项目，从项目业务需求、技术选型、架构设计、集群安装部署、集成开发以及项目可视化进行全方位实战讲解。
spark系列 - wordCount( java版)
cj2580的博客
06-06 3332
环境说明 spark版本：1.6.1(已经安装完毕) os: centos6.5 java: 1.8 hadoop:2.3网上许多spark教程都是基于scala或者python的，通过这篇文章我们来使用spark的JavaAPI， 来写一个wordCount程序， 首先我们写个Java程序： pom.xml如下：<?xml version="1.0" encoding="UTF-8"?
Spark初探案例WordCount之Java版本
xiaohu21的博客
09-17 334
Spark初探案例WordCount之Java版本 1. 背景 Spark作为大数据处理引擎的事实标准，基本可以涵盖大数据处理的90%以上场景。Spark SQL、Streaming、MLib、Graphx 作为大数据技术中最常见的案例，wordcount既简单，又可以很好跟实际企业开发中需求关联起来。因为很多企业需求其实就是换个面目的wordcount，或者组合型的wordcount需求而已 spark提供了对多种语言的接口api支持，本文就是使用java进行验证。 2. 案例代码 注意，本案例是使用
4.spark streaming：实时wordcount程序开发
04-06
Spark Streaming是一个可以对实时数据进行流处理的框架。实时处理是指对在流中不断生成的数据进行即时处理和分析，将所需的信息提取出来。在Spark Streaming中，可以通过类似于批处理的方式对实时数据进行处理和分析，例如实时Word Count程序。这种实时Word Count程序可以用于实时监控网络中的数据流量或者社交媒体中的话题热度等场景。开发时需要使用Scala或Java编写Spark Streaming程序，并在数据接收和处理上使用Spark的API。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

SuperBigData~

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值