Spark：用Scala和Java实现WordCount

最新推荐文章于 2022-04-08 11:18:27 发布
最新推荐文章于 2022-04-08 11:18:27 发布
阅读量1.5k 收藏 2
点赞数 1
分类专栏： spark 文章标签： spark scala WordCount

注：本文博客转至http://www.cnblogs.com/byrhuangqiang/p/4017725.html


为了在IDEA中编写scala，今天安装配置学习了IDEA集成开发环境。IDEA确实很优秀，学会之后，用起来很顺手。关于如何搭建scala和IDEA开发环境，请看文末的参考资料。

用Scala和Java实现WordCount，其中Java实现的JavaWordCount是spark自带的例子（$SPARK_HOME/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java）

1.环境

  • OS:Red Hat Enterprise Linux Server release 6.4 (Santiago)
  • Hadoop:Hadoop 2.4.1
  • JDK:1.7.0_60
  • Spark:1.1.0
  • Scala:2.11.2
  • 集成开发环境：IntelliJ IDEA 13.1.3

注意：需要在客户端windows环境下安装IDEA、Scala、JDK，并且为IDEA下载scala插件。

2.Scala实现单词计数

package com.hq

/**
 * User: hadoop
 * Date: 2014/10/10 0010
 * Time: 18:59
 */
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

/**
 * 统计字符出现次数
 */
object WordCount {
  def main(args: Array[String]) {
    if (args.length < 1) {
      System.err.println("Usage: <file>")
      System.exit(1)
    }

    val conf = new SparkConf()
    val sc = new SparkContext(conf)
    val line = sc.textFile(args(0))

    line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println)//此处需要注意的是flatMap这个函数，一开始的时候我没弄懂map和flatMap导致出现返回的是一个Array，后台执行报错。
    sc.stop()
  }
}


3.Java实现单词计数

package com.hq;

/**
 * User: hadoop
 * Date: 2014/10/10 0010
 * Time: 19:26
 */

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import scala.Tuple2;

import java.util.Arrays;
import java.util.List;
import java.util.regex.Pattern;

public final class JavaWordCount {
  private static final Pattern SPACE = Pattern.compile(" ");

  public static void main(String[] args) throws Exception {

    if (args.length < 1) {
      System.err.println("Usage: JavaWordCount <file>");
      System.exit(1);
    }

    SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");
    JavaSparkContext ctx = new JavaSparkContext(sparkConf);
    JavaRDD<String> lines = ctx.textFile(args[0], 1);

    JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
      @Override
      public Iterable<String> call(String s) {
        return Arrays.asList(SPACE.split(s));
      }
    });

    JavaPairRDD<String, Integer> ones = words.mapToPair(new PairFunction<String, String, Integer>() {
      @Override
      public Tuple2<String, Integer> call(String s) {
        return new Tuple2<String, Integer>(s, 1);
      }
    });

    JavaPairRDD<String, Integer> counts = ones.reduceByKey(new Function2<Integer, Integer, Integer>() {
      @Override
      public Integer call(Integer i1, Integer i2) {
        return i1 + i2;
      }
    });

    List<Tuple2<String, Integer>> output = counts.collect();
    for (Tuple2<?, ?> tuple : output) {
      System.out.println(tuple._1() + ": " + tuple._2());
    }
    ctx.stop();
  }
}

4.IDEA打包和运行

4.1 IDEA的工程结构

在IDEA中建立Scala工程，并导入spark api编程jar包（spark-assembly-1.1.0-hadoop2.4.0.jar：$SPARK_HOME/lib/里面）



4.2 打成jar包

File ---> Project Structure 



配置完成后，在菜单栏中选择Build->Build Artifacts...，然后使用Build等命令打包。打包完成后会在状态栏中显示“Compilation completed successfully...”的信息，去jar包输出路径下查看jar包，如下所示。


ScalaTest1848.jar就是我们编程所产生的jar包，里面包含了三个类HelloWord、WordCount、JavaWordCount。

可以用这个jar包在spark集群里面运行java或者scala的单词计数程序。

 4.3 以Spark集群standalone方式运行单词计数

上传jar包到服务器，并放置在/home/ebupt/test/WordCount.jar路径下。

上传一个text文本文件到HDFS作为单词计数的输入文件：hdfs://eb170:8020/user/ebupt/text

内容如下

import org apache spark api java JavaPairRDD   
import org apache spark api java JavaRDD   
import org apache spark api java JavaSparkContext   
import org apache spark api java function FlatMapFunction   
import org apache spark api java function Function   
import org apache spark api java function Function2   
import org apache spark api java function PairFunction   
import scala Tuple2


用spark-submit命令提交任务运行，具体使用查看： spark-submit --help 
[ebupt@eb174 bin]$ spark-submit --help
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Usage: spark-submit [options] <app jar | python file> [app options]
Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --name NAME                 A name of your application.
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
                              on the PYTHONPATH for Python apps.
  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor.

  --conf PROP=VALUE           Arbitrary Spark configuration property.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.

  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 512M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).

  --help, -h                  Show this help message and exit
  --verbose, -v               Print additional debug output

 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).
  --supervise                 If given, restarts the driver on failure.

 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 YARN-only:
  --executor-cores NUM        Number of cores per executor (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.

①提交scala实现的单词计数：

[ebupt@eb174 test]$ spark-submit --master spark://eb174:7077 --name WordCountByscala --class com.hq.WordCount --executor-memory 1G --total-executor-cores 2 ~/test/WordCount.jar hdfs://eb170:8020/user/ebupt/text 

②提交java实现的单词计数：

[ebupt@eb174 test]$ spark-submit --master spark://eb174:7077 --name JavaWordCountByHQ --class com.hq.JavaWordCount --executor-memory 1G --total-executor-cores 2 ~/test/WordCount.jar hdfs://eb170:8020/user/ebupt/text

③2者运行结果类似，所以只写了一个：

Spark assembly has been built with Hive, including Datanucleus jars on classpath 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
14/10/10 19:24:51 INFO SecurityManager: Changing view acls to: ebupt, 
14/10/10 19:24:51 INFO SecurityManager: Changing modify acls to: ebupt, 
14/10/10 19:24:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ebupt, ); users with modify permissions: Set(ebupt, ) 
14/10/10 19:24:52 INFO Slf4jLogger: Slf4jLogger started 
14/10/10 19:24:52 INFO Remoting: Starting remoting 
14/10/10 19:24:52 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@eb174:56344] 
14/10/10 19:24:52 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@eb174:56344] 
14/10/10 19:24:52 INFO Utils: Successfully started service 'sparkDriver' on port 56344. 
14/10/10 19:24:52 INFO SparkEnv: Registering MapOutputTracker 
14/10/10 19:24:52 INFO SparkEnv: Registering BlockManagerMaster 
14/10/10 19:24:52 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141010192452-3398 
14/10/10 19:24:52 INFO Utils: Successfully started service 'Connection manager for block manager' on port 41110. 
14/10/10 19:24:52 INFO ConnectionManager: Bound socket to port 41110 with id = ConnectionManagerId(eb174,41110) 
14/10/10 19:24:52 INFO MemoryStore: MemoryStore started with capacity 265.4 MB 
14/10/10 19:24:52 INFO BlockManagerMaster: Trying to register BlockManager 
14/10/10 19:24:52 INFO BlockManagerMasterActor: Registering block manager eb174:41110 with 265.4 MB RAM 
14/10/10 19:24:52 INFO BlockManagerMaster: Registered BlockManager 
14/10/10 19:24:52 INFO HttpFileServer: HTTP File server directory is /tmp/spark-8051667e-bfdb-4ecd-8111-52992b16bb13 
14/10/10 19:24:52 INFO HttpServer: Starting HTTP Server 
14/10/10 19:24:52 INFO Utils: Successfully started service 'HTTP file server' on port 48233. 
14/10/10 19:24:53 INFO Utils: Successfully started service 'SparkUI' on port 4040. 
14/10/10 19:24:53 INFO SparkUI: Started SparkUI at http://eb174:4040 
14/10/10 19:24:53 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
14/10/10 19:24:53 INFO SparkContext: Added JAR file:/home/ebupt/test/WordCountByscala.jar at http://10.1.69.174:48233/jars/WordCountByscala.jar with timestamp 1412940293532 
14/10/10 19:24:53 INFO AppClient$ClientActor: Connecting to master spark://eb174:7077... 
14/10/10 19:24:53 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 
14/10/10 19:24:53 INFO MemoryStore: ensureFreeSpace(163705) called with curMem=0, maxMem=278302556 
14/10/10 19:24:53 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 159.9 KB, free 265.3 MB) 
14/10/10 19:24:53 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20141010192453-0009 
14/10/10 19:24:53 INFO AppClient$ClientActor: Executor added: app-20141010192453-0009/0 on worker-20141008204132-eb176-49618 (eb176:49618) with 1 cores 
14/10/10 19:24:53 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141010192453-0009/0 on hostPort eb176:49618 with 1 cores, 1024.0 MB RAM 
14/10/10 19:24:53 INFO AppClient$ClientActor: Executor added: app-20141010192453-0009/1 on worker-20141008204132-eb175-56337 (eb175:56337) with 1 cores 
14/10/10 19:24:53 INFO SparkDeploySchedulerBackend: Granted executor ID app-20141010192453-0009/1 on hostPort eb175:56337 with 1 cores, 1024.0 MB RAM 
14/10/10 19:24:53 INFO AppClient$ClientActor: Executor updated: app-20141010192453-0009/0 is now RUNNING 
14/10/10 19:24:53 INFO AppClient$ClientActor: Executor updated: app-20141010192453-0009/1 is now RUNNING 
14/10/10 19:24:53 INFO MemoryStore: ensureFreeSpace(12633) called with curMem=163705, maxMem=278302556 
14/10/10 19:24:53 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 12.3 KB, free 265.2 MB) 
14/10/10 19:24:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on eb174:41110 (size: 12.3 KB, free: 265.4 MB) 
14/10/10 19:24:53 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 
14/10/10 19:24:54 INFO FileInputFormat: Total input paths to process : 1 
14/10/10 19:24:54 INFO SparkContext: Starting job: collect at WordCount.scala:26 
14/10/10 19:24:54 INFO DAGScheduler: Registering RDD 3 (map at WordCount.scala:26) 
14/10/10 19:24:54 INFO DAGScheduler: Got job 0 (collect at WordCount.scala:26) with 2 output partitions (allowLocal=false) 
14/10/10 19:24:54 INFO DAGScheduler: Final stage: Stage 0(collect at WordCount.scala:26) 
14/10/10 19:24:54 INFO DAGScheduler: Parents of final stage: List(Stage 1) 
14/10/10 19:24:54 INFO DAGScheduler: Missing parents: List(Stage 1) 
14/10/10 19:24:54 INFO DAGScheduler: Submitting Stage 1 (MappedRDD[3] at map at WordCount.scala:26), which has no missing parents 
14/10/10 19:24:54 INFO MemoryStore: ensureFreeSpace(3400) called with curMem=176338, maxMem=278302556 
14/10/10 19:24:54 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.3 KB, free 265.2 MB) 
14/10/10 19:24:54 INFO MemoryStore: ensureFreeSpace(2082) called with curMem=179738, maxMem=278302556 
14/10/10 19:24:54 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.0 KB, free 265.2 MB) 
14/10/10 19:24:54 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on eb174:41110 (size: 2.0 KB, free: 265.4 MB) 
14/10/10 19:24:54 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0 
14/10/10 19:24:54 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (MappedRDD[3] at map at WordCount.scala:26) 
14/10/10 19:24:54 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks 
14/10/10 19:24:56 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@eb176:35482/user/Executor#1456950111] with ID 0 
14/10/10 19:24:56 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 0, eb176, ANY, 1238 bytes) 
14/10/10 19:24:56 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@eb175:35502/user/Executor#-1231100997] with ID 1 
14/10/10 19:24:56 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 1, eb175, ANY, 1238 bytes) 
14/10/10 19:24:56 INFO BlockManagerMasterActor: Registering block manager eb176:33296 with 530.3 MB RAM 
14/10/10 19:24:56 INFO BlockManagerMasterActor: Registering block manager eb175:32903 with 530.3 MB RAM 
14/10/10 19:24:57 INFO ConnectionManager: Accepted connection from [eb176/10.1.69.176:39218] 
14/10/10 19:24:57 INFO ConnectionManager: Accepted connection from [eb175/10.1.69.175:55227] 
14/10/10 19:24:57 INFO SendingConnection: Initiating connection to [eb176/10.1.69.176:33296] 
14/10/10 19:24:57 INFO SendingConnection: Initiating connection to [eb175/10.1.69.175:32903] 
14/10/10 19:24:57 INFO SendingConnection: Connected to [eb175/10.1.69.175:32903], 1 messages pending 
14/10/10 19:24:57 INFO SendingConnection: Connected to [eb176/10.1.69.176:33296], 1 messages pending 
14/10/10 19:24:57 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on eb175:32903 (size: 2.0 KB, free: 530.3 MB) 
14/10/10 19:24:57 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on eb176:33296 (size: 2.0 KB, free: 530.3 MB) 
14/10/10 19:24:57 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on eb176:33296 (size: 12.3 KB, free: 530.3 MB) 
14/10/10 19:24:57 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on eb175:32903 (size: 12.3 KB, free: 530.3 MB) 
14/10/10 19:24:58 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 1) in 1697 ms on eb175 (1/2) 
14/10/10 19:24:58 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 0) in 1715 ms on eb176 (2/2) 
14/10/10 19:24:58 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
14/10/10 19:24:58 INFO DAGScheduler: Stage 1 (map at WordCount.scala:26) finished in 3.593 s 
14/10/10 19:24:58 INFO DAGScheduler: looking for newly runnable stages 
14/10/10 19:24:58 INFO DAGScheduler: running: Set() 
14/10/10 19:24:58 INFO DAGScheduler: waiting: Set(Stage 0) 
14/10/10 19:24:58 INFO DAGScheduler: failed: Set() 
14/10/10 19:24:58 INFO DAGScheduler: Missing parents for Stage 0: List() 
14/10/10 19:24:58 INFO DAGScheduler: Submitting Stage 0 (ShuffledRDD[4] at reduceByKey at WordCount.scala:26), which is now runnable 
14/10/10 19:24:58 INFO MemoryStore: ensureFreeSpace(2096) called with curMem=181820, maxMem=278302556 
14/10/10 19:24:58 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.0 KB, free 265.2 MB) 
14/10/10 19:24:58 INFO MemoryStore: ensureFreeSpace(1338) called with curMem=183916, maxMem=278302556 
14/10/10 19:24:58 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1338.0 B, free 265.2 MB) 
14/10/10 19:24:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on eb174:41110 (size: 1338.0 B, free: 265.4 MB) 
14/10/10 19:24:58 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0 
14/10/10 19:24:58 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (ShuffledRDD[4] at reduceByKey at WordCount.scala:26) 
14/10/10 19:24:58 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 
14/10/10 19:24:58 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 2, eb175, PROCESS_LOCAL, 1008 bytes) 
14/10/10 19:24:58 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 3, eb176, PROCESS_LOCAL, 1008 bytes) 
14/10/10 19:24:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on eb175:32903 (size: 1338.0 B, free: 530.3 MB) 
14/10/10 19:24:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on eb176:33296 (size: 1338.0 B, free: 530.3 MB) 
14/10/10 19:24:58 INFO MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@eb175:59119 
14/10/10 19:24:58 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 144 bytes 
14/10/10 19:24:58 INFO MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@eb176:39028 
14/10/10 19:24:58 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 3) in 109 ms on eb176 (1/2) 
14/10/10 19:24:58 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 2) in 120 ms on eb175 (2/2) 
14/10/10 19:24:58 INFO DAGScheduler: Stage 0 (collect at WordCount.scala:26) finished in 0.123 s 
14/10/10 19:24:58 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
14/10/10 19:24:58 INFO SparkContext: Job finished: collect at WordCount.scala:26, took 3.815637915 s 
(scala,1) 
(Function2,1) 
(JavaSparkContext,1) 
(JavaRDD,1) 
(Tuple2,1) 
(,1) 
(org,7) 
(apache,7) 
(JavaPairRDD,1) 
(java,7) 
(function,4) 
(api,7) 
(Function,1) 
(PairFunction,1) 
(spark,7) 
(FlatMapFunction,1) 
(import,8) 
14/10/10 19:24:58 INFO SparkUI: Stopped Spark web UI at http://eb174:4040 
14/10/10 19:24:58 INFO DAGScheduler: Stopping DAGScheduler 
14/10/10 19:24:58 INFO SparkDeploySchedulerBackend: Shutting down all executors 
14/10/10 19:24:58 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 
14/10/10 19:24:58 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(eb176,33296) 
14/10/10 19:24:58 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(eb176,33296) 
14/10/10 19:24:58 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(eb176,33296) not found 
14/10/10 19:24:58 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(eb175,32903) 
14/10/10 19:24:58 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(eb175,32903) 
14/10/10 19:24:58 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(eb175,32903) 
14/10/10 19:24:58 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@5e92c11b 
14/10/10 19:24:58 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@5e92c11b 
java.nio.channels.CancelledKeyException 
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:310) 
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 
14/10/10 19:24:59 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped! 
14/10/10 19:24:59 INFO ConnectionManager: Selector thread was interrupted! 
14/10/10 19:24:59 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(eb176,33296) 
14/10/10 19:24:59 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(eb176,33296) not found 
14/10/10 19:24:59 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(eb176,33296) 
14/10/10 19:24:59 WARN ConnectionManager: All connections not cleaned up 
14/10/10 19:24:59 INFO ConnectionManager: ConnectionManager stopped 
14/10/10 19:24:59 INFO MemoryStore: MemoryStore cleared 
14/10/10 19:24:59 INFO BlockManager: BlockManager stopped 
14/10/10 19:24:59 INFO BlockManagerMaster: BlockManagerMaster stopped 
14/10/10 19:24:59 INFO SparkContext: Successfully stopped SparkContext 
14/10/10 19:24:59 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 
14/10/10 19:24:59 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 
14/10/10 19:24:59 INFO Remoting: Remoting shut down 
14/10/10 19:24:59 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

5.参考资料

关于IDEA的使用：Scala从零开始：使用Intellij IDEA写hello world

scala编写WC: Spark wordcount开发并提交到集群运行

java编写WC:用java编写spark程序，简单示例及运行、Spark在Yarn上运行Wordcount程序

Spark Programming Guide




coco_ethan
关注
专栏目录
Spark实现WordCount
春水初生，春林初盛，春风十里，不如随笔
11-08 9769
首先，编写第一个Spark应用程序 ，我们是如何建立起来的，其入口在哪里呢，需要创建两个对象。  一：val conf = new SparkConf()                             .setAppName("WordCount")                             .setMaster("local")     创建SparkConf对象...
Spark：用Java和Scala实现WordCount
菜如张学清的博客
12-06 550
Java版本package cn.spark.study.core;import java.util.Arrays;import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.sp
spark实现wordcount
csncd的博客
04-08 4076
创建spark上下文环境 def main(args: Array[String]): Unit = { val sc = new SparkContext(new SparkConf().setMaster("local[*]").setAppName("wc")) val rdd: RDD[String] = sc.makeRDD(List( "spark hello", "hive", "hadoop hbase", "spark ha
spark下实现wordcount
11-14
亲自测试的
Spark入门：Scala环境配置与WordCount实战
3. **安装JDK**: 作为Java开发工具包，JDK是运行Scala和Spark程序必不可少的组件。至少需要1.6版本以上的JDK，推荐使用更高版本。 4. **创建并配置Spark工程**: 在ScalaIDE中，创建一个新的Scala项目，命名为"spark...
IDEA下Scala与Java实现Spark WordCount教程
JavaWordCount是Spark自带的示例，它同样实现了WordCount算法，但使用的是Java语言。相比之下，Scala的语法更加简洁，因为它是一种函数式编程语言，提供了强大的集合操作和表达能力。 这篇文章详细地展示了如何在...
spark实现word count程序
健康平安的活着的专栏
05-30 489
5.1 使用scala 实现wordcount程序 object WordCount { def main(args:Array[String]): Unit ={ //1.创建sparkConf对象，设置appName和master地址，local[2]表示本地采用2个线程去执行 val sparkConf:SparkConf=new SparkConf().setAppName("wordCount").setMaster("local[2]") //2..
spark运行wordcount例子
10-31
本例子是让新手熟悉了解如果用idea开发程序，提交到spakr集群上读取hdfs数据这么一个流程。博客讲解链接：https://blog.csdn.net/cuicanxingchen123456/article/details/83573838
spark实现WordCount(多种实现方式)
qq_44160357的博客
02-21 2132
第一种方式: 文件内容: object WordCount { def main(args: Array[String]): Unit = { val conf: SparkConf = new SparkConf().setMaster("local").setAppName("WordCount") /// 创建Spark上下文对象 val sc = new ...
Spark：wordcount
从一点一滴做起
06-12 275
object text { def main(args: Array[String]): Unit = { // 创建sparksession val sparkSession = SparkSession.builder().master("local").appName("Word_Count").getOrCreate() import sparkSessio...
【Spark】Spark实现WordCount功能讲解
晚风中的自由
10-27 348
一、读取hdfs上的文件作为rdd hdfs的文件/user/root/mapreduce/wordcount/input/wc.input 有以下内容 hadoop hive hive hadoop hbase sqoop hbase sqoop hadoop hive 启动spark-shell bin/spark-shell 读取hdfs的文件/user/root/mapr...
用Spark实现word Count实例——入门大数据实例
qq_30899339的博客
06-14 7936
参考资料： https://blog.csdn.net/weixin_43893397/article/details/105289992 https://zhuanlan.zhihu.com/p/75779188 https://blog.csdn.net/u010675669/article/details/81744386 https://blog.csdn.net/weixin_43924642/article/details/89388432 首先将需要的input文件本地传至HDF.
Spark实现WordCount的11种方式
weixin_43750933的博客
09-17 610
方法1: groupBy import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} object WordCount_01_groupBy { def main(args: Array[String]): Unit = { val conf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("WordCount.
Spark-Word Count实例
Dead_morning的博客
05-03 1157
运行环境：Spark1.4集群 语言:Scala一、简单实例演示1、文本内容：2、操作Step1. 读取hdfs 上文件：val wcrdd1=sc.textFile("hdfs://master:9000/wordcount/words.txt").cache Step2. flatMap 操作：将文本以空格形式切割val wcrdd2=wcrdd1.flatMap(_.split(" "))
Spark实现的hello world<转>
yongjian_luo的专栏
01-08 1736
spark,又一个传说中的分布式实现,详情:http://spark-project.org/, 安装这里就不写了，因为网上已有中文介绍，这里主要是介绍一下入门，和hadoop一样，学习的时候，首先学习spark提供的字符统计例子：javaWordCount 原始代码如下： Java代码   import scala.Tuple2;   import spar
Spark用Java实现的WordCount
大数据
12-04 1089
java版本的sparkWordCount /** * Java版本的spark WordCount */ public class JavaWordCount { public static void main(String[] args) { //1.先创建conf对象进行配置，主要是设置名称，为了设置运行模式 SparkConf conf = ne...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值