spark-wordcount

最新推荐文章于 2024-05-06 08:39:42 发布
最新推荐文章于 2024-05-06 08:39:42 发布
阅读量1.4k 收藏 1
点赞数
分类专栏： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xiewenbo/article/details/22943273
版权

在本地模式运行spark wordcount，需要引入spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar  需要引入SparkContext._ 否则reduceByKey无法使用

package com.spark
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object wordcount {
  def main(args: Array[String]) {
    val sc = new SparkContext("local", "Simple App", "E:\\00_spark_scala\\spark\\spark-0.9.0-incubating-bin-hadoop1",
    							null)
    							//List("lib/spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar")
    val logFile = sc.textFile("hdfs://namenode:9000/user/input/input.txt") // Should be some file on your system
    val file = sc.textFile("D:\\test.txt")
    val counts = logFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
    println(counts)
    counts.saveAsTextFile("D:\\output\\20140404")
  }
}
input:

keyword1
keyword3
keyword2

cosole output:

log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
14/04/04 16:13:31 INFO SparkEnv: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/04/04 16:13:31 INFO SparkEnv: Registering BlockManagerMaster
14/04/04 16:13:31 INFO DiskBlockManager: Created local directory at C:\Users\WENBO_~1\AppData\Local\Temp\spark-local-20140404161331-aac1
14/04/04 16:13:31 INFO MemoryStore: MemoryStore started with capacity 1068.8 MB.
14/04/04 16:13:31 INFO ConnectionManager: Bound socket to port 49790 with id = ConnectionManagerId(XA-NA18818395.allyes.group,49790)
14/04/04 16:13:31 INFO BlockManagerMaster: Trying to register BlockManager
14/04/04 16:13:31 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager XA-NA18818395.allyes.group:49790 with 1068.8 MB RAM
14/04/04 16:13:31 INFO BlockManagerMaster: Registered BlockManager
14/04/04 16:13:31 INFO HttpServer: Starting HTTP Server
14/04/04 16:13:31 INFO HttpBroadcast: Broadcast server started at http://10.200.33.176:49791
14/04/04 16:13:31 INFO SparkEnv: Registering MapOutputTracker
14/04/04 16:13:31 INFO HttpFileServer: HTTP File server directory is C:\Users\WENBO_~1\AppData\Local\Temp\spark-19614fe9-9131-4382-8491-8725dd29bfdf
14/04/04 16:13:31 INFO HttpServer: Starting HTTP Server
14/04/04 16:13:31 INFO SparkUI: Started Spark Web UI at http://XA-NA18818395.allyes.group:4040
14/04/04 16:13:31 INFO MemoryStore: ensureFreeSpace(32960) called with curMem=0, maxMem=1120744243
14/04/04 16:13:31 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 32.2 KB, free 1068.8 MB)
14/04/04 16:13:31 INFO MemoryStore: ensureFreeSpace(32960) called with curMem=32960, maxMem=1120744243
14/04/04 16:13:31 INFO MemoryStore: Block broadcast_1 stored as values to memory (estimated size 32.2 KB, free 1068.8 MB)
14/04/04 16:13:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/04 16:13:32 WARN LoadSnappy: Snappy native library not loaded
14/04/04 16:13:32 INFO FileInputFormat: Total input paths to process : 1
MapPartitionsRDD[8] at reduceByKey at wordcount.scala:12
14/04/04 16:13:32 INFO SparkContext: Starting job: saveAsTextFile at wordcount.scala:14
14/04/04 16:13:32 INFO DAGScheduler: Registering RDD 6 (reduceByKey at wordcount.scala:12)
14/04/04 16:13:32 INFO DAGScheduler: Got job 0 (saveAsTextFile at wordcount.scala:14) with 1 output partitions (allowLocal=false)
14/04/04 16:13:32 INFO DAGScheduler: Final stage: Stage 0 (saveAsTextFile at wordcount.scala:14)
14/04/04 16:13:32 INFO DAGScheduler: Parents of final stage: List(Stage 1)
14/04/04 16:13:32 INFO DAGScheduler: Missing parents: List(Stage 1)
14/04/04 16:13:32 INFO DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[6] at reduceByKey at wordcount.scala:12), which has no missing parents
14/04/04 16:13:32 INFO DAGScheduler: Submitting 1 missing tasks from Stage 1 (MapPartitionsRDD[6] at reduceByKey at wordcount.scala:12)
14/04/04 16:13:32 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
14/04/04 16:13:32 INFO TaskSetManager: Starting task 1.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL)
14/04/04 16:13:32 INFO TaskSetManager: Serialized task 1.0:0 as 1919 bytes in 4 ms
14/04/04 16:13:32 INFO Executor: Running task ID 0
14/04/04 16:13:32 INFO BlockManager: Found block broadcast_0 locally
14/04/04 16:13:32 INFO HadoopRDD: Input split: hdfs://namenode:9000/user/input/input.txt:0+27
14/04/04 16:13:32 INFO Executor: Serialized size of result for 0 is 762
14/04/04 16:13:32 INFO Executor: Sending result for 0 directly to driver
14/04/04 16:13:32 INFO Executor: Finished task ID 0
14/04/04 16:13:32 INFO TaskSetManager: Finished TID 0 in 188 ms on localhost (progress: 0/1)
14/04/04 16:13:32 INFO TaskSchedulerImpl: Remove TaskSet 1.0 from pool 
14/04/04 16:13:32 INFO DAGScheduler: Completed ShuffleMapTask(1, 0)
14/04/04 16:13:32 INFO DAGScheduler: Stage 1 (reduceByKey at wordcount.scala:12) finished in 0.195 s
14/04/04 16:13:32 INFO DAGScheduler: looking for newly runnable stages
14/04/04 16:13:32 INFO DAGScheduler: running: Set()
14/04/04 16:13:32 INFO DAGScheduler: waiting: Set(Stage 0)
14/04/04 16:13:32 INFO DAGScheduler: failed: Set()
14/04/04 16:13:32 INFO DAGScheduler: Missing parents for Stage 0: List()
14/04/04 16:13:32 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[9] at saveAsTextFile at wordcount.scala:14), which is now runnable
14/04/04 16:13:32 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (MappedRDD[9] at saveAsTextFile at wordcount.scala:14)
14/04/04 16:13:32 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
14/04/04 16:13:32 INFO TaskSetManager: Starting task 0.0:0 as TID 1 on executor localhost: localhost (PROCESS_LOCAL)
14/04/04 16:13:32 INFO TaskSetManager: Serialized task 0.0:0 as 5336 bytes in 0 ms
14/04/04 16:13:32 INFO Executor: Running task ID 1
14/04/04 16:13:32 INFO BlockManager: Found block broadcast_0 locally
14/04/04 16:13:32 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 1 non-zero-bytes blocks out of 1 blocks
14/04/04 16:13:32 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  3 ms
14/04/04 16:13:32 INFO FileOutputCommitter: Saved output of task 'attempt_201404041613_0000_m_000000_1' to file:/D:/output/20140404
14/04/04 16:13:32 INFO SparkHadoopWriter: attempt_201404041613_0000_m_000000_1: Committed
14/04/04 16:13:32 INFO Executor: Serialized size of result for 1 is 817
14/04/04 16:13:32 INFO Executor: Sending result for 1 directly to driver
14/04/04 16:13:32 INFO Executor: Finished task ID 1
14/04/04 16:13:32 INFO TaskSetManager: Finished TID 1 in 65 ms on localhost (progress: 0/1)
14/04/04 16:13:32 INFO TaskSchedulerImpl: Remove TaskSet 0.0 from pool 
14/04/04 16:13:32 INFO DAGScheduler: Completed ResultTask(0, 0)
14/04/04 16:13:32 INFO DAGScheduler: Stage 0 (saveAsTextFile at wordcount.scala:14) finished in 0.066 s
14/04/04 16:13:32 INFO SparkContext: Job finished: saveAsTextFile at wordcount.scala:14, took 0.358104066 s

part-00000:

(keyword1,1)
(keyword3,1)
(keyword2,1)


run spark-shell as distributed mode :  

  • spark://hostname:7077 ./spark-shell
这里的hostname从http://namenode:8080/ 页面去查看 


package com.sc.spark
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object wordcount {
  def show(args : (String, Int)){
    println(args._1+" "+args._2)
  }
  def main(args: Array[String]) {
    val sc = new SparkContext("local", "Simple App", "E:\\00_spark_scala\\spark\\spark-0.9.0-incubating-bin-hadoop1",
    							null)
    							//List("lib/spark-assembly_2.10-0.9.0-incubating-hadoop1.0.4.jar")
    val logFile = sc.textFile("hdfs://namenode:9000/user/input/input.txt") // Should be some file on your system
    val file = sc.textFile("D:\\test.txt")
    val counts = logFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
    counts.foreach(f  => println(f._1+" "+f._2))
    println("word counts:"+counts)
    var filteres = counts.filter(f => f._2>2)
    println("filteres:"+filteres)
    filteres.foreach(f  => println(f._1+" "+f._2))
    var arrs = counts.collect;
    println("===================================")
    arrs.foreach(f => show(f))
    var res = counts.lookup("keyword3")
    println("counts.lookup:"+res(0))
    println("res.length:"+res.length)
    //counts.saveAsTextFile("D:\\output\\20140404")
  }
}

通过命令行提交任务：

hadoop fs -rmr hdfs://localhost:9000/user/output-2
./bin/spark-submit \
  --class wordcount \
  --master spark://192.168.72.128:7077 \
  --num-executors 1 \
  --executor-cores 1 \
  --driver-memory 1g \
  --executor-memory 1g \
   --jars  /opt/spark/lib/spark-assembly-1.0.1-hadoop2.2.0.jar \
  /home/xiewenbo/wordcount.jar \
  spark://192.168.72.128:7077  \
  hdfs://localhost:9000/user/input.txt \
  hdfs://localhost:9000/user/output-2



import org.apache.spark._
import org.apache.spark.SparkContext._
object wordcount {
  def main(args: Array[String]) {
    if (args.length != 3 ){
      println("usage is org.test.WordCount <master> <input> <output>")
      return
    }else{
      println("args[0]:"+args(0));
      println("args[1]:"+args(1));
      println("args[2]:"+args(2));
    }
    val sc = new SparkContext(args(0), "WordCount",System.getenv("SPARK_HOME"), null)
    val textFile = sc.textFile(args(1))
    val result = textFile.flatMap(line => line.split("\\s+"))
        .map(word => (word, 1)).reduceByKey(_ + _)
    result.saveAsTextFile(args(2))
  }
}

xiewenbo
关注
专栏目录
spark-assembly_2.10-0.9.0-incubating
09-15
spark正常运行所需要的jar包，适用于spark1版本, 一个jar包中包含所有使用spark编程所需要的关键类
Spark部署
lianhedaxue的专栏
07-04 337
Spark部署 Spark应用程序使用spark-submit(shell命令)来部署在集群中的Spark应用程序。它通过一个统一的接口采用全各自的集群管理器。因此，您不必每一个应用程序配置。 示例 让我们同样以计算字数为例子，在使用之前，使用shell命令。 在这里，我们考虑同样 spark 应用程序的例子。 简单输入 下面的文字是输入数据，并命
在 Yarn 上 安装 Spark 0.9.0
samhacker的专栏
02-23 5370
今天在自己的Yarn cluster 上搭建了Spark 0.9.0，这里粗要地记录一下主要步骤。 详细的步骤主要参照了以下两篇blog，在此对两位作者表示感谢！ -Apache Spark学习：将Spark部署到Hadoop 2.2.0上： http://dongxicheng.org/framework-on-yarn/build-spark-on-hadoop-2-yarn/ -sp
手写 Spark-WordCount
星空下的那个人影
06-18 347
手写wordcount
spark-wordCount案例
互联网知识分享
05-14 123
如果在平常启动spark-shell --master spark://bigData113:7077的时候，出现一下情况，那就检查下所有节点的进程是不是还开启的，或者关机重启，就解决。（4）注意：读写最好在HDFS里面读写，如果在本地读，可能其他的节点本地是没文件的会报错，而写到本地，可能并不是所有节点都可以写到本地，有的没有参与计算。或用这种格式 file:\\\\\\C:\\Users\\HMTX\\Desktop\\tmp\\file.txt。
Spark-WordCount案例
dafsq的博客
03-06 257
Spark-WordCount案例
spark-WordCount 源码分析图解
Angzush的博客
09-14 702
spark-WordCount 源码分析图解
Spark--wordcount的9种实现方式
小兔子乖乖
09-22 214
object Spark_rdd_01 { def main(args: Array[String]): Unit = { val conf = new SparkConf().setAppName("RDD").setMaster("local[*]") val sc = new SparkContext(conf) wordcount1(sc) println("-------") wordcount2(sc) println("-------")
Spark快速上手-WordCount案例
a369966697的博客
05-06 834
自我介绍一下，小编13年上海交大毕业，曾经在小公司待过，也去过华为、OPPO等大厂，18年进入阿里一直到现在。深知大多数Java工程师，想要提升技能，往往是自己摸索成长，自己不成体系的自学效果低效漫长且无助。因此收集整理了一份《2024年Java开发全套学习资料》，初衷也很简单，就是希望能够帮助到想自学提升又不知道该从何学起的朋友，同时减轻大家的负担。
Spark-wordcount:spark wordcount示例| 在Eclipse + Maven + Scala Project + Spark中构建
04-28
Eclipse+Maven+Scala Project+Spark | 编译并打包wordcount程序 学习用Eclipse+Maven来构建并打包一个简单的单词统计的例程。 第一步 在EclipseIDE中安装Scala插件 在Eclipse中安装Scala插件 第二步 创建Scala ...
spark-1.6.0-bin-hadoop2.4.tgz
12-16
6. **测试运行**：使用简单的Spark应用，如WordCount，验证Spark是否安装和配置成功。 在实际应用中，Spark可以与其他大数据工具如Hive、HBase、Cassandra等结合使用，提供更强大的数据分析功能。此外，Spark的弹性...
Spark-通过 API实现WordCount案例的数据集和jar包
07-12
下载资料后，免费获取数据集和jar包
Spark 之WordCount
congge_study的博客
04-10 6943
Spark 之WordCount
Spark-wordCount程序
weixin_42782897的博客
05-14 133
object WordCount{ def main (args: Array[String]){ val conf = new SparkConf().setAppName("WC") val sc = new SparkContext(conf) val rdd1 = sc.textFile(args(0...
【Spark-WordCount】Spark做简单的WordCount
weixin_53543905的博客
01-23 198
【代码】【Spark-WordCount】Spark做简单的WordCount。
run spark on local
04-04 1066
run a simple spark example on local
Amazon S3：S3静态网站托管教程.docx
最新发布
11-12
Amazon S3：S3静态网站托管教程.docx
掌握Spark-submit：部署与应用提交指南
学习如何编写一个Spark应用程序，包括基础的Scala或Python代码，如Spark WordCount示例，这通常是使用Scala的`SparkContext`对象进行操作。 2. **部署模式（Deploy Mode）**： Spark-submit支持两种部署模式：...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值