spark中wordCount案例

最新推荐文章于 2023-03-06 19:46:29 发布
最新推荐文章于 2023-03-06 19:46:29 发布
阅读量310 收藏 1
点赞数
分类专栏： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/LULU_lulu666/article/details/103122328
版权

数据源

http://bigdata.edu360.cn/laozhang
http://bigdata.edu360.cn/laozhang
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laoduan
http://bigdata.edu360.cn/laoduan
http://javaee.edu360.cn/xiaoxu
http://javaee.edu360.cn/xiaoxu
http://javaee.edu360.cn/laoyang
http://javaee.edu360.cn/laoyang
http://javaee.edu360.cn/laoyang
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laoduan
http://bigdata.edu360.cn/laoduan
http://javaee.edu360.cn/xiaoxu
http://javaee.edu360.cn/xiaoxu
http://javaee.edu360.cn/laoyang
http://javaee.edu360.cn/laoyang
http://javaee.edu360.cn/laoyang
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laozhao
http://bigdata.edu360.cn/laoduan
http://bigdata.edu360.cn/laoduan
http://javaee.edu360.cn/xiaoxu
http://javaee.edu360.cn/xiaoxu
http://javaee.edu360.cn/laoyang
http://javaee.edu360.cn/laoyang
http://javaee.edu360.cn/laoyang
http://php.edu360.cn/laoli
http://php.edu360.cn/laoliu
http://php.edu360.cn/laoli
http://php.edu360.cn/laoli

测试一：查询姓名

package com.grace.scalawc

/**
 * 查询姓名
 */
object wcTest {
  def main(args: Array[String]): Unit = {
    val sr:String = "http://bigdata.edu360.cn/laozhang"

    val x:Int = sr.lastIndexOf("/")
    val m:String = sr.substring(x+1)
    print(m)
    //结果：laozhang
  }
}

测试二：按照名字分组

package com.grace.scalawc

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

/**
 * 按照名字分组
 */
object wcTest2 {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("wcTeast2").setMaster("local")
    val sc = new SparkContext(conf)
    val lines = sc.textFile(args(0))
    val wordAndOne: RDD[(String, Int)] = lines.map(line => {

      val i = line.lastIndexOf("/")
      val person = line.substring(i+1)
      (person,1)
    })
    val reduced: RDD[(String, Int)] = wordAndOne.reduceByKey((x,y) => x+y)
    val sorted: RDD[(String, Int)] = reduced.sortBy(_._2,false)
    val tuples = sorted.collect()

    print("........................."+tuples.toBuffer)
    sc.stop()
    //结果：.........................ArrayBuffer((laozhao,15), (laoyang,9), (laoduan,6), (xiaoxu,6), (laoli,3), (laozhang,2), (laoliu,1))
  }
}

测试三：查询同一职业的相同名字的前三名

package com.grace.scalawc

import java.net.URL

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

/**
 * 查询同一职业的相同名字的前三名
 * 想要的结果：（（javaee，xiaoxu），6)
 */
object wcTest3 {
  def main(args: Array[String]): Unit = {
    /*  部分数据源
      http://bigdata.edu360.cn/laozhang
      http://bigdata.edu360.cn/laozhao
      http://javaee.edu360.cn/xiaoxu
        */
  //    spark配置
    val conf: SparkConf = new SparkConf()
    conf.setAppName("wcTest3")
    conf.setMaster("local")
    val sc: SparkContext = new SparkContext(conf)
//  指定读取文件路径
//    val lines: RDD[String] = sc.textFile(args(0)) //方法一
    val lines: RDD[String] = sc.textFile("D:\\Study\\实训\\Java基础\\hadoop\\spark\\person.log")//方法二
    //map():第一次处理数据；（job,person）=> key;6 => value
    val personJobAndOne: RDD[((String, String), Int)] = lines.map(line => {
      val i: Int = line.lastIndexOf("/")
      val person: String = line.substring(i + 1)
      val urlString: String = line.substring(0, i)
      val url: URL = new URL(urlString)
      val hostname: String = url.getHost
      val str: Array[String] = hostname.split("\\.")
      val job: String = str(0)
      ((job, person), 1)
//ArrayBuffer(((bigdata,laozhang),1), ((bigdata,laozhang),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laoduan),1), ((bigdata,laoduan),1), ((javaee,xiaoxu),1), ((javaee,xiaoxu),1), ((javaee,laoyang),1), ((javaee,laoyang),1), ((javaee,laoyang),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laoduan),1), ((bigdata,laoduan),1), ((javaee,xiaoxu),1), ((javaee,xiaoxu),1), ((javaee,laoyang),1), ((javaee,laoyang),1), ((javaee,laoyang),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laozhao),1), ((bigdata,laoduan),1), ((bigdata,laoduan),1), ((javaee,xiaoxu),1), ((javaee,xiaoxu),1), ((javaee,laoyang),1), ((javaee,laoyang),1), ((javaee,laoyang),1), ((php,laoli),1), ((php,laoliu),1), ((php,laoli),1), ((php,laoli),1))
    })
    //聚合，把相同key(bigdata,laozhang)聚合在一起
    val keySame: RDD[((String, String), Int)] = personJobAndOne.reduceByKey(_+_)//方法一
//    personJobAndOne.reduceByKey((x,y) =>  x+y)//方法二
    //ArrayBuffer(((javaee,xiaoxu),6), ((php,laoliu),1), ((bigdata,laozhang),2), ((bigdata,laozhao),15), ((javaee,laoyang),9), ((php,laoli),3), ((bigdata,laoduan),6))
    //按照javaee排序_(String, String), Int)     _1代表(String, String)     _1第一个String
    val jobGroup: RDD[(String, Iterable[((String, String), Int)])] = keySame.groupBy(_._1._1)
    //ArrayBuffer((javaee,CompactBuffer(((javaee,xiaoxu),6), ((javaee,laoyang),9))), (php,CompactBuffer(((php,laoliu),1), ((php,laoli),3))), (bigdata,CompactBuffer(((bigdata,laozhang),2), ((bigdata,laozhao),15), ((bigdata,laoduan),6))))
        //key不变，对每个Iterable[((String, String), Int)]进行处理
    val valueList: RDD[(String, List[((String, String), Int)])] = jobGroup.mapValues(x => {
          val valueList: List[((String, String), Int)] = x.toList
          //按照value排序_代表(String, String), Int)   _2表示Int
          val valueGroup: List[((String, String), Int)] = valueList.sortBy(_._2)
          //倒序
          val valueReverse: List[((String, String), Int)] = valueGroup.reverse
          //ArrayBuffer((javaee,List(((javaee,laoyang),9), ((javaee,xiaoxu),6))), (php,List(((php,laoli),3), ((php,laoliu),1))), (bigdata,List(((bigdata,laozhao),15), ((bigdata,laoduan),6), ((bigdata,laozhang),2))))
          //取前2名
          val three: List[((String, String), Int)] = valueReverse.take(2)
          //重点不能忘记three，作用是向下传递
          three
    //ArrayBuffer((javaee,List(((javaee,laoyang),9), ((javaee,xiaoxu),6))), (php,List(((php,laoli),3), ((php,laoliu),1))), (bigdata,List(((bigdata,laozhao),15), ((bigdata,laoduan),6))))          
    })
    val tuples = valueList.collect()
    System.out.println(tuples.toBuffer)
    sc.stop()
  }
}

测试四：过滤器：每个job的前三名

package com.grace.scalawc

import java.net.URL

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}


/**
 * 过滤器：每个job的前三名
 */
object PersonFilter {
  private val jobs: Array[String] = Array("javaee","php","bigdata")
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf()
    conf.setAppName("PersonFilter")
    conf.setMaster("local")
    val sc: SparkContext = new SparkContext(conf)
    val lines: RDD[String] = sc.textFile(args(0))
    val personAndJob: RDD[((String, String), Int)] = lines.map(line => {
      val i: Int = line.lastIndexOf("/")
      val person: String = line.substring(i + 1)
      val urlStr: String = line.substring(0, i)
      val urlString: URL = new URL(urlStr)
      val urlhost: String = urlString.getHost
      val jobString: Array[String] = urlhost.split("\\.")
      val job: String = jobString(0)
      ((job, person), 1)
    })
    val reduced: RDD[((String, String), Int)] = personAndJob.reduceByKey(_+_)
    //ArrayBuffer(((javaee,xiaoxu),6), ((php,laoliu),1), ((bigdata,laozhang),2), ((bigdata,laozhao),15), ((javaee,laoyang),9), ((php,laoli),3), ((bigdata,laoduan),6))
    for (job <- jobs){
      val filter: RDD[((String, String), Int)] = reduced.filter(_._1._1  == job)
      val reverse: Array[((String, String), Int)] = filter.sortBy(_._2,true).take(3).reverse
      reverse
      println(reverse.toBuffer)
      //ArrayBuffer(((javaee,laoyang),9), ((javaee,xiaoxu),6))
      //ArrayBuffer(((php,laoli),3), ((php,laoliu),1))
      //ArrayBuffer(((bigdata,laozhao),15), ((bigdata,laoduan),6), ((bigdata,laozhang),2))
    }

//    val tuples = reduced.collect()
//    System.out.println(tuples.toBuffer)
    sc.stop()
  }
}

测试五：自定义分区（一）

package com.grace.scalawc

import java.io.File
import java.net.URL

import org.apache.commons.io.FileUtils
import org.apache.spark.rdd.RDD
import org.apache.spark.{Partitioner, SparkConf, SparkContext}

import scala.collection.mutable

/**
 * 分区
 */
object PersonPartitioner {
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf()
    conf.setAppName("PersonPartitioner")
    conf.setMaster("local")
    val sc: SparkContext = new SparkContext(conf)
    val lines: RDD[String] = sc.textFile("D:\\Study\\实训\\Java基础\\hadoop\\spark\\person.log")
    val personAndJob: RDD[((String, String), Int)] = lines.map(line => {
      val i: Int = line.lastIndexOf("/")
      val person: String = line.substring(i + 1)
      val urlStr: String = line.substring(0, i)
      val urlString: URL = new URL(urlStr)
      val urlhost: String = urlString.getHost
      val jobString: Array[String] = urlhost.split("\\.")
      val job: String = jobString(0)
      ((job, person), 1)
    })


//  jobs:(javaee,php,bigdata)
    val job: Array[String] = personAndJob.map(_._1._1).distinct().collect()
    val ownPartitioner: MyPartitioner2 = new MyPartitioner2(job)

    val reduced: RDD[((String, String), Int)] = personAndJob.reduceByKey(ownPartitioner,_+_)
    //.........reduced:    ArrayBuffer(((javaee,xiaoxu),6), ((php,laoliu),1), ((bigdata,laozhang),2), ((bigdata,laozhao),15), ((javaee,laoyang),9), ((php,laoli),3), ((bigdata,laoduan),6))

    val sorted: RDD[((String, String), Int)] = reduced.mapPartitions(it => {
      it.toList.sortBy(_._2).reverse.iterator
    })
    //.........sorted:    ArrayBuffer(((bigdata,laozhao),15), ((javaee,laoyang),9), ((bigdata,laoduan),6), ((javaee,xiaoxu),6), ((php,laoli),3), ((bigdata,laozhang),2), ((php,laoliu),1))
    val tuplesArr: Array[((String, String), Int)] = sorted.collect()
//.........tuplesArr:    ArrayBuffer(((bigdata,laozhao),15), ((javaee,laoyang),9), ((bigdata,laoduan),6), ((javaee,xiaoxu),6), ((php,laoli),3), ((bigdata,laozhang),2), ((php,laoliu),1))
//    写入数据到文件
    val file = new File("D:\\personAndJob.log")
    if(file.exists()) {
      FileUtils.deleteDirectory(file)
    }
    sorted.saveAsTextFile("D:\\personAndJob.log")



//        val tuples = sorted.collect()
//        System.out.println(".........tuples:    "+tuplesArr.toBuffer)
    sc.stop()
  }
}

class MyPartitioner2(jobs:Array[String]) extends Partitioner{
//  print("...jobs:    "+jobs.toBuffer)
  //...jobs:    ArrayBuffer(javaee, php, bigdata)
  private val rules: mutable.HashMap[String, Int] = new mutable.HashMap[String,Int]()

  var i = 0
  for (job <- jobs){
    rules.put(job,i)
    i += 1
//    print("...rules:    "+rules)
    //...rules:    Map(javaee -> 0)...rules:    Map(javaee -> 0, php -> 1)...rules:    Map(javaee -> 0, php -> 1, bigdata -> 2)
  }

  override def numPartitions: Int = jobs.length

  override def getPartition(key: Any): Int = {
//    key是reduced: RDD[((String, String), Int)]中的key
//    print("...key:  "+key.toString)
//...key:  (bigdata,laozhang)...key:  (bigdata,laozhang)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laoduan)...key:  (bigdata,laoduan)...key:  (javaee,xiaoxu)...key:  (javaee,xiaoxu)...key:  (javaee,laoyang)...key:  (javaee,laoyang)...key:  (javaee,laoyang)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laoduan)...key:  (bigdata,laoduan)...key:  (javaee,xiaoxu)...key:  (javaee,xiaoxu)...key:  (javaee,laoyang)...key:  (javaee,laoyang)...key:  (javaee,laoyang)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laozhao)...key:  (bigdata,laoduan)...key:  (bigdata,laoduan)...key:  (javaee,xiaoxu)...key:  (javaee,xiaoxu)...key:  (javaee,laoyang)...key:  (javaee,laoyang)...key:  (javaee,laoyang)...key:  (php,laoli)...key:  (php,laoliu)...key:  (php,laoli)...key:  (php,laoli)
    val job: String = key.asInstanceOf[(String,String)]._1
    //    print("...rulesJob:    "+job)
    //...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    bigdata...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    javaee...rulesJob:    php...rulesJob:    php...rulesJob:    php...rulesJob:    php19/11/19 16:13:03 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1027 bytes result sent to driver    rules(job)
      rules(job)
  }
}

在D:\personAndJob.log盘中出现

分区成功！

测试五：自定义分区（二）

package com.grace.scalawc
import java.io.File

import org.apache.commons.io.FileUtils
import org.apache.hadoop.fs.FileUtil
import org.apache.spark.rdd.RDD
import org.apache.spark.{HashPartitioner, Partitioner, SparkConf, SparkContext}

object FavPerson3Partitioner {
  def main(args: Array[String]): Unit = {
    val sc = new SparkContext(new SparkConf().setAppName("FavPerson3Partitioner").setMaster("local"))

    val data = Array(("aaa", 2), ("aaa", 3), ("aaa", 1), ("aaa", 0), ("aaa", 4),
      ("aa", 2), ("aa", 3), ("aa", 1), ("aa", 0), ("aa", 4),
      ("a", 2), ("a", 3), ("a", 1), ("a", 0), ("a", 4))

    val dataRdd: RDD[(String, Int)] = sc.parallelize(data)

    val res: RDD[(String, Int)] = dataRdd.partitionBy(new MyPartitioner(3))
    val file = new File("D:\\log.log")
    if(file.exists()) {
      FileUtils.deleteDirectory(file)
    }
    res.saveAsTextFile("D:\\log.log")
    sc.stop()
  }
}

class MyPartitioner(num: Int) extends Partitioner {
  override def numPartitions: Int = num

  override def getPartition(key: Any): Int = {
    key.toString.length % num
  }
}


在D:\\log.log文件夹中出现

分区成功！

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Glace.♥
关注
专栏目录
Spark wordCount案例
埋葬青春
06-03 574
1、构建一个RDD ##path指定文件所在的位置，第一个默认的路径是HDFS的路径，而且可以省略hdfs:主机名:8020/,第二个如果是linux文件的路径，那么需要写file:// + 文件的绝对路径 val textFile = sc.textFile("README.md") org.apache.hadoop.mapred.InvalidInputException:  Input p...
Spark中WordCount案例实操（linux上直接运行）下篇
Queen0809的博客
05-07 327
6.打包jar包 6.1 点击Maven --> package 生成jar包 6.2 选择不带有依赖的jar包，因为linux系统中有相关环境，如果没有，则选择带有依赖的jar包 6.3 将jar包上传到linux系统上（我上传到了/opt/module/spark-yarn/WordCount.jar） 7. 编译spark代码进行运行 7.1模板 bin/spark-submit –calss idea中的含有main方法的完整名字 –master 选择运行模式 –deploy-mode
spark运行wordcount例子
10-31
本例子是让新手熟悉了解如果用idea开发程序，提交到spakr集群上读取hdfs数据这么一个流程。博客讲解链接：https://blog.csdn.net/cuicanxingchen123456/article/details/83573838
spark wordcount 实例
bitcarmanlee的博客
07-21 8059
spark集群搭建完毕以后，自然需要来测试一下。大数据领域的第一个程序自然就是wordcount了，就好像我们新接触一门编码语言，第一个程序就是hello world一样。接下来，我们就尝试用各种方式在spark里实现wordcount。1.准备数据首先准备一个简单文件aaa，然后put到hdfs上面：[root@namenodetest01 spark-1.6.0-bin-hadoop2.4]#
Spark实现WordCount经典案例
Kuber
04-17 2278
Spark实现WordCount经典案例 创建数据源 数据源随便弄一下，两份一样即可，数据用空格隔开，换行 仅用spark进行连接，不适用spark提供的两种方法 方法一 package com.kuber.spark.core.wordcount import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} object Spark01_WordCount { def main(args: Ar
Spark WordCount 案例
m0_72168501的博客
01-14 1873
Spark WordCount 案例
Spark中WordCount案例实操（linux上直接运行）上篇
Queen0809的博客
04-23 322
编写程序（linux上运行） 1.新建一个maven工程 1.1 建好后，在项目上右击 --> Add Framework Support --> 勾选scala 1.2 在src/main下新建一个directory （scala）–> 点击scala,右键 --> Mark Directory AS --> Sources Root 2.日志文件配置（设置只打印Error级别的日志） 2.1 在src/main/resources下新建 --> File（名为log4
Spark基础【介绍、入门WordCount案例】
weixin_43923463的博客
08-04 1749
HadoopSparkHadoopSpark由上面的信息可以获知，Spark出现的时间相对较晚，并且主要功能主要是用于数据计算，所以其实Spark一直被认为是Hadoop框架的升级版，但其实不是NameNode是单点的，容易出现单点故障，制约了集群的发展NameNode是单点的，受到了硬件的制约，制约了集群的发展MapReduce运行速度太慢，主要因为设计理念的问题，MR早期就是用于单一数据计算，在当前数据挖掘和数据迭代计算情景中不适用MR框架和Hadoop耦合性非常强，无法分离增加了一个资源调度框架，将计
SparkStreaming之WordCount案例
01-07
一、案例简介 使用 netcat 工具向 9999 端口不断的发送数据，通过 SparkStreaming 读取端口数据并统 计不同单词出现的次数 。 二、netcat操作 1、虚拟机中安装netcat [root@hadoop1 spark]# yum install -y nc 2、...
Spark-通过 API实现WordCount案例的数据集和jar包
07-12
下载资料后，免费获取数据集和jar包
Spark之WordCount案例
qq_44403259的博客
03-12 368
在idea中完成wordcount程序编写、在windows本机和集群环境下分别运行 Windows本机运行 scala程序如下： object SparkWorldCount { def main(args: Array[String]): Unit = { //1、做出sc val conf = new SparkConf() conf.setMaster("local") conf.s...
Spark-WordCount案例
dafsq的博客
03-06 260
Spark-WordCount案例
spark简单入门案例wordCount
weixin_43924642的博客
04-18 1973
创建idea工程 package test.Spark import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} object WordCount { def main(args: Array[String]): Unit = { //alt+上箭头 向上拉取 ...
基于spark开发wordcount案例
changshupx的博客
09-05 461
spark的WordCount 原理: 数据流分析： textFile(“in”)：读取本地文件input文件夹数据； flatMap(.split(" "))：压平操作，按照空格分割符将一行数据映射成一个个单词； map((,1))：对每一个元素操作，将单词映射为元组； reduceByKey(+)：按照key将值进行聚合，相加； collect：将数据收集到Driver端展示。 package day0904 import org.apache.spark.rdd.RDD import org.apa
spark入门实例 wordCount
深海小黄鱼
03-14 757
数据格式： 撒地方 213 萨芬 123 657 jhkjhk 萨芬 123 gjh 123 意图以 2354 萨芬 123 package com.agm.words import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.
Spark-wordcount案例
iilegend的博客
06-14 177
def main(args: Array[String]): Unit = { //创建一个配置信息对象 //设置应用程序名称 val config:SparkConf = new SparkConf().setAppName("scala_worldcount") //设置运行模式 /* * local：使用本地一个线程模拟集群执行任...
Spark案例之WordCount
m0_55834564的博客
06-08 1388
目录IntelliJ IDEA一、编写WordCount程序1.创建一个Maven项目WordCount并导入依赖2.编写代码3.打包插件4.创建数据，打包完，导入包5.集群测试（在包的路径下输入）hdfs的方式：本地方式：6.查看结果 二、远程调用Spark1.启动Spark下的start-all.sh Jps查看进程:2.导入依赖 3.编写代码4.打包5.在把代码加到创建sparkConf的后面原代码修改后，加上包的路径6.运行输出 2.编写代码 3.打包插件 4.创建数据，打包完，
spark-WordCount案例
reddy_Hu的博客
08-05 207
Scala程序 package com.doit.spark.day01 import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} object WordCount { def main(args: Array[String]): Unit = { //创建SparkContext val conf = new SparkConf().setAppName("WordCoun
spark案例——wordcount
fengjian的博客
10-09 593
local本地模式 添加pom依赖 <properties> <scala.version>2.12.0</scala.version> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</arti
spark实现wordcount案例
最新发布
09-19
Spark实现WordCount案例是一种经典的大数据处理案例，主要用于统计文本中每个单词出现的次数。具体实现步骤如下： 1. 读取文本文件：使用Spark的API读取文本文件，将文本文件中的每一行作为一个RDD的元素。 2. ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值