101.Spark实现聚合功能(面试题)

最新推荐文章于 2020-11-19 23:42:05 发布

weixin_33941350

最新推荐文章于 2020-11-19 23:42:05 发布

阅读量97

点赞数

文章标签：数据库 scala java

原文链接：https://my.oschina.net/repine/blog/668045

版权

为什么80%的码农都做不了架构师？>>>

互联网公司-面试题：
/**  
举个例子,比如要统计用户的总访问次数和去除访问同一个URL之后的总访问次数,随便造了几条样例数据(四个字段:id,name,vtm,url,vtm字段本例没用,不用管)如下:

id1,user1,2,http://www.hupu.com
id1,user1,2,http://www.hupu.com
id1,user1,3,http://www.hupu.com
id1,user1,100,http://www.hupu.com
id2,user2,2,http://www.hupu.com
id2,user2,1,http://www.hupu.com
id2,user2,50,http://www.hupu.com
id2,user2,2,http://touzhu.hupu.com

根据这个数据集,我们可以写hql 实现:
select id,name, count(0) as ct,count(distinct url) as urlcount 
from table 
group by id,name;

得出结果应该是:

id1,user1,4,1
id2,user2,4,2

下面用Spark实现这个聚合功能<发现Spark还是有难度的，卧槽>
简单说说MR的解析过程：

map阶段： id和name组合为key， url为value
reduce阶段： len(urls) 出现次数, len(set(urls)) 出现用户数
由于本人是不写MR，导致面试很尴尬。
想装逼写个Spark, 发现难度很大，因为的确很多函数不熟悉。

代码如下：

import org.apache.spark.SparkContext._  
import org.apache.spark._   

object SparkDemo2 {  
  def main(args: Array[String]) {  
  
    case class User(id: String, name: String, vtm: String, url: String)  
    //val rowkey = (new RowKey).evaluate(_)  
    // val HADOOP_USER = "hdfs"  
    // 设置访问spark使用的用户名  
    // System.setProperty("user.name", HADOOP_USER);  
    // 设置访问hadoop使用的用户名  
    // System.setProperty("HADOOP_USER_NAME", HADOOP_USER);  
  
    val conf = new SparkConf().setAppName("wordcount").setMaster("local") //.setExecutorEnv("HADOOP_USER_NAME", HADOOP_USER)  
    val sc = new SparkContext(conf)  
    val data = sc.textFile("/Users/jiangzl/Desktop/test.txt")  
    val rdd1 = data.map(line => {  
      val r = line.split(",")  
      User(r(0), r(1), r(2), r(3))  
    })
    val rdd2 = rdd1.map(r => ((r.id, r.name), r))  
  
    val seqOp = (a: (Int, List[String]), b: User) => a match {  
      case (0, List()) => (1, List(b.url))  
      case _ => (a._1 + 1, b.url :: a._2)  
    }  
  
    val combOp = (a: (Int, List[String]), b: (Int, List[String])) => {  
      (a._1 + b._1, a._2 ::: b._2)  
    }
  
    println("-----------------------------------------")  
    val rdd3 = rdd2.aggregateByKey((0, List[String]()))(seqOp, combOp).map(a => {  
      (a._1, a._2._1, a._2._2.distinct.length)  
    })  
    rdd3.collect.foreach(println)  
    println("-----------------------------------------")  
    sc.stop()  
  }  
}

解决方案－报错Scala版本问题：Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.VolatileObjectRef.zero()Lscala/runtime/VolatileObjectRef;

修改Scala版本2.11.7改为2.10.4

simple.sbt

name := "SparkDemo Project"
version := "1.0"
scalaVersion := "2.11.7"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"

———————————————————————————修改为：——————————————————————————

name := "SparkDemo Project"
version := "1.0"
scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"

运行过程

jiangzhongliandeMacBook-Pro:spark-1.4.1-bin-hadoop2.6 jiangzl$ ./bin/spark-submit --class "SparkDemo2" ~/Desktop/tmp/target/scala-2.11/simple-project_2.11-1.0.jar
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.VolatileObjectRef.zero()Lscala/runtime/VolatileObjectRef;
    at SparkDemo2$.main(tmp_spark.scala)
    at SparkDemo2.main(tmp_spark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

———————————————————————————修改为：——————————————————————————

jiangzhongliandeMacBook-Pro:spark-1.4.1-bin-hadoop2.6 jiangzl$ ./bin/spark-submit --class "SparkDemo2" ~/Desktop/tmp/target/scala-2.10/sparkdemo-project_2.10-1.0.jar
16/04/29 12:40:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-----------------------------------------
((id1,user1),4,1)
((id2,user2),4,2)
-----------------------------------------

转载于:https://my.oschina.net/repine/blog/668045