Hadoop与Spark的区别
搜狗文件分析例子
以交互的方式:
val rdd = sc.textFile("/input/SogouQ.reduced")
① ⽤户在00:00:00到12:00:00之间的查询数?
以本地的模式:
package cn.chinahadoop.scala
import org.apache.spark.{SparkContext, SparkConf}
object SogouA {
def main(args: Array[String]) {
if (args.length == 0) {
System.err.println("Usage: SogouA <file1>")
System.exit(1)
}
val conf = new SparkConf().setAppName("SogouA")
val sc = new SparkContext(conf)
val sgRDD=sc.textFile(args(0))
sgRDD.map(_.split('\t')(0)).filter(x => x >= "00:00:00" && x <= "12:00:00").saveAsTextFile(args(1))
sc.stop()
}
}
客户端运行命令:
./spark-submit
–master spark://SparkMaster:7077
–name chinahadoop
–class cn.chinahadoop.scala.SogouA
/home/chinahadoop.jar
hdfs://SparkMaster:9000/data/SogouQ.reduced
hdfs://SparkMaster:9000/data/a
rdd.map(_.split('\t')).map(_(0)).filter(_>"00:00:00").filter(_<"12:00:00").count
② 搜索结果排名第⼀,但是点击次序排在第⼆的数据有多少?
rdd.map(_.split('\t')).map(_(3)).filter(line=>(line=="1 2")).count
package cn.chinahadoop.scala
import org.apache.spark.{SparkContext, SparkConf}
object SogouB {
def main(args: Array[String]) {
if (args.length == 0) {
System.err.println("Usage: SogouB <file1>")
System.exit(1)
}
val conf = new SparkConf().setAppName("SogouB")
val sc = new SparkContext(conf)
val sgRDD=sc.textFile(args(0))
println(sgRDD.map(_.split('\t')).filter(_.length ==5).map(_(3).split(' ')).filter(_(0).toInt ==1).filter(_(1).toInt ==2).count)
sc.stop()
}
}
② 计算最多出的网址
rdd.map(_.split('\t')(5)).map((_,1)).reduceByKey(_+_).map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1)).take(10)
一个session内查询次数最多的用户的session与相应的查询次数?
package cn.chinahadoop.scala
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.SparkContext._
object SogouC {
def main(args: Array[String]) {
if (args.length == 0) {
System.err.println("Usage: SogouC <file1>")
System.exit(1)
}
val conf = new SparkConf().setAppName("SogouC")
val sc = new SparkContext(conf)
val sgRDD=sc.textFile(args(0))
sgRDD.map(_.split('\t')).filter(_.length ==5).map(x=>(x(1),1)).reduceByKey(_+_).map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1)).take(10).foreach(println)
sc.stop()
}
}
用户查询日志(SogouQ)
http://www.sogou.com/labs/resource/q.php