Spark构建推荐引擎之二:基于Spark Streaming 实时推荐计算

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/sunbow0/article/details/43456805

1.1 数据输入模型

1)用户数据输入数据格式:

用户ID,物品ID,点击次数。

2)相似矩阵输入数据格式:

物品ID,物品ID,相似度

1.2 物品相似矩阵

 采用SparkContext读取物品的相似矩阵:

    //2 sc 读取相似矩阵

    valsimi_path1 ="hdfs://192.168.180.100:9000/data/simi/simi.txt"

    valsimi_rdd1 =sc.textFile(simi_path1,10)

    valsimi_rdd2 =simi_rdd1.map(line => {

      valfileds = line.split(",")

      (fileds(0),fileds(1),fileds(2).toDouble)

    })

    simi_rdd2.cache()

1.3 用户实时评分计算

 采用Spark Streaming实时计算用户的评分数据:

//3 构建Streaming对象

    valssc =new StreamingContext(sc,batchDuration)

    ssc.checkpoint("hdfs://192.168.180.100:9000/data/check")

 

    //4 用户实时数据收集

    //4.1 设置监听目录

    valSpath1 =directory

    valStream_User_Act1 =ssc.fileStream[LongWritable, Text, TextInputFormat](Spath1).map(_._2.toString)

    Stream_User_Act1.checkpoint(slideDuration)

    //4.2 用户操作数据——流式窗口处理

    //用户ID,物品ID,点击次数

    valStream_User_Act2 =Stream_User_Act1.map(line => {

      valfileds = line.split(",")

      ((fileds(0),fileds(1)),fileds(2).toInt)

    })

    valStream_User_Act3 =Stream_User_Act2.reduceByKeyAndWindow(_ + _, _ - _,windowDuration,slideDuration)

    valStream_User_Act4 =Stream_User_Act3.reduceByKey(_ + _).map(f => (f._1._1, f._1._2, f._2))

1.4 实时推荐计算

//5 实时推荐计算

    valRecommend_app1 =Stream_User_Act4.transform(rdd => Recommend1(simi_rdd2, rdd,r_number))

def Recommend1(

    items_similar: RDD[(String, String, Double)],

    user_prefer: RDD[(String, String, Int)],

    r_number: Int): (RDD[(String, String, Double)]) = {

    //   1 矩阵计算——i行与jjoin

    valrdd_app1_R2 =items_similar.map(f => (f._1, (f._2, f._3))).

      join(user_prefer.map(f => (f._2, (f._1, f._3))))

    //   2 矩阵计算——i行与j列元素相乘

    valrdd_app1_R3 =rdd_app1_R2.map(f => ((f._2._2._1, f._2._1._1), f._2._2._2 * f._2._1._2))

    //   3 矩阵计算——用户:元素累加求和

    valrdd_app1_R4 =rdd_app1_R3.reduceByKey((x, y) => x + y).map(f => (f._1._1, (f._1._2, f._2)))

    //   4 矩阵计算——用户:用户对结果排序,过滤

    valrdd_app1_R5 =rdd_app1_R4.groupByKey()

    valrdd_app1_R6 =rdd_app1_R5.map(f => {

      vali2 = f._2.toBuffer

      vali2_2 =i2.sortBy(_._2)

      if (i2_2.length > r_number)i2_2.remove(0, (i2_2.length - r_number))

      (f._1, i2_2.toIterable)

    })

    valrdd_app1_R7 =rdd_app1_R6.flatMap(f => {

      valid2 = f._2

      for (w <-id2) yield (f._1,w._1,w._2)

    })

    rdd_app1_R7.map(f => f._1).distinct.count

    rdd_app1_R7

  }

1.5 结果输出

    //6 结果输出   

    valoutpath ="hdfs://192.168.180.79:9000/output/recommend"

    Recommend_app1.saveAsTextFiles(outpath,"rdd")

    //7 监听启动

    ssc.start()

    ssc.awaitTermination()

 

转载请注明出处:

http://blog.csdn.net/sunbow0/article/details/43456805

 

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页