spark ml 推荐源码笔记二

上次我们讲到als对象train方法的

val solver = if (nonnegative) new NNLSSolver else new CholeskySolver

接下来是

    val blockRatings = partitionRatings(ratings, userPart, itemPart)//有个方法partitionRatings，传入数据源，user的HashPartitioner，item的HashPartitioner,并缓存
      .persist(intermediateRDDStorageLevel)

我们来看下partitionRatings

   /**
   * Partitions raw ratings into blocks.//把原始ratings分块
   *
   * @param ratings raw ratings//原始ratings
   * @param srcPart partitioner for src IDs//源HashPartitioner
   * @param dstPart partitioner for dst IDs//目的HashPartitioner
   *
   * @return an RDD of rating blocks in the form of ((srcBlockId, dstBlockId), ratingBlock)//以((srcBlockId, dstBlockId), ratingBlock)形式返回rating
   */
  private def partitionRatings[ID: ClassTag](
      ratings: RDD[Rating[ID]],
      srcPart: Partitioner,
      dstPart: Partitioner): RDD[((Int, Int), RatingBlock[ID])] = {


     /* The implementation produces the same result as the following but generates less objects.//下面的实现结果一样但是产生更少的对象


     ratings.map { r =>
       ((srcPart.getPartition(r.user), dstPart.getPartition(r.item)), r)//ratings的每条记录转成(user的HashPartitioner索引，item的HashPartitioner索引，rating)的形式
     }.aggregateByKey(new RatingBlockBuilder)(//然后用aggregateByKey聚合，(srcPart.getPartition(r.user), dstPart.getPartition(r.item))是k,，r是value，RatingBlockBuilder是初始值，我们先进入RatingBlockBuilder

  /**
   * Builder for [[RatingBlock]]. [[mutable.ArrayBuilder]] is used to avoid boxing/unboxing.//构建[RatingBlock]，使用mutable.ArrayBuilder避免装箱/拆箱
   */
  private[recommendation] class RatingBlockBuilder[@specialized(Int, Long) ID: ClassTag]//@specialized(Int, Long) ID: ClassTag代表运行时确定推断类型
    extends Serializable {


    private val srcIds = mutable.ArrayBuilder.make[ID]//创建一个空的可变数组srcIds ，这种创建方法适合只声明元素类型的方式
    private val dstIds = mutable.ArrayBuilder.make[ID]//创建一个空的可变数组dstIds 
    private val ratings = mutable.ArrayBuilder.make[Float]//创建一个空的可变数组ratings
    var size = 0


    /** Adds a rating. */
    def add(r: Rating[ID]): this.type = {//加一个Rating[ID]，数组长度size自增，srcIds 可变数组加元素r.user，dstIds 可变数组加元素r.item，ratings 可变数组加元素 r.rating，返回当前对象，这样当前对象就有了加的这个对象的信息
      size += 1
      srcIds += r.user
      dstIds += r.item
      ratings += r.rating
      this
    }


    /** Merges another [[RatingBlockBuilder]]. */
    def merge(other: RatingBlock[ID]): this.type = {//合并另一个RatingBlockBuilder，数组长度增加，两个srcIds数组合并，两个dstIds 数组合并，两个ratings 数组合并，返回当前对象
      size += other.srcIds.length
      srcIds ++= other.srcIds
      dstIds ++= other.dstIds
      ratings ++= other.ratings
      this
    }


    /** Builds a [[RatingBlock]]. */
    def build(): RatingBlock[ID] = {//构建RatingBlock[ID] 对象，传入参数，.result()方法把不可变数组变为可变数组
      RatingBlock[ID](srcIds.result(), dstIds.result(), ratings.result())
    }
  }

回到aggregateByKey，下面就是

}.aggregateByKey(new RatingBlockBuilder)(//(srcPart.getPartition(r.user), dstPart.getPartition(r.item))是k,，r是value，RatingBlockBuilder是初始值

         seqOp = (b, r) => b.add(r),//这里面b是RatingBlockBuilder，r是Rating[ID]，把r的user，item，rating加到RatingBlockBuilder的数组中，这里每个value都会产生一个RatingBlockBuilder，印证了注释，所以没有采用这种方法
         combOp = (b0, b1) => b0.merge(b1.build()))//相同key的RatingBlockBuilder合并，.build()是RatingBlockBuilder的方法
       .mapValues(_.build())aggregateByKey的最终结果也是k-value类型，通过mapValues取value,通过_.build()转成RatingBlock[ID] ，RatingBlock成员分别是srcIds，dstIds，ratings数组
     */
我们看看更优的方法

    val numPartitions = srcPart.numPartitions * dstPart.numPartitions//分块数乘积
    ratings.mapPartitions { iter =>//分块计算
      val builders = Array.fill(numPartitions)(new RatingBlockBuilder[ID])每个分块生成一个RatingBlockBuilder数组，长度是分块乘积数
      iter.flatMap { r =>//拿出每个rating，由于是flatMap，最终结果是一个数组
        val srcBlockId = srcPart.getPartition(r.user)//确定rating中user的srcBlockId 
        val dstBlockId = dstPart.getPartition(r.item)//确定rating中item的dstBlockId 
        val idx = srcBlockId + srcPart.numPartitions * dstBlockId//组成便宜索引，看来还是数组好使
        val builder = builders(idx)//取出对应的RatingBlockBuilder
        builder.add(r)//和并单个Rating[ID]
        if (builder.size >= 2048) { // 2048 * (3 * 4) = 24k//如果RatingBlockBuilder数组的size超过2048，2048*3(3个数组)*4(数组每个元素4个字节)=24k
          builders(idx) = new RatingBlockBuilder//超过就把数组对应位置的RatingBlockBuilder弄成新的，又有新的空间，旧的弄成RatingBlock[ID]返回成为结果数组的一个元素，如果没超过相当于无返回结果，这个感觉好牛逼，一下控制了RatingBlockBuilder对象的数量
          Iterator.single(((srcBlockId, dstBlockId), builder.build()))
        } else {
          Iterator.empty
        }
      } ++ {//这里++的原因是flatMap 里只包含超过2048的builder，最终不满2048的还没有处理，这里就在处理不满2048的builder
        builders.view.zipWithIndex.filter(_._1.size > 0).map { case (block, idx) =>//取出size>0的builder
          val srcBlockId = idx % srcPart.numPartitions//获取要处理的builder对应的srcBlockId ，dstBlockId 
          val dstBlockId = idx / srcPart.numPartitions
          ((srcBlockId, dstBlockId), block.build())//和flatMap 元素同样的数据结构，最终合并，我再加个牛逼，以前确实没见过这种写法
        }
      }
    }.groupByKey().mapValues { blocks =>//聚合只操作builder
      val builder = new RatingBlockBuilder[ID]//由于blocks是iterator，所以聚合成一个builder，最终的结果就是RatingBlock[ID](srcIds.result(), dstIds.result(), ratings.result())构成的rdd，这里可以感受到一个套路&