Spark实现ALS最小二乘法

ALS算法流程：

• 初始化数据集和Spark环境
• 切分测试机和检验集
• 训练ALS模型
• 验证结果
• 检验满足结果，直接推荐商品，否则继续训练ALS模型

数据集的含义

数据根式 用户ID，产品ID，评分

1,11,2
1,12,3
1,13,1
1,14,0
1,15,1
2,11,2
2,12,2
2,13,2
2,14,1
2,15,4
3,11,2
3,12,3
3,13,1
3,14,0
3,15,1
4,11,1
4,12,2
4,13,2
4,14,1
4,15,4
5,11,1
5,12,2
5,14,2
5,14,2
5,15,4

源码解析

ALS类的所有字段如下:

@Since("0.8.0")
class ALS private (
    private var numUserBlocks: Int,
    private var numProductBlocks: Int,
    private var rank: Int,
    private var iterations: Int,
    private var lambda: Double,
    private var implicitPrefs: Boolean,  使用显式反馈ALS变量或隐式反馈
    private var alpha: Double,    ALS隐式反馈变化率用于控制每次拟合修正的幅度
    private var seed: Long = System.nanoTime()
  ) extends Serializable with Logging {

ALS.train方法:

/**
  * Train a matrix factorization model given an RDD of ratings given by users to some products,
  * in the form of (userID, productID, rating) pairs. We approximate the ratings matrix as the
 * product of two lower-rank matrices of a given rank (number of features). To solve for these
  * features, we run a given number of iterations of ALS. This is done using a level of
 * parallelism given by `blocks`.
  *
  * @param ratings    RDD of (userID, productID, rating) pairs
 * @param rank       number of features to use  
 * @param iterations number of iterations of ALS (recommended: 10-20)
 * @param lambda     regularization factor (recommended: 0.01)
* @param blocks     level of parallelism to split computation into  将并行度分解为等级
 * @param seed       random seed  随机种子
 */
 @Since("0.9.1")
 def train(
      ratings: RDD[Rating], //RDD序列由用户ID 产品ID和评分组成
      rank: Int,    //模型中的隐藏因子数目
      iterations: Int,  //算法迭代次数
      lambda: Double,  //ALS正则化参数
      blocks: Int,   //块
      seed: Long ): MatrixFactorizationModel = {
        new ALS(blocks, blocks, rank, iterations, lambda, false, 1.0, seed).run(ratings)
  }

基于ALS算法的协同过滤推荐实例代码:

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.mllib.recommendation.Rating
import org.apache.spark.mllib.recommendation.ALS

/**
 * ALS最小二乘法
 */
object CollaborativeFilter {
   /**
    * 程序入口
    */
  def main(args: Array[String]): Unit = {
    val conf=new SparkConf().setAppName("ALS").setMaster("local[*]")
    val context=new SparkContext(conf)
    //第一列位用户编号，第二列位产品编号，第三列的评分Rating为Double类型
    val data=context.textFile("D:\\opt\\data.txt")
    //处理数据
    val ratings=data.map(_.split(",") match{
       //数据集的转换
       case Array(user,item,rate) =>
         //将数据集转化为专用的Rating
         Rating(user.toInt,item.toInt,rate.toDouble)
     })
    //设置隐藏因子
     val rank=4
    //设置迭代次数
    val numIterations=3
    //进行模型训练
    val model =ALS.train(ratings,rank,numIterations,0.01)
   //为用户4推荐一个商品
   val rs=model.recommendProducts(4,1)
   //打印结果
   rs.foreach(println)
   //预测的评分结果
   val result:Double=rs(0).rating   
   val realilty=data.map(_.split(",") match {
      case Array(user,item,rate) =>
        Rating(user.toInt,item.toInt,rate.toDouble)
    }).map(num=>{
      if(num.user==4&&num.product==15)
        num.rating                                            //返回实际评分结果
      else
        0
    }).foreach(num=>{
      if(num!=0)
       println("预测的准确率为："+(1-(math.abs(result-num)/1)))
    })
   context.stop()
  }
}

运行推荐结果: