Spark学习笔记－推荐系统（协同过滤算法为用户推荐播放歌手）

最新推荐文章于 2023-07-14 14:46:32 发布
最新推荐文章于 2023-07-14 14:46:32 发布
阅读量4.3k 收藏 4
点赞数 1
分类专栏： spark学习笔记
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/richard_more/article/details/52514130
版权

这是Spark高级数据分析的第二个项目，基于用户，歌手，播放次数的简单数据记录，来为用户推荐歌手。


（1）获取数据

miaofu@miaofu-Virtual-Machine:~/user_artist_data$ wget http://www.iro.umontreal.ca/~lisa/datasets/profiledata_06-May-2005.tar.gz
--2016-09-12 14:14:10--  http://www.iro.umontreal.ca/~lisa/datasets/profiledata_06-May-2005.tar.gz
正在解析主机 www.iro.umontreal.ca (www.iro.umontreal.ca)... 132.204.26.36
正在连接 www.iro.umontreal.ca (www.iro.umontreal.ca)|132.204.26.36|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 135880312 (130M) [application/x-gzip]
正在保存至: “profiledata_06-May-2005.tar.gz”

profiledata_06-May-2005.tar.gz                100%[================================================================================================>] 129.58M  1.40MB/s   用时 90s  

2016-09-12 14:15:44 (1.44 MB/s) - 已保存 “profiledata_06-May-2005.tar.gz” [135880312/135880312])

miaofu@miaofu-Virtual-Machine:~/user_artist_data$ ls
profiledata_06-May-2005.tar.gz
miaofu@miaofu-Virtual-Machine:~/user_artist_data$ tar -zxvf profiledata_06-May-2005.tar.gz 
profiledata_06-May-2005/
profiledata_06-May-2005/artist_data.txt
profiledata_06-May-2005/README.txt
profiledata_06-May-2005/user_artist_data.txt
profiledata_06-May-2005/artist_alias.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data$ ls
profiledata_06-May-2005  profiledata_06-May-2005.tar.gz
miaofu@miaofu-Virtual-Machine:~/user_artist_data$ cd profiledata_06-May-2005/
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ ls
artist_alias.txt  artist_data.txt  README.txt  user_artist_data.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ ls -l -h
总用量 464M
-rw-r--r-- 1 miaofu miaofu 2.8M  5月  6  2005 artist_alias.txt
-rw-r--r-- 1 miaofu miaofu  54M  5月  6  2005 artist_data.txt
-rw-r--r-- 1 miaofu miaofu 1.3K  5月 11  2005 README.txt
-rw-r--r-- 1 miaofu miaofu 407M  5月  5  2005 user_artist_data.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ vi user_artist_data.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ vi artist_alias.txt 
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ vi artist_data.txt 
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ vi README.txt 
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ pwd
/home/miaofu/user_artist_data/profiledata_06-May-2005
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -put /home/miaofu/user_artist_data/profiledata_06-May-2005/
artist_alias.txt      artist_data.txt       README.txt            user_artist_data.txt  
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -put /home/miaofu/user_artist_data/profiledata_06-May-2005/
artist_alias.txt      artist_data.txt       README.txt            user_artist_data.txt  
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -put /home/miaofu/user_artist_data/profiledata_06-May-2005/user_artist_data.txt 
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs ls /user/miaofu/
ls: Unknown command
Did you mean -ls?  This command begins with a dash.
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -ls /user/miaofu/
Found 4 items
drwxr-xr-x   - miaofu supergroup          0 2016-07-25 21:03 /user/miaofu/follower
drwxr-xr-x   - miaofu supergroup          0 2016-07-25 20:14 /user/miaofu/grep-temp-31205318
drwxr-xr-x   - miaofu supergroup          0 2016-08-27 22:02 /user/miaofu/linkage
-rw-r--r--   2 miaofu supergroup  426761761 2016-09-12 14:22 /user/miaofu/user_artist_data.txt

再把其它的两个数据放入到hdfs中，

miaofu@miaofu-Virtual-Machine:~$ cd user_artist_data/profiledata_06-May-2005/
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ ls
artist_alias.txt  artist_data.txt  README.txt  user_artist_data.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ pwd
/home/miaofu/user_artist_data/profiledata_06-May-2005
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -put artist_alias.txt artist_data.txt 
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -ls .
Found 5 items
-rw-r--r--   2 miaofu supergroup    2932731 2016-09-12 16:18 artist_data.txt
drwxr-xr-x   - miaofu supergroup          0 2016-07-25 21:03 follower
drwxr-xr-x   - miaofu supergroup          0 2016-07-25 20:14 grep-temp-31205318
drwxr-xr-x   - miaofu supergroup          0 2016-08-27 22:02 linkage
-rw-r--r--   2 miaofu supergroup  426761761 2016-09-12 14:22 user_artist_data.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -put artist_alias.txt 
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$ hadoop fs -ls .
Found 6 items
-rw-r--r--   2 miaofu supergroup    2932731 2016-09-12 16:19 artist_alias.txt
-rw-r--r--   2 miaofu supergroup    2932731 2016-09-12 16:18 artist_data.txt
drwxr-xr-x   - miaofu supergroup          0 2016-07-25 21:03 follower
drwxr-xr-x   - miaofu supergroup          0 2016-07-25 20:14 grep-temp-31205318
drwxr-xr-x   - miaofu supergroup          0 2016-08-27 22:02 linkage
-rw-r--r--   2 miaofu supergroup  426761761 2016-09-12 14:22 user_artist_data.txt
miaofu@miaofu-Virtual-Machine:~/user_artist_data/profiledata_06-May-2005$


（2）看一看代码纵览

Last login: Mon Sep 12 13:38:36 on ttys006
miaofudeMacBook-Pro:aas miaofu$ ls
LICENSE			ch03-recommender	ch06-lsa		ch09-risk		common
README.md		ch04-rdf		ch07-graph		ch10-genomics		pom.xml
ch02-intro		ch05-kmeans		ch08-geotime		ch11-neuro		simplesparkproject
miaofudeMacBook-Pro:aas miaofu$ 
miaofudeMacBook-Pro:aas miaofu$ vi ch03-recommender/src/main/scala/com/cloudera/datascience/recommender/RunRecommender.scala 
miaofudeMacBook-Pro:aas miaofu$ cat ch
ch02-intro/       ch03-recommender/ ch04-rdf/         ch05-kmeans/      ch06-lsa/         ch07-graph/       ch08-geotime/     ch09-risk/        ch10-genomics/    ch11-neuro/
miaofudeMacBook-Pro:aas miaofu$ cat ch03-recommender/src/main/scala/com/cloudera/datascience/recommender/RunRecommender.scala 
/*
 * Copyright 2015 and onwards Sanford Ryza, Juliet Hougland, Uri Laserson, Sean Owen and Joshua Wills
 *
 * See LICENSE file for further information.
 */

package com.cloudera.datascience.recommender

import scala.collection.Map
import scala.collection.mutable.ArrayBuffer
import scala.util.Random
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.ml.recommendation.{ALS, ALSModel}
import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
import org.apache.spark.sql.functions._

object RunRecommender {

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().getOrCreate()

    val base = "hdfs:///user/ds/"
    val rawUserArtistData = spark.read.textFile(base + "user_artist_data.txt")
    val rawArtistData = spark.read.textFile(base + "artist_data.txt")
    val rawArtistAlias = spark.read.textFile(base + "artist_alias.txt")

    val runRecommender = new RunRecommender(spark)
    runRecommender.preparation(rawUserArtistData, rawArtistData, rawArtistAlias)
    runRecommender.model(rawUserArtistData, rawArtistData, rawArtistAlias)
    runRecommender.evaluate(rawUserArtistData, rawArtistAlias)
    runRecommender.recommend(rawUserArtistData, rawArtistData, rawArtistAlias)
  }

}

class RunRecommender(private val spark: SparkSession) {

  import spark.implicits._

  def preparation(
      rawUserArtistData: Dataset[String],
      rawArtistData: Dataset[String],
      rawArtistAlias: Dataset[String]): Unit = {

    val userArtistDF = rawUserArtistData.map { line =>
      val Array(user, artist, _*) = line.split(' ')
      (user.toInt, artist.toInt)
    }.toDF("user", "artist")

    userArtistDF.agg(min("user"), max("user"), min("artist"), max("artist")).show()

    val artistByID = buildArtistByID(rawArtistData)
    val artistAlias = buildArtistAlias(rawArtistAlias)

    val (badID, goodID) = artistAlias.head
    artistByID.filter($"id" isin (badID, goodID)).show()
  }

  def model(
      rawUserArtistData: Dataset[String],
      rawArtistData: Dataset[String],
      rawArtistAlias: Dataset[String]): Unit = {

    val bArtistAlias = spark.sparkContext.broadcast(buildArtistAlias(rawArtistAlias))

    val trainData = buildCounts(rawUserArtistData, bArtistAlias).cache()

    val model = new ALS().
      setImplicitPrefs(true).
      setRank(10).
      setRegParam(0.01).
      setAlpha(1.0).
      setMaxIter(5).
      setUserCol("user").
      setItemCol("artist").
      setRatingCol("count").
      setPredictionCol("prediction").
      fit(trainData)

    trainData.unpersist()

    model.userFactors.select("features").show(truncate = false)

    val userID = 2093760

    val existingArtistIDs = trainData.
      filter($"user" === userID).
      select("artist").as[Int].collect()

    val artistByID = buildArtistByID(rawArtistData)

    artistByID.filter($"id" isin (existingArtistIDs:_*)).show()

    val topRecommendations = makeRecommendations(model, userID, 5)
    topRecommendations.show()

    val recommendedArtistIDs = topRecommendations.select("artist").as[Int].collect()

    artistByID.filter($"id" isin (recommendedArtistIDs:_*)).show()

    model.userFactors.unpersist()
    model.itemFactors.unpersist()
  }

  def evaluate(
      rawUserArtistData: Dataset[String],
      rawArtistAlias: Dataset[String]): Unit = {

    val bArtistAlias = spark.sparkContext.broadcast(buildArtistAlias(rawArtistAlias))

    val allData = buildCounts(rawUserArtistData, bArtistAlias)
    val Array(trainData, cvData) = allData.randomSplit(Array(0.9, 0.1))
    trainData.cache()
    cvData.cache()

    val allArtistIDs = allData.select("artist").as[Int].distinct().collect()
    val bAllArtistIDs = spark.sparkContext.broadcast(allArtistIDs)

    val mostListenedAUC = areaUnderCurve(cvData, bAllArtistIDs, predictMostListened(trainData))
    println(mostListenedAUC)

    val evaluations =
      for (rank     <- Seq(5,  30);
           regParam <- Seq(1.0, 0.0001);
           alpha    <- Seq(1.0, 40.0))
      yield {
        val model = new ALS().
          setImplicitPrefs(true).
          setRank(rank).setRegParam(regParam).
          setAlpha(alpha).setMaxIter(20).
          setUserCol("user").setItemCol("artist").
          setRatingCol("count").setPredictionCol("prediction").
          fit(trainData)

        val auc = areaUnderCurve(cvData, bAllArtistIDs, model.transform)

        model.userFactors.unpersist()
        model.itemFactors.unpersist()

        (auc, (rank, regParam, alpha))
      }

    evaluations.sorted.reverse.foreach(println)

    trainData.unpersist()
    cvData.unpersist()
  }

  def recommend(
      rawUserArtistData: Dataset[String],
      rawArtistData: Dataset[String],
      rawArtistAlias: Dataset[String]): Unit = {

    val bArtistAlias = spark.sparkContext.broadcast(buildArtistAlias(rawArtistAlias))
    val allData = buildCounts(rawUserArtistData, bArtistAlias).cache()
    val model = new ALS().
      setImplicitPrefs(true).
      setRank(10).setRegParam(1.0).setAlpha(40.0).setMaxIter(20).
      setUserCol("user").setItemCol("artist").
      setRatingCol("count").setPredictionCol("prediction").
      fit(allData)
    allData.unpersist()

    val userID = 2093760
    val topRecommendations = makeRecommendations(model, userID, 5)

    val recommendedArtistIDs = topRecommendations.select("artist").as[Int].collect()
    val artistByID = buildArtistByID(rawArtistData)
    artistByID.join(spark.createDataset(recommendedArtistIDs).toDF("id"), "id").
      select("name").show()

    model.userFactors.unpersist()
    model.itemFactors.unpersist()
  }

  def buildArtistByID(rawArtistData: Dataset[String]): DataFrame = {
    rawArtistData.flatMap { line =>
      val (id, name) = line.span(_ != '\t')
      if (name.isEmpty) {
        None
      } else {
        try {
          Some((id.toInt, name.trim))
        } catch {
          case _: NumberFormatException => None
        }
      }
    }.toDF("id", "name")
  }

  def buildArtistAlias(rawArtistAlias: Dataset[String]): Map[Int,Int] = {
    rawArtistAlias.flatMap { line =>
      val Array(artist, alias) = line.split('\t')
      if (artist.isEmpty) {
        None
      } else {
        Some((artist.toInt, alias.toInt))
      }
    }.collect().toMap
  }

  def buildCounts(
      rawUserArtistData: Dataset[String],
      bArtistAlias: Broadcast[Map[Int,Int]]): DataFrame = {
    rawUserArtistData.map { line =>
      val Array(userID, artistID, count) = line.split(' ').map(_.toInt)
      val finalArtistID = bArtistAlias.value.getOrElse(artistID, artistID)
      (userID, finalArtistID, count)
    }.toDF("user", "artist", "count")
  }

  def makeRecommendations(model: ALSModel, userID: Int, howMany: Int): DataFrame = {
    val toRecommend = model.itemFactors.
      select($"id".as("artist")).
      withColumn("user", lit(userID))
    model.transform(toRecommend).
      select("artist", "prediction").
      orderBy($"prediction".desc).
      limit(howMany)
  }

  def areaUnderCurve(
      positiveData: DataFrame,
      bAllArtistIDs: Broadcast[Array[Int]],
      predictFunction: (DataFrame => DataFrame)): Double = {

    // What this actually computes is AUC, per user. The result is actually something
    // that might be called "mean AUC".

    // Take held-out data as the "positive".
    // Make predictions for each of them, including a numeric score
    val positivePredictions = predictFunction(positiveData.select("user", "artist")).
      withColumnRenamed("prediction", "positivePrediction")

    // BinaryClassificationMetrics.areaUnderROC is not used here since there are really lots of
    // small AUC problems, and it would be inefficient, when a direct computation is available.

    // Create a set of "negative" products for each user. These are randomly chosen
    // from among all of the other artists, excluding those that are "positive" for the user.
    val negativeData = positiveData.select("user", "artist").as[(Int,Int)].
      groupByKey { case (user, _) => user }.
      flatMapGroups { case (userID, userIDAndPosArtistIDs) =>
        val random = new Random()
        val posItemIDSet = userIDAndPosArtistIDs.map { case (_, artist) => artist }.toSet
        val negative = new ArrayBuffer[Int]()
        val allArtistIDs = bAllArtistIDs.value
        var i = 0
        // Make at most one pass over all artists to avoid an infinite loop.
        // Also stop when number of negative equals positive set size
        while (i < allArtistIDs.length && negative.size < posItemIDSet.size) {
          val artistID = allArtistIDs(random.nextInt(allArtistIDs.length))
          // Only add new distinct IDs
          if (!posItemIDSet.contains(artistID)) {
            negative += artistID
          }
          i += 1
        }
        // Return the set with user ID added back
        negative.map(artistID => (userID, artistID))
      }.toDF("user", "artist")

    // Make predictions on the rest:
    val negativePredictions = predictFunction(negativeData).
      withColumnRenamed("prediction", "negativePrediction")

    // Join positive predictions to negative predictions by user, only.
    // This will result in a row for every possible pairing of positive and negative
    // predictions within each user.
    val joinedPredictions = positivePredictions.join(negativePredictions, "user").
      select("user", "positivePrediction", "negativePrediction").cache()

    // Count the number of pairs per user
    val allCounts = joinedPredictions.
      groupBy("user").agg(count(lit("1")).as("total")).
      select("user", "total")
    // Count the number of correctly ordered pairs per user
    val correctCounts = joinedPredictions.
      filter($"positivePrediction" > $"negativePrediction").
      groupBy("user").agg(count("user").as("correct")).
      select("user", "correct")

    // Combine these, compute their ratio, and average over all users
    val meanAUC = allCounts.join(correctCounts, "user").
      select($"user", ($"correct" / $"total").as("auc")).
      agg(mean("auc")).
      as[Double].first()

    joinedPredictions.unpersist()

    meanAUC
  }

  def predictMostListened(train: DataFrame)(allData: DataFrame): DataFrame = {
    val listenCounts = train.groupBy("artist").
      agg(sum("count").as("prediction")).
      select("artist", "prediction")
    allData.
      join(listenCounts, Seq("artist"), "left_outer").
      select("user", "artist", "prediction")
  }

}miaofudeMacBook-Pro:aas miaofu$


(3) spark

登陆spark

miaofu@miaofu-Virtual-Machine:~$ spark-shell
16/09/14 10:19:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.2
      /_/

Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.7.0_95)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/09/14 10:19:14 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/14 10:19:14 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/14 10:19:22 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/09/14 10:19:22 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/09/14 10:19:26 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/14 10:19:26 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
SQL context available as sqlContext.

导入数据 

scala>     val base = "hdfs:///user/ds/"
base: String = hdfs:///user/ds/

scala>     val base = "hdfs:///user/miaofu/"
base: String = hdfs:///user/miaofu/

scala>     val rawArtistData = spark.read.textFile(base + "artist_data.txt")
<console>:27: error: object read is not a member of package spark
             val rawArtistData = spark.read.textFile(base + "artist_data.txt")
                                       ^

scala>     val rawArtistData = sc.textFile(base + "artist_data.txt")
rawArtistData: org.apache.spark.rdd.RDD[String] = hdfs:///user/miaofu/artist_data.txt MapPartitionsRDD[1] at textFile at <console>:29

scala> val head = rawArtistData.take(10)
head: Array[String] = Array(1134999	06Crazy Life, 6821360	Pang Nakarin, 10113088	Terfel, Bartoli- Mozart: Don, 10151459	The Flaming Sidebur, 6826647	Bodenstandig 3000, 10186265	Jota Quest e Ivete Sangalo, 6828986	Toto_XX (1977, 10236364	U.S Bombs -, 1135000	artist formaly know as Mat, 10299728	Kassierer - Musik für beide Ohren)

scala> head.foreach(println)
1134999	06Crazy Life
6821360	Pang Nakarin
10113088	Terfel, Bartoli- Mozart: Don
10151459	The Flaming Sidebur
6826647	Bodenstandig 3000
10186265	Jota Quest e Ivete Sangalo
6828986	Toto_XX (1977
10236364	U.S Bombs -
1135000	artist formaly know as Mat
10299728	Kassierer - Musik für beide Ohren

清洗数据 

scala> def f1(line:String)={
     | val g = line.split("\t")
     | (g(0).toInt,g(1).trim)
     | }
f1: (line: String)(Int, String)

scala> head.foreach(println)
1134999	06Crazy Life
6821360	Pang Nakarin
10113088	Terfel, Bartoli- Mozart: Don
10151459	The Flaming Sidebur
6826647	Bodenstandig 3000
10186265	Jota Quest e Ivete Sangalo
6828986	Toto_XX (1977
10236364	U.S Bombs -
1135000	artist formaly know as Mat
10299728	Kassierer - Musik für beide Ohren
scala> head.map(f1).foreach(println)
(1134999,06Crazy Life)
(6821360,Pang Nakarin)
(10113088,Terfel, Bartoli- Mozart: Don)
(10151459,The Flaming Sidebur)
(6826647,Bodenstandig 3000)
(10186265,Jota Quest e Ivete Sangalo)
(6828986,Toto_XX (1977)
(10236364,U.S Bombs -)
(1135000,artist formaly know as Mat)
(10299728,Kassierer - Musik für beide Ohren)

可以看出，这样清洗是有问题的，比如第三行就不是按照制表符分割的，因此这样直接做是不合理的。为此我们应当引入flatMap，option等概念

option是 scala 中collection的一个数据类型，有some，none两个子类。

scala> val myMap: Map[String, String] = Map("key1" -> "value")
myMap: Map[String,String] = Map(key1 -> value)

scala> val value1: Option[String] = myMap.get("key1")
value1: Option[String] = Some(value)

scala> val value2: Option[String] = myMap.get("key2")
value2: Option[String] = None


为什么需要使用flatMap？

然而,map() 函数要求对每个输入必须严格返回一个值,因此这里不能用这个函数。另一种可行的方法是用 filter() 方法删除那些无法解析的行,但这会重复解析逻辑。当需要将每个元素映射为零个、一个或更多结果时,我们应该使用 flatMap() 函数,因为它将每个输入对应的零个或多个结果组成的集合简单展开,然后放入到一个更大的 RDD 中。它可以和 Scala 集合一起使用,也可以和 Scala 的 Option 类一起使用。Option 代表一个值可以不存在,有点儿像只有 1 或 0 的一个简单集合,1 对应子类 Some,0 对应子类 None。因此在以下代码中,虽然 flatMap 中的函数本可以简单返回一个空 List,或一个只有一个元素 的 List,但使用 Some 和 None 更合理,这种方法简单明了。 

为此我们把上看的解析函数f1重新写成下面的形式。

scala> val artistByID = rawArtistData.flatMap { line =>
     |       val (id, name) = line.span(_ != '\t')
     |       if (name.isEmpty) {
     |         None
     |       } else {
     |         try {
     |           Some((id.toInt, name.trim))
     |         } catch {
     |           case _: NumberFormatException => None
     |         }
     |       }
     |     }
artistByID: org.apache.spark.rdd.RDD[(Int, String)] = MapPartitionsRDD[2] at flatMap at <console>:31

scala> artistByID
res9: org.apache.spark.rdd.RDD[(Int, String)] = MapPartitionsRDD[2] at flatMap at <console>:31

scala> val rawArtistAlias  = sc.textFile(base+"artist_alias.txt")
rawArtistAlias: org.apache.spark.rdd.RDD[String] = hdfs:///user/miaofu/artist_alias.txt MapPartitionsRDD[4] at textFile at <console>:29

scala> val artistAlias = rawArtistAlias.flatMap { line =>
     |       val Array(artist, alias) = line.split('\t')
     |       if (artist.isEmpty) {
     |         None
     |       } else {
     |         Some((artist.toInt, alias.toInt))
     |       }
     |     }.collect().toMap
artistAlias: scala.collection.immutable.Map[Int,Int] = Map(1208690 -> 1003926, 2012757 -> 4569, 6949139 -> 1085752, 1109727 -> 1239120, 6772751 -> 1244705, 2070533 -> 1021544, 1157679 -> 2194, 9969617 -> 5630, 2034496 -> 1116214, 6764342 -> 40, 1272489 -> 1278238, 2108744 -> 1009267, 10349857 -> 1000052, 2145319 -> 1020463, 2126338 -> 2717, 10165456 -> 1001169, 6779368 -> 1239506, 10278137 -> 1001523, 9939075 -> 1329390, 2037201 -> 1274155, 1248585 -> 2885, 1106945 -> 1399, 6811322 -> 1019016, 9978396 -> 1784, 6676961 -> 1086433, 2117821 -> 2611, 6863616 -> 1277013, 6895480 -> 1000993, 6831632 -> 1246136, 1001719 -> 1009727, 10135633 -> 4250, 7029291 -> 1034635, 6967939 -> 1002734, 6864694 -> 1017311, 1237279 -> 1029752, 6793956 -> 1283231, 1208609 -> 1000699, 6693428 -> 1100258, 685174...


构建训练集 

scala> import org.apache.spark.mllib.recommendation._
import org.apache.spark.mllib.recommendation._

scala> val bArtistAlias = sc.broadcastcast(artistAlias)
<console>:36: error: value broadcastcast is not a member of org.apache.spark.SparkContext
         val bArtistAlias = sc.broadcastcast(artistAlias)
                               ^

scala> val bArtistAlias = sc.broadcast(artistAlias)
bArtistAlias: org.apache.spark.broadcast.Broadcast[scala.collection.immutable.Map[Int,Int]] = Broadcast(3)

scala> 

scala> val rawUserArtistData = sc.textFile(base + "user_artist_data.txt")
rawUserArtistData: org.apache.spark.rdd.RDD[String] = hdfs:///user/miaofu/user_artist_data.txt MapPartitionsRDD[7] at textFile at <console>:32

scala> val trainData = rawUserArtistData.map{line => 
     | val Array(userID,artistID,count) = line.split(" ").map(_.toInt)
     | val finalArtistID =
     | bArtistAlias.value.getOrElse(artistID,artistID)
     | Rating(userID,finalArtistID,count)
     | }.cache()
trainData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.recommendation.Rating] = MapPartitionsRDD[8] at map at <console>:40

scala> trainData.count()
[Stage 1:==============>                                            (1 + 3) / 4]16/09/14 12:47:18 WARN MemoryStore: Not enough space to cache rdd_8_1 in memory! (computed 172.8 MB so far)
16/09/14 12:47:18 WARN MemoryStore: Not enough space to cache rdd_8_0 in memory! (computed 172.8 MB so far)
16/09/14 12:47:21 WARN MemoryStore: Not enough space to cache rdd_8_2 in memory! (computed 172.8 MB so far)
res0: Long = 24296858


模型训练

scala> val model = ALS.trainImplicit(trainData,10,5,0.01,1.0)
[Stage 2:==============>                                            (1 + 3) / 4]16/09/14 12:48:29 WARN MemoryStore: Not enough space to cache rdd_8_2 in memory! (computed 172.8 MB so far)
16/09/14 12:48:29 WARN MemoryStore: Not enough space to cache rdd_8_0 in memory! (computed 172.8 MB so far)
16/09/14 12:48:30 WARN MemoryStore: Not enough space to cache rdd_8_1 in memory! (computed 172.8 MB so far)
[Stage 3:>                                                          (0 + 4) / 4]16/09/14 12:49:12 ERROR Executor: Managed memory leak detected; size = 115703758 bytes, TID = 12
16/09/14 12:49:13 ERROR Executor: Exception in task 2.0 in stage 3.0 (TID 12)
因为个人的主机配置内存有限，所以出错了，先写到着，一会去申请新的配置。


Richard_More
关注
  • 1
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 4
    评论
专栏目录
基于Spark的协同过滤音乐推荐(python源码分析)
明月几时有，把酒问青天
11-24 4020
因为要进行大数据场景下利用Spark做音乐推荐，所以记录一下学习Spark的过程和实验过程。 目录一 对Spark整体上的认识二 spark mllib recommendation三 本文实验代码 一 对Spark整体上的认识 Spark是一个分布式计算引擎，将大量的计算任务分发到计算结点，实现分布式计算，和hadoop不同的是，Spark计算的中间结果存储在内存中，所以同等配置机器情况下，...
实现音乐推荐系统（源代码）
03-01
recommendation_engines.py Recommenders.py 推荐系统.ipynb
Spark实现的音乐艺术家推荐系统
03-08
Spark实现的音乐艺术家推荐系统。 搜集用户行为信息，某用户点击播放了哪位艺术家的歌曲，及其次数 利用Spark ALS 算法,给用户推荐适合他的艺术家的作品. 基于Yarn平台，使用Spark，数据清洗，预处理，定时同步增量数据至HDFS，定期重新构建模型，根据模型预测得分结果，进行推荐。使用Quatarz定时任务调度 scala bigdata
分享思路：Python+Spark爬虫音乐推荐系统 音乐数据分析可视化 音乐推荐app 音乐网站 大数据毕业设计(app+web+爬虫+spark+可视化)
从业计算机毕业设计10年，打算用一年时间分享10年毕设经验！
11-30 4055
目前市面上Python+Spark的爬虫音乐推荐系统+音乐数据分析可视化系统很少，于是我们设计了一套，希望给大家一套完整的设计方案和思路，助力大数据开发！
Spark高级数据分析· 3推荐引擎
weixin_33950035的博客
01-04 105
推荐算法流程 推荐算法 预备 wget http://www.iro.umontreal.ca/~lisa/datasets/profiledata_06-May-2005.tar.gz cd /Users/erichan/garden/spark-1.6.0-bin-hadoop2.6/bin ./spark-shell --master loca...
基于Python和Spark的大数据音乐推荐系统的设计与实现
m0_73268882的博客
07-14 2103
用户：注册登录、查看推荐歌单、数据分析、评论、歌单管理。图3.1 用户用例图管理员：登录、权限管理、系统管理、系统监控管理、开发平台管理、数据分析管理。图3.2 管理员用例图3.3.2。
spark mllib机器学习之六 ALS
深海小黄鱼
03-14 662
协同过滤采用音乐推荐的数据 http://www.iro.umontreal.ca/~lisa/datasets/profiledata_06-May-2005.tar.gz   package com.agm.practice import java.io.File import org.apache.spark.SparkConf import org.apache.
根据用户对类目偏好打分训练ALS模型
qq_41810183的博客
01-16 320
ALS的意思是交替最小二乘法（Alternating Least Squares），是Spark2.*中加入的进行基于模型的协同过滤（model-based CF）的推荐系统算法。但这里我们将使用的Spark的ALS模型进行CF推荐，因此注意这里数据输入不需要提前转换为矩阵，直接是 USER-ITEM-RATE的数据。注意：由于数据量巨大，因此这里也不考虑基于内存的CF算法。同SVD，它也是一种矩阵分解技术，对数据进行降维处理。基于Spark的ALS隐因子模型进行CF评分预测。
spark1.6.1及scala2.11.8安装配置
WRM的博客
05-31 1917
首先，安装spark之前需要安装配置的软件有：JDK，Scala,ssh,Hadoop这些开发平台的安装配置在我之前的博客中都有详细的攻略，需要的请去看看。 hadoop安装配置再此提一句，无论是hadoop，hbase，hive，spark都是需要版本适配的，不然就会多很多步的不必要操作，版本的适配官网上都有，这里写者是使用：jdk1.7+hadoop2.6.4+scala2.11.8+spar
spark高级数据分析系列之第三章音乐推荐和 Audioscrobbler 数据集
sam的博客
07-14 3284
3.1数据集和整体思路数据集本章实现的是歌曲推荐，使用的是ALS算法，ALS是spark.mllib中唯一的推荐算法，因为只有ALS算法可以进行并行运算。使用数据集在这里，里面包含该三个文件：表一：user_artist_data.txt   包含该的是（用户ID、歌曲ID、用户听的次数）  表二：artist_data.txt   这个文件包含的是（歌曲ID，歌曲名字）表三：artist_ali...
Spark 运行内存不足Not enough space to cache rdd in memory，Container killed by YARN for exceeding memory
热门推荐
jast
04-15 1万+
日志报错（WARN类型最后执行成功可以忽略）： 19/04/15 12:35:37 INFO memory.MemoryStore: Will not store rdd_2_51 19/04/15 12:35:37 WARN memory.MemoryStore: Not enough space to cache rdd_2_51 in memory! (computed 1109.7 ...
Spark平台下基于协同过滤算法的实时电影推荐系统源代码
12-10
基于深度学习的监督学习，使用梯度下降、ALS、LFM算法，使用AngularJS2生成前端框架，数据库为MongoDB，使用ElasticSearch作为搜索服务器，Redis作为缓存数据库，其中包括Spark的离线统计服务、Azkaban的工作调度...
Spark MLlib协同过滤电影推荐系统Scala源码
最新发布
03-25
项目核心功能：基于Spark MLlib强大的计算能力，本项目实现了高效的协同过滤算法，为用户推荐电影，提升用户体验。 简而言之，该项目是一个集成了多种技术，专注于提供个性化电影推荐服务的Scala应用。
基于spark的机器学习-基于物品的协同过滤算法实现
11-09
2、本项目适合计算机相关专业(如计科、人工智能、通信工程、自动化、电子信息等)的在校学生、老师或者企业员工下载学习，也适合小白学习进阶，当然也可作为毕设项目、课程设计、作业、项目初期立项演示等。...
基于Spark实现的协同过滤推荐算法的电影推荐系统+源代码+文档说明
12-28
1、资源内容：基于Spark实现的协同过滤推荐算法的电影推荐系统+源代码+文档说明 2、代码特点：内含运行结果，不会运行可私信，参数化编程、参数可方便更改、代码编程思路清晰、注释明细，都经过测试运行成功，功能ok...
评估深度学习架构在Spark集群的应用：从theano，keras到elephas
Richard_More的博客
11-18 4685
最终目标 最终目标：为了提高机器学习特别是深度学习的计算速度。提供的速度有三个方法：让算法的变得更加聪明（数据结构）；让单个机器的计算能力增强（更好的CPU／GPU）；让计算并行化（多线程；Hadoop／Spark）。本文仅仅探索第三种方式：评估不同的深度学习框架（是否支持GPU，易于实现，速度快）以及其如何并行化于分布式的集群之上。 theano是一个python的包，用数组向量来定义和计算
解析Spark开源框架elephas之一
Richard_More的博客
11-19 4050
写在前面的话 elephas是一个把python深度学习框架keras衔接到Spark集群的第三方python包。由于这个版本并不稳定，并且没有什么资料，我打算剖析其源代码。 分析代码要从其主程序开始，就是spark_model.py，其网址在 https://github.com/maxpumperla/elephas/blob/master/elephas/spark_model.
基于Spark分析寻医问药网的问答数据
Richard_More的博客
09-19 2766
scala> val in = sqlContext.jsonFile("hdfs:///user/miaofu/healthcare/2016-05-01/content.txt") warning: there were 1 deprecation warning(s); re-run with -deprecation for details 16/09/19 15:05:17 INFO B
sparkmllib 协同过滤做用户推荐算法
06-08
好的，关于使用 Spark MLlib 实现协同过滤的用户推荐算法，我可以为您提供一些思路和步骤： 1. 数据准备：需要准备用户对物品的评分数据。评分数据可以是一个矩阵，其中每一行表示一个用户对所有物品的评分，每一列表示一个物品被所有用户的评分。评分可以是连续的实数值，也可以是离散的整数值。 2. 数据预处理：对评分数据进行缺失值、异常值和重复值的处理，以及对评分数据进行归一化或标准化。 3. 模型训练：使用 Spark MLlib 中的 ALS（Alternating Least Squares）算法进行模型训练。该算法是一种迭代优化算法，通过交替优化用户和物品的隐含特征向量来进行训练。训练过程需要指定模型的超参数，如隐含特征向量的维度、迭代次数、正则化参数等。 4. 模型评估：使用测试集或交叉验证等方法对模型进行评估，比较不同超参数设置下的模型性能，并选择最优的模型。 5. 推荐生成：使用训练好的模型对未评分的物品进行预测，生成推荐列表。推荐列表的排序可以根据预测评分值或其他策略进行。 以上是使用 Spark MLlib 实现协同过滤的用户推荐算法的基本步骤。希望能对您有所帮助。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值