文章目录
- 课程地址:尚硅谷大数据项目教程(大数据实战电商推荐系统)
- 尚硅谷电商推荐系统的配套资料及虚拟机
链接:https://pan.baidu.com/s/1iSMqV2wPkEfIsO1FrkxRNQ?pwd=1996
提取码:1996 - 1.项目体系架构设计(电商推荐系统)
- 2.工具环境搭建(电商推荐系统)
- 3.创建项目并初始化业务数据(电商推荐系统)
- 4.离线推荐服务建设(电商推荐系统)
- 5.实时推荐服务建设(电商推荐系统)
- 6.冷启动问题处理(电商推荐系统)
- 7.基于内容的相似推荐与基于物品的协同过滤推荐
- 8.尚硅谷电商推荐系统预览
一、基于内容的相似推荐
原始数据中的tag
文件,是用户给商品打上的标签,这部分内容想要直接转成评分并不容易,不过我们可以将标签内容进行提取,得到商品的内容特征向量,进而可以通过求取相似度矩阵。这部分可以与实时推荐系统直接对接,计算出与用户当前评分商品的相似商品,实现基于内容的实时推荐。为了避免热门标签对特征提取的影响,我们还可以通过TF-IDF
算法对标签的权重进行调整,从而尽可能地接近用户偏好。
基于以上思想,加入TF-IDF
算法的求取商品特征向量的核心代码如下:
在pom.xml
中添加如下依赖:
<dependencies>
<dependency>
<groupId>org.scalanlp</groupId>
<artifactId>jblas</artifactId>
<version>${
jblas.version}</version>
</dependency>
<!-- Spark的依赖引入 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
</dependency>
<!-- 引入Scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<!-- 加入MongoDB的驱动 -->
<!-- 用于代码方式连接MongoDB -->
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>casbah-core_2.11</artifactId>
<version>${
casbah.version}</version>
</dependency>
<!-- 用于Spark和MongoDB的对接 -->
<dependency>
<groupId>org.mongodb.spark</groupId>
<artifactId>mongo-spark-connector_2.11</artifactId>
<version>${
mongodb-spark.version}</version>
</dependency>
</dependencies>
添加日志文件log4j.properties
log4j.rootLogger=info, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{
yyyy-MM-dd HH:mm:ss,SSS} %5p --- [%50t] %-80c(line:%5L) : %m%n
新建com.atguigu.content.ContentRecommender
package com.atguigu.content
import org.apache.spark.SparkConf
import org.apache.spark.ml.feature.{
HashingTF, IDF, IDFModel, Tokenizer}
import org.apache.spark.ml.linalg.SparseVector
import org.apache.spark.sql.{
DataFrame, SparkSession}
import org.jblas.DoubleMatrix
/*
*Product数据集
* 6797 商品ID
* PHILIPS飞利浦HQ912/15两刀头充电式 商品名称
* 222,621,691 商品分类ID,不需要
* B002TKLK0S 亚马逊ID,不需要
* https://images-cn-4.ssl-images-a 商品的网址URI
* 家用电器|个人护理电器|电动剃须刀 商品的分类
* 飞利浦|剃须刀|家用电器|好用|外观漂亮 商品UGC标签
* */
//定义样例类
case class Product(productId: Int, name: String, imageUri: String, categories: String, tags: String)
/*
* MongoDB连接配置
* uri:MongoDB连接的uri
* db:要操作的db
* */
//将mongodb封装成样例类
case class MongoConfig(uri: String, db: String)
//定义标准推荐对象
case class Recommendation(productId: Int, score: Double)
//定义商品相似度列表
case class ProductRecs(productId: Int, recs: Seq[Recommendation])
object ContentRecommender {
//定义mongodb中存储的表名
val MONGODB_PRODUCT_COLLECTION = "Product"
//定义商品推荐列表
val Content_PRODUCT_RECS = "ContentBasedProductRecs"
//定义推荐的个数(最大个数)
val USER_MAX_RECOMMENDATION = 20
def main(args: Array[String]): Unit = {
//配置信息
val config: Map[String, String] = Map(
"spark.cores" -> "local[*]",
"mongo.uri" -> "mongodb://localhost:27017/recommender",
"mongo.db" -> "recommender"
)
//构建SparkSession执行环境入口对象
val conf: SparkConf = new SparkConf().setMaster(config("spark.cores")).setAppName("ContentRecommender")
val spark: SparkSession = SparkSession.builder().config(conf).getOrCreate()
import spark.implicits._
//声明一个隐式的配置对象,隐式参数--mongodb的连接配置
implicit val mongoConfig: MongoConfig = MongoConfig(config("mongo.uri"), config("mongo.db"))
//载入数据,做预处理
val productTagsDF: DataFrame = spark.read
.option("uri", mongoConfig.uri)
.option("collection", MONGODB_PRODUCT_COLLECTION)
.format("com.mongodb.spark.sql")
.load()
.as[Product]
.map {
x => (x.productId, x.name, x.tags.map(c => if (c == '|') ' ' else c))
}
.toDF("productId", "name", "tags")
.cache()
//TODO:用TF-IDF提取商品的特征向量
//1.实例化一个分词器用来做分词,默认按空格分
val tokenizer: Tokenizer = new Tokenizer().setInputCol("tags").setOutputCol("words")
// 用分词器做转换,得到增加新列words的DF
val wordsData: DataFrame = tokenizer.transform(productTagsDF)
// 定义一个HashingTF工具,计算TF,可以设置特征数量setNumFeatures
val hashingTF: HashingTF = new HashingTF().setInputCol("words").setOutputCol("rawFeatures")
// 用 HashingTF 做处理
val featurizedData: DataFrame = hashingTF.transform(wordsData)
// 定义一个IDF工具,计算TF-IDF
val idf: IDF = new IDF().setInputCol("rawFeatures").setOutputCol("features")
// 将词频数据传入,得到idf模型(统计文档)
val idfModel: IDFModel = idf.fit(featurizedData)
//得到增加新列feature的DF
val rescaledDataDF: DataFrame = idfModel.transform(featurizedData)
// rescaledDataDF.show(truncate = false)
//对数据做格式转换,得到RDD形式的features
//SparseVector为稀疏向量格式
val productFeatures = rescaledDataDF.map {
row => (row.getAs[Int]("productId"), row.getAs[SparseVector]("features").toArray)
}
.rdd
.map {
case (productId, features) => (productId, new DoubleMatrix(features))
}
//两两配对商品,计算余弦相似度
val productRecs: DataFrame = productFeatures.cartesian(productFeatures)
.filter {
//模式匹配,将自己过滤掉
case (a, b) => a._1 != b._1
}
.map {
case (a, b) =>
val simScore: Double = consinSim(a._2, b._2)
(a._1, (b._1, simScore))
}
.filter(_._2._2 > 0.6)
.groupByKey()
.map {
//模式匹配
case (productId, recs) =>
ProductRecs(productId, recs.toList.sortWith(_._2 > _._2).map(x => Recommendation(x._1, x._2)))
}
.toDF()
storeDFInMongoDB(productRecs, Content_PRODUCT_RECS)
//关闭spark
spark.stop()
}
// 计算余弦相似度
def consinSim(product1: DoubleMatrix, product2: DoubleMatrix): Double = {
product1.dot(product2) / (product1.norm2() * product2.norm2())
}
def storeDFInMongoDB(df: DataFrame, collection_name: String)(implicit mongoConfig: MongoConfig): Unit = {
df
.write
.option("uri", mongoConfig.uri)
.option("collection", collection_name)
.mode("overwrite")
.format("com.mongodb.spark.sql")
.save()
}
}
在mongodb中查看运行结果:
> db
test
> user recommender
2022-08-08T16:56:30.006+0800 E QUERY [js] uncaught exception: SyntaxError: unexpected token: identifier :
@(shell):1:5
> use recommender
switched to db recommender
> show tables
AverageProducts
ContentBasedProductRecs
Product
ProductRecs
RateMoreProducts
RateMoreRecentlyProducts
Rating
UserRecs
> db.ContentBasedProductRecs.find().pretty()
{
"_id" : ObjectId("62f0ce71cde8324a787721ad"),
"productId" : 457976,
"recs" : [
{
"productId" : 425715,
"score" : 1
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721ae"),
"productId" : 442353,
"recs" : [
{
"productId" : 492579,
"score" : 0.6285606335977485
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721af"),
"productId" : 425715,
"recs" : [
{
"productId" : 457976,
"score" : 1
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b0"),
"productId" : 367460,
"recs" : [
{
"productId" : 352021,
"score" : 0.8444577485612076
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b1"),
"productId" : 302217,
"recs" : [
{
"productId" : 90897,
"score" : 0.6481013731409726
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b2"),
"productId" : 13316,
"recs" : [
{
"productId" : 8195,
"score" : 0.6207527403278441
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b3"),
"productId" : 156103,
"recs" : [
{
"productId" : 102383,
"score" : 0.7723406183991424
},
{
"productId" : 116405,
"score" : 0.7701510843253493
},
{
"productId" : 183418,
"score" : 0.6372189590966008
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b4"),
"productId" : 492579,
"recs" : [
{
"productId" : 442353,
"score" : 0.6285606335977485
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b5"),
"productId" : 203971,
"recs" : [
{
"productId" : 387693,
"score" : 0.7060608810487017
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b6"),
"productId" : 90897,
"recs" : [
{
"productId" : 302217,
"score" : 0.6481013731409726
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b7"),
"productId" : 183418,
"recs" : [
{
"productId" : 383502,
"score" : 0.6373419620037125
},
{
"productId" : 156103,
"score" : 0.6372189590966008
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b8"),
"productId" : 102383,
"recs" : [
{
"productId" : 116405,
"score" : 0.8558026164164717
},
{
"productId" : 156103,
"score" : 0.7723406183991424
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721b9"),
"productId" : 352021,
"recs" : [
{
"productId" : 367460,
"score" : 0.8444577485612076
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721ba"),
"productId" : 474201,
"recs" : [
{
"productId" : 387693,
"score" : 0.6745125587706321
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721bb"),
"productId" : 275707,
"recs" : [
{
"productId" : 387693,
"score" : 0.619549946090946
}
]
}
{
"_id" : ObjectId("62f0ce71cde8324a787721bc"),
"productId" : 383502,
"recs" : [
{
"productId" :