背景:
在向量大火的行情下,不管是召回还是精排,user和item的相似度计算,必不可少;很多情况下,为了节省线上加载计算的时间,会将user和item的向量,离线计算好,放到存储系统中,线上使用的时候,直接通过KV的形式读取。(没有AB过会节省多少时间,在排序服务中,直接将二者向量导入KV存储系统,直接在线上进行计算,目测也没有增加多少耗时)
先来看看离线如何计算。通常使用spark进行大量的数据计算。其优势见spark优势。
例子:
假设table1中存放user的向量,用逗号分隔,形如:
user1 vector
1213 1,2,3,4,5
5432 3,4,5,6,7
table2中存放item的向量,用逗号分隔,形如:
item vector
9676 1,2,3,4,5
4563 3,4,5,6,7
要求:分别计算user和item的相似度,返回每一个用户top10的item;
代码:
先熟悉下计算相似度的方式有哪些?也就是衡量“距离的方法”,这里可供参考相似度计算。
语言上使用scala。逻辑会在代码中注释。
1.读取文件
从idea上建设一个文件(这个类别选择object,因为这个文件中包含了执行入口main函数):calculation.scala
先将user和item的向量做一个内积。
import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
import org.apache.spark.sql.functions._
import scala.collection.mutable
import java.util
import org.apache.hadoop.fs.Path
import org.apache.spark.rdd.RDD
import scala.io.Source
import org.apache.spark.sql._
import org.apache.spark.sql.types.{StringType, StructField, StructType}
import org.apache.spark.sql.SparkSession
object calculation {
def main (args: Array[String]): Unit = {
val spark = SparkSession.builder().enableHiveSupport().getOrCreate()
import spark.implicits._
val userData = getUserSqlData().na.fill(0)
val itemData = getItemSqlData().na.fill(0)
val data = userData.join(itemData)
}
//读取table中的字段
def getUserSql(): String {
"""
select userid, userVector from table1
""".stripMargin
}
def getItemSql(): String {
"""
select itemid, itemVector from table2
""".stripMargin
}
def getUserSqlData ():DataFrame {
spark.sql(getUserSql())
}
def getItemSqlData ():DataFrame {
spark.sql(getUserSql())
}
}
2.计算相似度
新建一个类(这里建设的时候选择class,我们想要的是一个方法,将这个方法在main函数中调用即可)similarity.scala,用于计算相似度。
在main函数中的调用形式:
//similarity计算返回结果:userid itemid similarity
val similarity = new similarity()
val similarityData = similarity.vectorSimilarity(data)
similarity.scala文件详情:
def vectorSimilarity(data: DataFrame): DataFrame = {
var df_res: DataFrame = data
// df.printSchema()
val innerPro: (String, String) => Double = (c1: String, c2: String) =>{
val c1Arr = c1.split(",", -1).map(_.toDouble)
val c2Arr = c2.split(",", -1).map(_.toDouble)
var inner = 0.0
if (c1 == null || c2 == null){
inner = 0.0
}
else{
inner = c1Arr.zip(c2Arr).map(x => x._1 * x._2).sum
}
inner
}
val innerUdf = udf(innerPro)
val cosineSim: (String, String) => Double = (c1: String, c2: String) =>{
val c1Arr: Array[Double] = c1.split(",", -1).map(_.toDouble)
val c2Arr = c2.split(",", -1).map(_.toDouble)
var cos = 0.0
var v1: Double = c1Arr.zip(c1Arr).map(x=>x._1*x._2).sum
var v2 = c2Arr.zip(c2Arr).map(x=>x._1*x._2).sum
val v3: Double = innerPro(c1,c2)
if (v1 != 0.0 && v2 != 0.0) {
v1 = math.sqrt(v1)
v2 = math.sqrt(v2)
cos = v3 / (v1 * v2)
}
cos
}
val cosUdf = udf(cosineSim)
val EuclideanDis: (String,String) => Double =(c1:String,c2:String)=>{
val c1Arr = c1.split(",", -1).map(_.toDouble)
val c2Arr = c2.split(",", -1).map(_.toDouble)
val temp = c1Arr.zip(c2Arr).map(x=>(x._1 - x._2) * (x._1 - x._2)).sum
val sqrtTemp = math.sqrt(temp)
sqrtTemp
}
val EucUdf = udf(EuclideanDis)
val ManhattanDis :(String,String) =>Double=(c1:String,c2:String)=>{
val c1Arr = c1.split(",", -1).map(_.toDouble)
val c2Arr = c2.split(",", -1).map(_.toDouble)
val man = c1Arr.zip(c2Arr).map(x=> math.abs(x._1-x._2)).sum
man
}
val ManUdf = udf(ManhattanDis)
val col_list= df.columns.toList
df_res = df_res.withColumn("InnerProduct",innerUdf(col("userVector"),col("itemVector")))
df_res = df_res.withColumn("CosinProduct",innerUdf(col("userVector"),col("itemVector")))
df_res = df_res.withColumn("EucliProduct",innerUdf(col("userVector"),col("itemVector")))
df_res = df_res.withColumn("ManhaProduct",innerUdf(col("userVector"),col("itemVector")))
}
df_res
}
上面的相似度计算略微有点繁琐,参考一个tf-idf的文档相似度计算方式,地址:
import org.apache.spark.mllib.feature.{HashingTF, IDF}
import org.apache.spark.mllib.linalg.{SparseVector => SV}
import org.apache.spark.{SparkConf, SparkContext}
import scala.io.Source
/**
* Created by xiaojun on 2015/10/19.
*/
object TFIDFDemo {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("TfIdfTest").setMaster("local")
val sc = new SparkContext(conf)
// Load documents (one per line).要求每行作为一个document,这里zipWithIndex将每一行的行号作为doc id
val documents = sc.parallelize(Source.fromFile("CHANGELOG").getLines().filter(_.trim.length > 0).toSeq).map(_.split(" ").toSeq).zipWithIndex()
val hashingTF = new HashingTF(Math.pow(2, 18).toInt)
//这里将每一行的行号作为doc id,每一行的分词结果生成tf词频向量
val tf_num_pairs = documents.map {
case (seq, num) =>
val tf = hashingTF.transform(seq)
(num, tf)
}
tf_num_pairs.cache()
//构建idf model
val idf = new IDF().fit(tf_num_pairs.values)
//将tf向量转换成tf-idf向量
val num_idf_pairs = tf_num_pairs.mapValues(v => idf.transform(v))
//广播一份tf-idf向量集
val b_num_idf_pairs = sc.broadcast(num_idf_pairs.collect())
//计算doc之间余弦相似度
val docSims = num_idf_pairs.flatMap {
case (id1, idf1) =>
val idfs = b_num_idf_pairs.value.filter(_._1 != id1)
val sv1 = idf1.asInstanceOf[SV]
import breeze.linalg._
val bsv1 = new SparseVector[Double](sv1.indices, sv1.values, sv1.size)
idfs.map {
case (id2, idf2) =>
val sv2 = idf2.asInstanceOf[SV]
val bsv2 = new SparseVector[Double](sv2.indices, sv2.values, sv2.size)
val cosSim = bsv1.dot(bsv2).asInstanceOf[Double] / (norm(bsv1) * norm(bsv2))
(id1, id2, cosSim)
}
}
docSims.foreach(println)
sc.stop()
}
}
3.返回top10
深刻的体会到好久不写真的会无从下手,18年开发过类似的内容,今日顿觉无从下手。或许并没有真的会,哈哈哈^.^