一碗竹叶青的博客

参考：https://www.it-swarm.net/zh/apache-spark/%E5%A6%82%E4%BD%95%E4%BB%8Edataframe%E5%87%86%E5%A4%87%E6%95%B0%E6%8D%AE%E5%88%B0libsvm%E6%A0%BC%E5%BC%8F%EF%BC%9F/828617867/两个步骤：1.将特征转换为LabeledPoint数据X，或者将采用vectorAssembler将特征转化为稀疏矩阵2.然后以libsvm格式保存X.// 采用v

参考：https://my.oschina.net/kyo4321/blog/2050708https://cloud.tencent.com/developer/article/1021585对于one hot编码首先需要将数据数值化，然后把将数值进行onehot所有的操作都是对dataframe来进行的，参看如何RDD转换dataframe数值化import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer, VectorAs

将用户数据（u.user）复制到HDFS上hadoop fs -put u.user /home/hadoop/data/查看是否复制成功：hadoop fs -ls /home/hadoop/data/读取数据import org.apache.spark.ml.evaluation.RegressionEvaluatorimport org.apache . spark.ml.recommendation.ALSimport org.apache.spark.sql.SparkSessi

数据预处理scala> import org.apache.spark.ml.feature.LabeledPointscala> import org.apache.spark.ml.linalg.Vectorsval data = records.map {r =>val trimmed = r.map （一 .replaceAll （”＼””，””｝）val label= trimmed(r.size - l).torntval features = trimmed.

对2分类或者多分类模型的评估：// Precision by thresholdval precision = metrics.precisionByThresholdprecision.foreach { case (t, p) => println(s"Threshold: $t, Precision: $p")}// Recall by thresholdval recall = metrics.recallByThresholdrecall.foreach { case

// 显示每个类目的缺失值data.select(data.columns.map(c => sum(col(c).isNull.cast("int")).alias(c)): _*).showdata.groupBy("cate1").count().show(30)println(data.count()) //3046def SortedCate(df:DataFrame): DataFrame ={// 循环类别赋值label val N = 10 val window =

val embddding_seq_data = ss.read.option("sep","\t") .csv("hdfs://ss-sng-dc-v2/stage/outface/SNG/g_sng_weishi_ws_growth/tdw_parkerlin/qq/feature/embedding/bert_embedding/"+endTime_1 +"/") .toDF("feedid","embedding")embddding_seq_data以tfrecode形式写入hdfs

https://www.jianshu.com/p/c6f15063d521https://blog.csdn.net/qq_33938256/article/details/52584964#%E8%A1%8C%E7%9F%A9%E9%98%B5rowmatrix简介：MLLib提供了一系列基本数据类型以支持底层的机器学习算法。主要的数据类型包括标注点Labeled Point）、本地向量（Local Vector）、本地矩阵、分布式矩阵等。单机模式存储的本地向量与矩阵，以及基于一个或多个RDD的