2020年03月_楓尘林间

原创 Spark DataFrame转化为RDD[Vector]，应用于KMeans聚类，monotonically_increasing_id()方法进行结果合并

将DataFrame转化为RDD[Vector]，应用于KMeans聚类模型训练import org.apache.spark.mllib.linalg.Vectorsimport org.apache.spark.rdd.RDD import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}import org.apac...

2020-03-30 18:50:49 2092 3

原创 Scala语法向上取整和向下取整

// 向下取整scala.math.floor(7.5)结果：Double = 7.0// 向上取整scala.math.floor(7.5)结果：Double = 8.0// 四舍五入scala.math.round(7.5)结果：Long = 8round 将long和double转化为long型将float转化为int型def round(x : scala....

2020-03-30 14:09:56 9117 2

原创 Spark sql 单引号'' 使用问题

spark sql中单引号意味着要某个变量所代表的值如var q = 111spark.sql(s"select '$q' as t").show()+---+| t|+---+|111|+---+去掉单引号spark.sql(s"select $q as t").show()+---+| t|+---+|111|+---+两者在该情境下是...

2020-03-27 16:37:57 3376

原创 Spark scala中将Array(Row)转化为DataFrame（另一种抽样方式）

1.spark DataFrame常规的采样方式——sample()由于scala中对DataFrame 方式为 sample() 方法sample : 采样采样变换根据给定的随机种子，从RDD中随机地按指定比例选一部分记录，创建新的RDD。采样变换在机器学习中可用于进行交叉验证。语法def sample(withReplacement: Boolean, fraction: Do...

2020-03-20 14:10:03 7967

原创 Spark Dataframe 按某一行排序新加一列index

1.Dataframe按某一行排序新加一列index (index不重复)import spark.implicits._import spark.implicits._var data2 = Seq( ("0", "2002", "196", "1", "bai"), ("1", "4004", "192", "2", "wang"), ("0", "7007", "95...

2020-03-19 20:44:21 1860

原创 Spark DataFrame统计某列特征不同个数

Scala版本import spark.implicits._var data1 = Seq( ("0", "2002", "196", "1", "bai"), ("1", "4004", "192", "2", "wang"), ("0", "7007", "95", "3", "wang"), ("0", "4004", "4", "4", "wang"), ("0...

2020-03-19 17:18:58 10143

原创 SparkDataFrame求某一列的最大值和最小值和求和

1.利用Scala沟边import spark.implicits._var data1 = Seq( ("0", "2002", "196", "1", "bai"), ("1", "4004", "192", "2", "wang"), ("0", "7007", "95", "3", "wang"), ("0", "4004", "4", "4", "wang")...

2020-03-18 13:44:38 13726

转载 Scala Array数组和ArrayBuffer可变数组的相互转化

Array 转 ArrayBuffer将Array转换成已经声明的类型为ArrayBuffer的成员变量，需要使用asInstanceOf进行强制转换val array = Array[String]("a", "b", "c")println(s"array=$array")val arrayToBuffer = array.toBuffer.asInstanceOf[ArrayBuff...

2020-03-16 14:54:59 3511

原创 Spark DataFrame选择其中一列或几列

DataFrame使用抽取一列或几列import spark.implicits._var data1 = Seq( ("0", "ming", "tj","2019-09-06 17:15:15", "2002", "192.196", "win7", "bai"), ("1", "ming", "tj","2019-09-07 16:15:15", "4004", "192.19...

2020-03-16 14:38:49 15402 1

基本类型：类型Byte Short Int Long和Char被称为整数类型(integral type)，整数类型加上Float和Double被称为数类型(numeric type) NumericType。官方API说明文档:http://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/NumericType....

2020-03-16 11:25:08 4639

Buevara的博客