Spark2-oneHot编码-标准化-主成分-聚类

最新推荐文章于 2023-05-13 17:46:28 发布
最新推荐文章于 2023-05-13 17:46:28 发布
阅读量329 收藏 1
点赞数
文章标签： 大数据 人工智能 数据结构与算法

  1. // affairs：一年来婚外情的频率
  2. // gender：性别
  3. // age：年龄
  4. // yearsmarried：婚龄
  5. // children：是否有小孩
  6. // religiousness：宗教信仰程度（5分制，1分表示反对，5分表示非常信仰）
  7. // education：学历
  8. // occupation：职业（逆向编号的戈登7种分类）
  9. // rating：对婚姻的自我评分（5分制，1表示非常不幸福，5表示非常幸福）

  11. import org.apache.spark.sql.SparkSession
  12. import org.apache.spark.sql.Dataset
  13. import org.apache.spark.sql.Row
  14. import org.apache.spark.sql.DataFrame
  15. import org.apache.spark.sql.Column
  16. import org.apache.spark.sql.DataFrameReader
  17. import org.apache.spark.rdd.RDD
  18. import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
  19. import org.apache.spark.sql.Encoder
  20. import org.apache.spark.ml.linalg.Vectors
  21. import org.apache.spark.ml.feature.StringIndexer
  22. import org.apache.spark.ml.feature.OneHotEncoder
  23. import org.apache.spark.ml.feature.VectorAssembler
  24. import org.apache.spark.ml.feature.StandardScaler
  25. import org.apache.spark.ml.feature.PCA
  26. import org.apache.spark.ml.clustering.KMeans

  28. scala> val spark = SparkSession.builder().appName("Spark SQL basic example").config("spark.some.config.option", "some-value").getOrCreate()

  30. scala>

  32. scala> // For implicit conversions like converting RDDs to DataFrames
  33. scala> import spark.implicits._


  36. scala> val data: DataFrame = spark.read.format("csv").option("header", true).load("hdfs://ns1/datafile/wangxiao/Affairs.csv")
  37. data: org.apache.spark.sql.DataFrame = [affairs: string, gender: string ... 7 more fields]

  39. scala>

  41. scala> data.cache
  42. res0: data.type = [affairs: string, gender: string ... 7 more fields]

  44. scala>

  46. scala> data.limit(10).show()





  52. scala>

  54. scala> // 转换字符类型，将Double和String的字段分开放

  56. scala> val data1 = data.select(
  57.      | data("affairs").cast("Double"),
  58.      | data("age").cast("Double"),
  59.      | data("yearsmarried").cast("Double"),
  60.      | data("religiousness").cast("Double"),
  61.      | data("education").cast("Double"),
  62.      | data("occupation").cast("Double"),
  63.      | data("rating").cast("Double"),
  64.      | data("gender").cast("String"),
  65.      | data("children").cast("String"))
  66. data1: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 7 more fields]

  68. scala>

  70. scala> data1.printSchema()
  71. root
  72.  |-- affairs: double (nullable = true)
  73.  |-- age: double (nullable = true)
  74.  |-- yearsmarried: double (nullable = true)
  75.  |-- religiousness: double (nullable = true)
  76.  |-- education: double (nullable = true)
  77.  |-- occupation: double (nullable = true)
  78.  |-- rating: double (nullable = true)
  79.  |-- gender: string (nullable = true)
  80.  |-- children: string (nullable = true)


  83. scala> data1.limit(10).show




  88. scala>

  90. scala> val dataDF = data1
  91. dataDF: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 7 more fields]

  93. scala>

  95. scala> dataDF.cache()
  96. res4: dataDF.type = [affairs: double, age: double ... 7 more fields]

  98. scala>

  100. scala> //###################################

  102. scala> val indexer = new StringIndexer().setInputCol("gender").setOutputCol("genderIndex").fit(dataDF)
  103. indexer: org.apache.spark.ml.feature.StringIndexerModel = strIdx_19a888aff882

  105. scala> val indexed = indexer.transform(dataDF)
  106. indexed: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 8 more fields]

  108. scala> // OneHot编码，注意setDropLast设置为false

  110. scala> val encoder = new OneHotEncoder().setInputCol("genderIndex").setOutputCol("genderVec").setDropLast(false)
  111. encoder: org.apache.spark.ml.feature.OneHotEncoder = oneHot_f0f47e0b5b37

  113. scala> val encoded = encoder.transform(indexed)
  114. encoded: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 9 more fields]

  116. scala> encoded.show()



  120. scala>

  122. scala> val indexer1 = new StringIndexer().setInputCol("children").setOutputCol("childrenIndex").fit(encoded)
  123. indexer1: org.apache.spark.ml.feature.StringIndexerModel = strIdx_7e4d8c69b823

  125. scala> val indexed1 = indexer1.transform(encoded)
  126. indexed1: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 10 more fields]

  128. scala> val encoder1 = new OneHotEncoder().setInputCol("childrenIndex").setOutputCol("childrenVec").setDropLast(false)
  129. encoder1: org.apache.spark.ml.feature.OneHotEncoder = oneHot_9a8906781325

  131. scala> val encoded1 = encoder1.transform(indexed1)
  132. encoded1: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 11 more fields]

  134. scala> encoded1.show()



  138. scala>

  140. scala> val encodeDF: DataFrame = encoded1
  141. encodeDF: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 11 more fields]

  143. scala> encodeDF.show()



  147. scala> encodeDF.printSchema()
  148. root
  149.  |-- affairs: double (nullable = true)
  150.  |-- age: double (nullable = true)
  151.  |-- yearsmarried: double (nullable = true)
  152.  |-- religiousness: double (nullable = true)
  153.  |-- education: double (nullable = true)
  154.  |-- occupation: double (nullable = true)
  155.  |-- rating: double (nullable = true)
  156.  |-- gender: string (nullable = true)
  157.  |-- children: string (nullable = true)
  158.  |-- genderIndex: double (nullable = true)
  159.  |-- genderVec: vector (nullable = true)
  160.  |-- childrenIndex: double (nullable = true)
  161.  |-- childrenVec: vector (nullable = true)


  164. scala>

  166. scala> //#################################

  168. scala> val assembler = new VectorAssembler().setInputCols(Array("affairs", "age", "yearsmarried", "religiousness", "education", "occupation", "rating", "genderVec", "childrenVec")).setOutputCol("features")
  169. assembler: org.apache.spark.ml.feature.VectorAssembler = vecAssembler_8ccd528981cd

  171. scala>

  173. scala> val vecDF: DataFrame = assembler.transform(encodeDF)
  174. vecDF: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 12 more fields]

  176. scala> vecDF.select("features").show
  177. 16/11/05 15:56:14 WARN Executor: 1 block locks were not released by TID = 11:
  178. [rdd_17_0]
  179. +--------------------+
  180. | features|
  181. +--------------------+
  182. |[0.0,37.0,10.0,3....|
  183. |[0.0,27.0,4.0,4.0...|
  184. |[0.0,32.0,15.0,1....|
  185. |[0.0,57.0,15.0,5....|
  186. |[0.0,22.0,0.75,2....|
  187. |[0.0,32.0,1.5,2.0...|
  188. |[0.0,22.0,0.75,2....|
  189. |[0.0,57.0,15.0,2....|
  190. |[0.0,32.0,15.0,4....|
  191. |[0.0,22.0,1.5,4.0...|
  192. |[0.0,37.0,15.0,2....|
  193. |[0.0,27.0,4.0,4.0...|
  194. |[0.0,47.0,15.0,5....|
  195. |[0.0,22.0,1.5,2.0...|
  196. |[0.0,27.0,4.0,4.0...|
  197. |[0.0,37.0,15.0,1....|
  198. |[0.0,37.0,15.0,2....|
  199. |[0.0,22.0,0.75,3....|
  200. |[0.0,22.0,1.5,2.0...|
  201. |[0.0,27.0,10.0,2....|
  202. +--------------------+
  203. only showing top 20 rows


  206. scala>

  208. scala> // 标准化--均值标准差

  210. scala> val scaler = new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithStd(true).setWithMean(true)
  211. scaler: org.apache.spark.ml.feature.StandardScaler = stdScal_2e35fbc29084

  213. scala>

  215. scala> // Compute summary statistics by fitting the StandardScaler.

  217. scala> val scalerModel = scaler.fit(vecDF)
  218. scalerModel: org.apache.spark.ml.feature.StandardScalerModel = stdScal_2e35fbc29084

  220. scala>

  222. scala> // Normalize each feature to have unit standard deviation.

  224. scala> val scaledData: DataFrame = scalerModel.transform(vecDF)
  225. scaledData: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 13 more fields]

  227. scala> // scaledData:DataFrame = [features: vector, scaledFeatures: vector]

  229. scala>

  231. scala> scaledData.select("features", "scaledFeatures").show
  232. 16/11/05 15:56:20 WARN Executor: 1 block locks were not released by TID = 13:
  233. [rdd_17_0]
  234. +--------------------+--------------------+
  235. | features| scaledFeatures|
  236. +--------------------+--------------------+
  237. |[0.0,37.0,10.0,3....|[-0.4413500298573...|
  238. |[0.0,27.0,4.0,4.0...|[-0.4413500298573...|
  239. |[0.0,32.0,15.0,1....|[-0.4413500298573...|
  240. |[0.0,57.0,15.0,5....|[-0.4413500298573...|
  241. |[0.0,22.0,0.75,2....|[-0.4413500298573...|
  242. |[0.0,32.0,1.5,2.0...|[-0.4413500298573...|
  243. |[0.0,22.0,0.75,2....|[-0.4413500298573...|
  244. |[0.0,57.0,15.0,2....|[-0.4413500298573...|
  245. |[0.0,32.0,15.0,4....|[-0.4413500298573...|
  246. |[0.0,22.0,1.5,4.0...|[-0.4413500298573...|
  247. |[0.0,37.0,15.0,2....|[-0.4413500298573...|
  248. |[0.0,27.0,4.0,4.0...|[-0.4413500298573...|
  249. |[0.0,47.0,15.0,5....|[-0.4413500298573...|
  250. |[0.0,22.0,1.5,2.0...|[-0.4413500298573...|
  251. |[0.0,27.0,4.0,4.0...|[-0.4413500298573...|
  252. |[0.0,37.0,15.0,1....|[-0.4413500298573...|
  253. |[0.0,37.0,15.0,2....|[-0.4413500298573...|
  254. |[0.0,22.0,0.75,3....|[-0.4413500298573...|
  255. |[0.0,22.0,1.5,2.0...|[-0.4413500298573...|
  256. |[0.0,27.0,10.0,2....|[-0.4413500298573...|
  257. +--------------------+--------------------+
  258. only showing top 20 rows


  261. scala>

  263. scala> //##########################

  265. scala> // 主成分

  267. scala> val pca = new PCA().setInputCol("scaledFeatures").setOutputCol("pcaFeatures").setK(3).fit(scaledData)
  268. 16/11/05 15:56:21 WARN Executor: 1 block locks were not released by TID = 14:
  269. [rdd_17_0]
  270. 16/11/05 15:56:22 WARN Executor: 1 block locks were not released by TID = 15:
  271. [rdd_17_0]
  272. 16/11/05 15:56:24 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
  273. 16/11/05 15:56:24 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
  274. 16/11/05 15:56:25 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
  275. 16/11/05 15:56:25 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
  276. pca: org.apache.spark.ml.feature.PCAModel = pca_8569d580d6e4

  278. scala> pca.explainedVariance.values //解释变量方差
  279. res11: Array[Double] = Array(0.28779526464781313, 0.23798543640278289, 0.11742828783633019)

  281. scala> pca.pc //载荷（观测变量与主成分的相关系数）
  282. res12: org.apache.spark.ml.linalg.DenseMatrix =
  283. -0.12034310848156521 0.05153952289637974 0.6678769450480689
  284. -0.42860623714516627 0.05417889891307473 -0.05592377098140197
  285. -0.44404074412877986 0.1926596811059294 -0.017025575192258197
  286. -0.12233707317255231 0.08053139375662526 -0.5093149296300096
  287. -0.14664751606128462 -0.3872166556211308 -0.03406819489501708
  288. -0.145543746024348 -0.43054860653839705 0.07841454709046872
  289. 0.17703994181974803 -0.12792784984216296 -0.5173229755329072
  290. 0.2459668445061567 0.4915809641798787 0.010477548320795945
  291. -0.2459668445061567 -0.4915809641798787 -0.010477548320795945
  292. -0.44420980045271047 0.240652448514566 -0.089356723885704
  293. 0.4442098004527103 -0.24065244851456588 0.08935672388570405

  295. scala> pca.extractParamMap()
  296. res13: org.apache.spark.ml.param.ParamMap =
  297. {
  298.     pca_8569d580d6e4-inputCol: scaledFeatures,
  299.     pca_8569d580d6e4-k: 3,
  300.     pca_8569d580d6e4-outputCol: pcaFeatures
  301. }

  303. scala> pca.params
  304. res14: Array[org.apache.spark.ml.param.Param[_]] = Array(pca_8569d580d6e4__inputCol, pca_8569d580d6e4__k, pca_8569d580d6e4__outputCol)

  306. scala>

  308. scala> val pcaDF: DataFrame = pca.transform(scaledData)
  309. pcaDF: org.apache.spark.sql.DataFrame = [affairs: double, age: double ... 14 more fields]

  311. scala> // pcaDF:DataFrame = [features: vector, scaledFeatures: vector,pcaFeatures: vector]

  313. scala> pcaDF.cache()
  314. res15: pcaDF.type = [affairs: double, age: double ... 14 more fields]

  316. scala>

  318. scala> pcaDF.printSchema()
  319. root
  320.  |-- affairs: double (nullable = true)
  321.  |-- age: double (nullable = true)
  322.  |-- yearsmarried: double (nullable = true)
  323.  |-- religiousness: double (nullable = true)
  324.  |-- education: double (nullable = true)
  325.  |-- occupation: double (nullable = true)
  326.  |-- rating: double (nullable = true)
  327.  |-- gender: string (nullable = true)
  328.  |-- children: string (nullable = true)
  329.  |-- genderIndex: double (nullable = true)
  330.  |-- genderVec: vector (nullable = true)
  331.  |-- childrenIndex: double (nullable = true)
  332.  |-- childrenVec: vector (nullable = true)
  333.  |-- features: vector (nullable = true)
  334.  |-- scaledFeatures: vector (nullable = true)
  335.  |-- pcaFeatures: vector (nullable = true)


  338. scala> pcaDF.select("features", "scaledFeatures", "pcaFeatures").show
  339. 16/11/05 15:56:36 WARN Executor: 1 block locks were not released by TID = 18:
  340. [rdd_64_0]
  341. +--------------------+--------------------+--------------------+
  342. | features           | scaledFeatures     | pcaFeatures        |
  343. +--------------------+--------------------+--------------------+
  344. |[0.0,37.0,10.0,3....|[-0.4413500298573...|[0.27828160409293...|
  345. |[0.0,27.0,4.0,4.0...|[-0.4413500298573...|[2.42147114101165...|
  346. |[0.0,32.0,15.0,1....|[-0.4413500298573...|[0.18301418047489...|
  347. |[0.0,57.0,15.0,5....|[-0.4413500298573...|[-2.9795960667914...|
  348. |[0.0,22.0,0.75,2....|[-0.4413500298573...|[1.79299133565688...|
  349. |[0.0,32.0,1.5,2.0...|[-0.4413500298573...|[2.65694237441759...|
  350. |[0.0,22.0,0.75,2....|[-0.4413500298573...|[3.48234503794570...|
  351. |[0.0,57.0,15.0,2....|[-0.4413500298573...|[-2.4215838062079...|
  352. |[0.0,32.0,15.0,4....|[-0.4413500298573...|[-0.6964555195741...|
  353. |[0.0,22.0,1.5,4.0...|[-0.4413500298573...|[2.18771069800414...|
  354. |[0.0,37.0,15.0,2....|[-0.4413500298573...|[-2.4259075891377...|
  355. |[0.0,27.0,4.0,4.0...|[-0.4413500298573...|[-0.7743038356008...|
  356. |[0.0,47.0,15.0,5....|[-0.4413500298573...|[-2.6176149267534...|
  357. |[0.0,22.0,1.5,2.0...|[-0.4413500298573...|[2.95788535193022...|
  358. |[0.0,27.0,4.0,4.0...|[-0.4413500298573...|[2.50146472861263...|
  359. |[0.0,37.0,15.0,1....|[-0.4413500298573...|[-0.5123817022008...|
  360. |[0.0,37.0,15.0,2....|[-0.4413500298573...|[-0.9191740114044...|
  361. |[0.0,22.0,0.75,3....|[-0.4413500298573...|[2.97391491782863...|
  362. |[0.0,22.0,1.5,2.0...|[-0.4413500298573...|[3.17940505267806...|
  363. |[0.0,27.0,10.0,2....|[-0.4413500298573...|[0.74585406839527...|
  364. +--------------------+--------------------+--------------------+
  365. only showing top 20 rows


  368. scala>

  370. scala> //#####################################

  372. scala>

  374. scala> // 注意最大迭代次數和轮廓系数

  376. scala> val KSSE = (2 to 10 by 1).par.toList.map { k =>
  377.      | // 聚类
  378.      | // Trains a k-means model.
  379.      | val kmeans = new KMeans().setK(k).setSeed(1L).setFeaturesCol("pcaFeatures")
  380.      | val model = kmeans.fit(pcaDF)
  381.      |
  382.      | // Evaluate clustering by computing Within Set Sum of Squared Errors.
  383.      | val WSSSE = model.computeCost(pcaDF)
  384.      |
  385.      | (k, WSSSE)
  386.      | }
  387. KSSE: List[(Int, Double)] = List((2,2876.20580405469), (3,1680.6647048004902), (4,1395.7184052948346), (5,1239.9362814229812), (6,999.2793106095127), (7,849.0071338527408), (8,737.8560221633246), (9,771.8211752483357), (10,655.7836351785677))

  389. scala>

  391. scala> KSSE.foreach(println)
  392. (2,2876.20580405469)
  393. (3,1680.6647048004902)
  394. (4,1395.7184052948346)
  395. (5,1239.9362814229812)
  396. (6,999.2793106095127)
  397. (7,849.0071338527408)
  398. (8,737.8560221633246)
  399. (9,771.8211752483357)
  400. (10,655.7836351785677)

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/29070860/viewspace-2127855/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/29070860/viewspace-2127855/

conghuanliu9724
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
OneHotEncoder介绍单属性多属性scala实现
pan_haufei的博客
06-07 4393
因为项目的需要，将数据库中表的属性向量化，然后进行机器学习，所以去spark官网学习了一下OneHotEncoder,官网的相关介绍比较少，主要是针对单属性的处理，但是项目的要求是多属性的处理，网上找了很多的资料，研究了大半天终于将它集成到了自己的项目之中，下面分享一下自己的学习心得，说的不好的地方，还请各位大神多多指教。       介绍：将类别映射为二进制向量，其中至多一个值为1（其余为零）
Spark2 oneHot编码--标准化--主成分--聚类
weixin_34034261的博客
11-03 201
1.导入包 import org.apache.spark.sql.SparkSession import org.apache.spark.sql.Dataset import org.apache.spark.sql.Row import org.apache.spark.sql.DataFrame import org.apache.spark.sql.Column impo...
【机器学习】Spark ML 对数据特征进行 One-Hot 编码
JIE的博客 --- moon_coder
05-13 1616
在机器学习中，一般需要对非数值型的特征进行编码处理，将其转化为数值型的特征。其中，One-Hot 编码是一种常见的特征编码方式。One-Hot 编码是将一个离散特征的每个取值映射为一个唯一的整数编号，并将该编号表示成一个二进制向量的形式。具体来说，对于一个有kkk个不同取值的离散特征，其 One-Hot 编码将会得到一个kkk维的向量，其中只有一个元素为1，其余元素为0，该元素的下标对应着特征取值的编号。例如，对于一个颜色特征，其可能的取值为红、绿、蓝三种，可以将其编码为：[1, 0, 0]
Spark MLlib特征处理：OneHotEncoder OneHot编码 ---原理及实战
热门推荐
王佩的CSDN博客
11-12 1万+
原理 1)String字符串转换成索引IndexDouble 2)索引转换成SparseVector 总结：OneHotEncoder = String => IndexDouble => SparseVector 代码实战 import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer} im
Spark -- OneHotEncoder，Spark的实现
TheBigBlue的博客
07-24 2641
  由于业务需求，需要对多离散列进行OneHotEncoder编码，并扩展为N列，比如有一列性别sex列，只有两种情况：男male，女female，则将这一列横向扩展为两列：sex_male，sex_female，并追加到原数据中，如果某一列特征项多于设定的最大特征，比如100，则不对该列编码。 首先是获取需要转换的离散列 val disColsList: List[String] = c...
数据处理-主成分分析-数据降维-pca聚类-pca
最新发布
09-04
主成分分析应用 1用主成分分析法求2018年1-12月小河闸断面主要污染物（主成分）和对应的主成分得分，下面列出详细计算过程。 2pca实现人脸检测 代码思路：先用PCA实现对人脸特征的提取并降维，再使用感知机实现多...
基于主成分分析和K-means聚类的平行坐标可视化技术研究
01-20
为了解决多维数据的维数过高、数据量过大带来的平行坐标可视化图形线条密集交叠以及数据规律特征不易获取的问题，提出基于主成分分析和K-means聚类的平行坐标（PCAKP,principal component analysis and k-means ...
在Python中使用K-Means聚类和PCA主成分分析进行图像压缩
09-17
主要介绍了在Python中使用K-Means聚类和PCA主成分分析进行图像压缩，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧
基于K-均值聚类的彩色图像质量评价及优化
04-30
针对彩色图像质量无法实时评价及优化的问题，提出了基于K-均值聚类的彩色图像质量评价及优化算法。首先利用K-均值聚类的方式构建样本数据集；然后通过待评价图像与聚类数据集之间的相似性来构建特征集；之后再将待...
无监督学习PCA降维处理和K-means聚类
01-20
2、主成分分析 应用PCA实现特征的降维 ·定义：高维数据转化为低维数据的过程，在此过程中可能会舍弃原有数据、创造新的变量 ·作用：是数据维散压缩，尽可能降低原数据的维数（复杂度），损失少量信息。 ·应用：...
python数据分析数据标准化及离散化详解
09-20
主要为大家详细介绍了python数据分析数据标准化及离散化，具有一定的参考价值，感兴趣的小伙伴们可以参考一下
Spark-MLlib 学习入门到掌握-OneHotEncoder独热编码[18]
qq_41610493的博客
06-22 994
OneHotEncoder(独热编码):采用01编码的一种算法，具体细节可百度。 优点：独热编码解决了分类器不好处理属性数据的问题，在一定程度上也起到了扩充特征的作用。它的值只有0和1，不同的类型存储在垂直的空间。 缺点：当类别的数量很多时，特征空间会变得非常大。在这种情况下，一般可以用PCA来减少维度。而且one hot encoding+PCA这种组合在实际中也非常有用。 def OneHotEncoderTest(){ val spark: SparkSession = S..
spark中one-hot编码的使用及dataframe.show()的含义
Luckin的博客
12-03 1050
这里的show只是为了方便展示，实际上代表的是只有0,1的超多维。假如将上述多列合并，那么有200827=200479+187+2+77+82。简单说就是show（）不会展示20万列只包含1个0的矩阵。 val assembler = new VectorAssembler().setInputCols(assemblerInputs).setOutputCol("features") ...
Spark ML 特征工程之 One-Hot Encoding
腾飞的大象
08-10 8470
1.什么是One-Hot Encoding 2.One-Hot Encoding在Spark中的应用 2.1 数据集预览 2.2 加载数据集 2.3 使用OneHotEncoder处理数据集 2.4 使用VectorAssembler合并所有特征为单个向量 2.5 以Pipeline的形式运行各个PipelineStage 2.6 训练和评估模型 1.什么是One-Ho...
One hot 编码
一碗竹叶青的博客
11-04 555
参考： https://my.oschina.net/kyo4321/blog/2050708 https://cloud.tencent.com/developer/article/1021585 对于one hot编码首先需要将数据数值化，然后把将数值进行onehot 所有的操作都是对dataframe来进行的，参看如何RDD转换dataframe 数值化 import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer, VectorAs
算法进阶--最大似然，赔率，Fuzzywuzzy库，主成分分析，onehot
WslWslYYX的博客
10-17 246
算法进阶--第一天故地重游之最大似然估计赌徒之赔率Fuzzywuzzy库-Levenshtein distance 故地重游之最大似然估计 首先从贝叶斯公式开始： 给定某些样本D，在这些样本中计算某结论A1,A2…An出现的概率，即P(Ai|D),若求maxP(Ai|D)，则有以下式子： – 当样本给定时，P(D)是常数，可以忽略 maxP(Ai∣D)=maxP(D∣Ai)P(Ai)P(D)=max(P(D∣Ai)P(Ai))maxP(A_i|D)=max\frac{P(D|A_i)P(A_i)}{P(D)
Scala编码规范与最佳实践-编码风格
wang2leee的博客
02-22 345
编码风格 尽可能直接在函数定义的地方使用模式匹配。例如，在下面的写法中，match应该被折叠起来(collapse): list map { item => item match { case Some(x) => x case None => default } } 用下面的写法替代： list map { case Some(x) => x case None =&gt
Scala编码规范与最佳实践-编码模式
wang2leee的博客
02-22 195
编码模式 Loan Pattern: 确保打开的资源（如文件、数据库连接）能够在操作完毕后被安全的释放。 Loan Pattern的通用格式如下： def using[A](r : Resource)(f : Resource => A) : A = try { f(r) } finally { r.dispose() } 这个格式针对Resource类型进行操作。还有一种做法是：只要实现了close方法，都可以运用Loan Pattern： d
one-hot之后进行主成分分析
04-01
对于经过one-hot编码的数据进行主成分分析（PCA），需要注意以下几点： 1. 对于二元变量（只有0和1两种取值），不需要进行标准化处理，直接进行PCA即可。 2. 对于多元变量，需要先进行标准化处理。因为PCA是基于...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值