Spark ML 函数feature.VectorIndexer及VectorAssembler

assemble
英 [əˈsembl] 美 [əˈsembl]
v.
聚集;集合;收集;装配;组装

VectorAssembler

VectorAssembler(inputCols=None, outputCol=None, handleInvalid=‘error’)
特征转换器，将多个列合并为一个向量列。
VectorAssembler接受以下输入列类型:所有数值类型、布尔类型和向量类型。在每一行中，输入列的值将按照指定的顺序连接到一个向量中。
demo

dataset = spark.createDataFrame(
        [(0, 18, 1.0, Vectors.sparse(4,[1,3],[122,344]), 1.0,False,True)],
        ["id", "hour", "mobile", "userFeatures", "clicked",'flag','flag1'])

assembler = VectorAssembler(
        inputCols=["hour", "mobile", "userFeatures",'flag','flag1'],
        outputCol="features")
assembler.transform(dataset).show(truncate=False)

±–±—±-----±----------------------±------±----±----±-------------------------------------+
|id |hour|mobile|userFeatures |clicked|flag |flag1|features |
±–±—±-----±----------------------±------±----±----±-------------------------------------+
|0 |18 |1.0 |(4,[1,3],[122.0,344.0])|1.0 |false|true |[18.0,1.0,0.0,122.0,0.0,344.0,0.0,1.0]|
±–±—±-----±----------------------±------±----±----±-------------------------------------+

VectorIndexer

VectorIndexer(maxCategories=20, inputCol=None, outputCol=None, handleInvalid=‘error’)
demo

from pyspark.ml.linalg import Vectors

df = spark.createDataFrame([
   (Vectors.dense([1, 100,234,12]),),
   (Vectors.dense([123, 1.0,124,12]),),
   (Vectors.dense([13, 4.0,1234,12]),),
   (Vectors.dense([13, 3.0,-300,12]),),
   (Vectors.dense([123, 300,34,12]),)], 
   ["a"])
indexer = VectorIndexer(maxCategories=4, inputCol="a", outputCol="indexed")
model = indexer.fit(df)
model.transform(df).show(truncate=False)

±----------------------±--------------------+
|a |indexed |
±----------------------±--------------------+
|[1.0,100.0,234.0,12.0] |[0.0,100.0,234.0,0.0]|
|[123.0,1.0,124.0,12.0] |[2.0,1.0,124.0,0.0] |
|[13.0,4.0,1234.0,12.0] |[1.0,4.0,1234.0,0.0] |
|[13.0,3.0,-300.0,12.0] |[1.0,3.0,-300.0,0.0] |
|[123.0,300.0,34.0,12.0]|[2.0,300.0,34.0,0.0] |
±----------------------±--------------------+

修改maxCategories=5

±----------------------±----------------+
|a |indexed |
±----------------------±----------------+
|[1.0,100.0,234.0,12.0] |[0.0,3.0,3.0,0.0]|
|[123.0,1.0,124.0,12.0] |[2.0,0.0,2.0,0.0]|
|[13.0,4.0,1234.0,12.0] |[1.0,2.0,4.0,0.0]|
|[13.0,3.0,-300.0,12.0] |[1.0,1.0,0.0,0.0]|
|[123.0,300.0,34.0,12.0]|[2.0,4.0,1.0,0.0]|
±----------------------±----------------+

说明：
我们设置maxCategories为4，即只有种类小于4的特征才被认为是类别型特征，否则被认为是连续型特征。
其中类别型特征将被进行编号索引，为了索引的稳定性，规定如果这个特征值为0，则一定会被编号成0，这样可以保证向量的稀疏度。

• 0一定被索引为0
  

更多参考
pyspark.ml.feature模块详解（持续更新中）