pyspark- ml-features

最新推荐文章于 2024-08-18 09:40:17 发布
最新推荐文章于 2024-08-18 09:40:17 发布
阅读量4.3k 收藏 5
点赞数
分类专栏： spark 文章标签： spark

参考地址：

1、http://spark.apache.org/docs/latest/ml-guide.html

2、https://github.com/apache/spark/tree/v2.2.0

3、http://spark.apache.org/docs/latest/ml-features.html



Extracting, transforming and selecting features

Feature Extractors

TF-IDF

from pyspark.ml.feature import HashingTF, IDF, Tokenizer

sentenceData = spark.createDataFrame([
    (0.0, "Hi I heard about Spark"),
    (0.0, "I wish Java could use case classes"),
    (1.0, "Logistic regression models are neat")
], ["label", "sentence"])

tokenizer = Tokenizer(inputCol="sentence", outputCol="words")
wordsData = tokenizer.transform(sentenceData)

hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=20)
featurizedData = hashingTF.transform(wordsData)
# alternatively, CountVectorizer can also be used to get term frequency vectors

idf = IDF(inputCol="rawFeatures", outputCol="features")
idfModel = idf.fit(featurizedData)
rescaledData = idfModel.transform(featurizedData)

rescaledData.select("label", "features").show()

Word2Vec

from pyspark.ml.feature import Word2Vec

# Input data: Each row is a bag of words from a sentence or document.
documentDF = spark.createDataFrame([
    ("Hi I heard about Spark".split(" "), ),
    ("I wish Java could use case classes".split(" "), ),
    ("Logistic regression models are neat".split(" "), )
], ["text"])

# Learn a mapping from words to Vectors.
word2Vec = Word2Vec(vectorSize=3, minCount=0, inputCol="text", outputCol="result")
model = word2Vec.fit(documentDF)

result = model.transform(documentDF)
for row in result.collect():
    text, vector = row
    print("Text: [%s] => \nVector: %s\n" % (", ".join(text), str(vector)))

CountVectorizer

from pyspark.ml.feature import CountVectorizer

# Input data: Each row is a bag of words with a ID.
df = spark.createDataFrame([
    (0, "a b c".split(" ")),
    (1, "a b b c a".split(" "))
], ["id", "words"])

# fit a CountVectorizerModel from the corpus.
cv = CountVectorizer(inputCol="words", outputCol="features", vocabSize=3, minDF=2.0)

model = cv.fit(df)

result = model.transform(df)
result.show(truncate=False)

Feature Transformers

Tokenizer

from pyspark.ml.feature import Tokenizer, RegexTokenizer
from pyspark.sql.functions import col, udf
from pyspark.sql.types import IntegerType

sentenceDataFrame = spark.createDataFrame([
    (0, "Hi I heard about Spark"),
    (1, "I wish Java could use case classes"),
    (2, "Logistic,regression,models,are,neat")
], ["id", "sentence"])

tokenizer = Tokenizer(inputCol="sentence", outputCol="words")

regexTokenizer = RegexTokenizer(inputCol="sentence", outputCol="words", pattern="\\W")
# alternatively, pattern="\\w+", gaps(False)

countTokens = udf(lambda words: len(words), IntegerType())

tokenized = tokenizer.transform(sentenceDataFrame)
tokenized.select("sentence", "words")\
    .withColumn("tokens", countTokens(col("words"))).show(truncate=False)

regexTokenized = regexTokenizer.transform(sentenceDataFrame)
regexTokenized.select("sentence", "words") \
    .withColumn("tokens", countTokens(col("words"))).show(truncate=False)

StopWordsRemover

from pyspark.ml.feature import StopWordsRemover

sentenceData = spark.createDataFrame([
    (0, ["I", "saw", "the", "red", "balloon"]),
    (1, ["Mary", "had", "a", "little", "lamb"])
], ["id", "raw"])

remover = StopWordsRemover(inputCol="raw", outputCol="filtered")
remover.transform(sentenceData).show(truncate=False)


n-gram

from pyspark.ml.feature import NGram

wordDataFrame = spark.createDataFrame([
    (0, ["Hi", "I", "heard", "about", "Spark"]),
    (1, ["I", "wish", "Java", "could", "use", "case", "classes"]),
    (2, ["Logistic", "regression", "models", "are", "neat"])
], ["id", "words"])

ngram = NGram(n=2, inputCol="words", outputCol="ngrams")

ngramDataFrame = ngram.transform(wordDataFrame)
ngramDataFrame.select("ngrams").show(truncate=False)


Binarizer

from pyspark.ml.feature import Binarizer

continuousDataFrame = spark.createDataFrame([
    (0, 0.1),
    (1, 0.8),
    (2, 0.2)
], ["id", "feature"])

binarizer = Binarizer(threshold=0.5, inputCol="feature", outputCol="binarized_feature")

binarizedDataFrame = binarizer.transform(continuousDataFrame)

print("Binarizer output with Threshold = %f" % binarizer.getThreshold())
binarizedDataFrame.show()


PCA

from pyspark.ml.feature import PCA
from pyspark.ml.linalg import Vectors

data = [(Vectors.sparse(5, [(1, 1.0), (3, 7.0)]),),
        (Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),
        (Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)]
df = spark.createDataFrame(data, ["features"])

pca = PCA(k=3, inputCol="features", outputCol="pcaFeatures")
model = pca.fit(df)

result = model.transform(df).select("pcaFeatures")
result.show(truncate=False)


PolynomialExpansion

from pyspark.ml.feature import PolynomialExpansion
from pyspark.ml.linalg import Vectors

df = spark.createDataFrame([
    (Vectors.dense([2.0, 1.0]),),
    (Vectors.dense([0.0, 0.0]),),
    (Vectors.dense([3.0, -1.0]),)
], ["features"])

polyExpansion = PolynomialExpansion(degree=3, inputCol="features", outputCol="polyFeatures")
polyDF = polyExpansion.transform(df)

polyDF.show(truncate=False)


Discrete Cosine Transform (DCT)

from pyspark.ml.feature import DCT
from pyspark.ml.linalg import Vectors

df = spark.createDataFrame([
    (Vectors.dense([0.0, 1.0, -2.0, 3.0]),),
    (Vectors.dense([-1.0, 2.0, 4.0, -7.0]),),
    (Vectors.dense([14.0, -2.0, -5.0, 1.0]),)], ["features"])

dct = DCT(inverse=False, inputCol="features", outputCol="featuresDCT")

dctDf = dct.transform(df)

dctDf.select("featuresDCT").show(truncate=False)


StringIndexer

from pyspark.ml.feature import StringIndexer

df = spark.createDataFrame(
    [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
    ["id", "category"])

indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
indexed = indexer.fit(df).transform(df)
indexed.show()


IndexToString

from pyspark.ml.feature import IndexToString, StringIndexer

df = spark.createDataFrame(
    [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
    ["id", "category"])

indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
model = indexer.fit(df)
indexed = model.transform(df)

print("Transformed string column '%s' to indexed column '%s'"
      % (indexer.getInputCol(), indexer.getOutputCol()))
indexed.show()

print("StringIndexer will store labels in output column metadata\n")

converter = IndexToString(inputCol="categoryIndex", outputCol="originalCategory")
converted = converter.transform(indexed)

print("Transformed indexed column '%s' back to original string column '%s' using "
      "labels in metadata" % (converter.getInputCol(), converter.getOutputCol()))
converted.select("id", "categoryIndex", "originalCategory").show()


OneHotEncoder

from pyspark.ml.feature import OneHotEncoder, StringIndexer

df = spark.createDataFrame([
    (0, "a"),
    (1, "b"),
    (2, "c"),
    (3, "a"),
    (4, "a"),
    (5, "c")
], ["id", "category"])

stringIndexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
model = stringIndexer.fit(df)
indexed = model.transform(df)

encoder = OneHotEncoder(inputCol="categoryIndex", outputCol="categoryVec")
encoded = encoder.transform(indexed)
encoded.show()


VectorIndexer

from pyspark.ml.feature import VectorIndexer

data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

indexer = VectorIndexer(inputCol="features", outputCol="indexed", maxCategories=10)
indexerModel = indexer.fit(data)

categoricalFeatures = indexerModel.categoryMaps
print("Chose %d categorical features: %s" %
      (len(categoricalFeatures), ", ".join(str(k) for k in categoricalFeatures.keys())))

# Create new column "indexed" with categorical values transformed to indices
indexedData = indexerModel.transform(data)
indexedData.show()


Interaction

Normalizer

from pyspark.ml.feature import Normalizer
from pyspark.ml.linalg import Vectors

dataFrame = spark.createDataFrame([
    (0, Vectors.dense([1.0, 0.5, -1.0]),),
    (1, Vectors.dense([2.0, 1.0, 1.0]),),
    (2, Vectors.dense([4.0, 10.0, 2.0]),)
], ["id", "features"])

# Normalize each Vector using $L^1$ norm.
normalizer = Normalizer(inputCol="features", outputCol="normFeatures", p=1.0)
l1NormData = normalizer.transform(dataFrame)
print("Normalized using L^1 norm")
l1NormData.show()

# Normalize each Vector using $L^\infty$ norm.
lInfNormData = normalizer.transform(dataFrame, {normalizer.p: float("inf")})
print("Normalized using L^inf norm")
lInfNormData.show()


StandardScaler

from pyspark.ml.feature import StandardScaler

dataFrame = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")
scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures",
                        withStd=True, withMean=False)

# Compute summary statistics by fitting the StandardScaler
scalerModel = scaler.fit(dataFrame)

# Normalize each feature to have unit standard deviation.
scaledData = scalerModel.transform(dataFrame)
scaledData.show()


MinMaxScaler

from pyspark.ml.feature import MinMaxScaler
from pyspark.ml.linalg import Vectors

dataFrame = spark.createDataFrame([
    (0, Vectors.dense([1.0, 0.1, -1.0]),),
    (1, Vectors.dense([2.0, 1.1, 1.0]),),
    (2, Vectors.dense([3.0, 10.1, 3.0]),)
], ["id", "features"])

scaler = MinMaxScaler(inputCol="features", outputCol="scaledFeatures")

# Compute summary statistics and generate MinMaxScalerModel
scalerModel = scaler.fit(dataFrame)

# rescale each feature to range [min, max].
scaledData = scalerModel.transform(dataFrame)
print("Features scaled to range: [%f, %f]" % (scaler.getMin(), scaler.getMax()))
scaledData.select("features", "scaledFeatures").show()


MaxAbsScaler

from pyspark.ml.feature import MaxAbsScaler
from pyspark.ml.linalg import Vectors

dataFrame = spark.createDataFrame([
    (0, Vectors.dense([1.0, 0.1, -8.0]),),
    (1, Vectors.dense([2.0, 1.0, -4.0]),),
    (2, Vectors.dense([4.0, 10.0, 8.0]),)
], ["id", "features"])

scaler = MaxAbsScaler(inputCol="features", outputCol="scaledFeatures")

# Compute summary statistics and generate MaxAbsScalerModel
scalerModel = scaler.fit(dataFrame)

# rescale each feature to range [-1, 1].
scaledData = scalerModel.transform(dataFrame)

scaledData.select("features", "scaledFeatures").show()


Bucketizer

from pyspark.ml.feature import Bucketizer

splits = [-float("inf"), -0.5, 0.0, 0.5, float("inf")]

data = [(-999.9,), (-0.5,), (-0.3,), (0.0,), (0.2,), (999.9,)]
dataFrame = spark.createDataFrame(data, ["features"])

bucketizer = Bucketizer(splits=splits, inputCol="features", outputCol="bucketedFeatures")

# Transform original data into its bucket index.
bucketedData = bucketizer.transform(dataFrame)

print("Bucketizer output with %d buckets" % (len(bucketizer.getSplits())-1))
bucketedData.show()


ElementwiseProduct

from pyspark.ml.feature import ElementwiseProduct
from pyspark.ml.linalg import Vectors

# Create some vector data; also works for sparse vectors
data = [(Vectors.dense([1.0, 2.0, 3.0]),), (Vectors.dense([4.0, 5.0, 6.0]),)]
df = spark.createDataFrame(data, ["vector"])
transformer = ElementwiseProduct(scalingVec=Vectors.dense([0.0, 1.0, 2.0]),
                                 inputCol="vector", outputCol="transformedVector")
# Batch transform the vectors to create new column:
transformer.transform(df).show()


SQLTransformer

from pyspark.ml.feature import SQLTransformer

df = spark.createDataFrame([
    (0, 1.0, 3.0),
    (2, 2.0, 5.0)
], ["id", "v1", "v2"])
sqlTrans = SQLTransformer(
    statement="SELECT *, (v1 + v2) AS v3, (v1 * v2) AS v4 FROM __THIS__")
sqlTrans.transform(df).show()

VectorAssembler

from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler

dataset = spark.createDataFrame(
    [(0, 18, 1.0, Vectors.dense([0.0, 10.0, 0.5]), 1.0)],
    ["id", "hour", "mobile", "userFeatures", "clicked"])

assembler = VectorAssembler(
    inputCols=["hour", "mobile", "userFeatures"],
    outputCol="features")

output = assembler.transform(dataset)
print("Assembled columns 'hour', 'mobile', 'userFeatures' to vector column 'features'")
output.select("features", "clicked").show(truncate=False)


QuantileDiscretizer

from pyspark.ml.feature import QuantileDiscretizer

data = [(0, 18.0), (1, 19.0), (2, 8.0), (3, 5.0), (4, 2.2)]
df = spark.createDataFrame(data, ["id", "hour"])

discretizer = QuantileDiscretizer(numBuckets=3, inputCol="hour", outputCol="result")

result = discretizer.fit(df).transform(df)
result.show()


Imputer

from pyspark.ml.feature import Imputer

df = spark.createDataFrame([
    (1.0, float("nan")),
    (2.0, float("nan")),
    (float("nan"), 3.0),
    (4.0, 4.0),
    (5.0, 5.0)
], ["a", "b"])

imputer = Imputer(inputCols=["a", "b"], outputCols=["out_a", "out_b"])
model = imputer.fit(df)

model.transform(df).show()


Feature Selectors

VectorSlicer

from pyspark.ml.feature import VectorSlicer
from pyspark.ml.linalg import Vectors
from pyspark.sql.types import Row

df = spark.createDataFrame([
    Row(userFeatures=Vectors.sparse(3, {0: -2.0, 1: 2.3})),
    Row(userFeatures=Vectors.dense([-2.0, 2.3, 0.0]))])

slicer = VectorSlicer(inputCol="userFeatures", outputCol="features", indices=[1])

output = slicer.transform(df)

output.select("userFeatures", "features").show()


RFormula

from pyspark.ml.feature import RFormula

dataset = spark.createDataFrame(
    [(7, "US", 18, 1.0),
     (8, "CA", 12, 0.0),
     (9, "NZ", 15, 0.0)],
    ["id", "country", "hour", "clicked"])

formula = RFormula(
    formula="clicked ~ country + hour",
    featuresCol="features",
    labelCol="label")

output = formula.fit(dataset).transform(dataset)
output.select("features", "label").show()


ChiSqSelector

from pyspark.ml.feature import ChiSqSelector
from pyspark.ml.linalg import Vectors

df = spark.createDataFrame([
    (7, Vectors.dense([0.0, 0.0, 18.0, 1.0]), 1.0,),
    (8, Vectors.dense([0.0, 1.0, 12.0, 0.0]), 0.0,),
    (9, Vectors.dense([1.0, 0.0, 15.0, 0.1]), 0.0,)], ["id", "features", "clicked"])

selector = ChiSqSelector(numTopFeatures=1, featuresCol="features",
                         outputCol="selectedFeatures", labelCol="clicked")

result = selector.fit(df).transform(df)

print("ChiSqSelector output with top %d features selected" % selector.getNumTopFeatures())
result.show()


LSH Algorithms

Bucketed Random Projection for Euclidean Distance

from pyspark.ml.feature import BucketedRandomProjectionLSH
from pyspark.ml.linalg import Vectors
from pyspark.sql.functions import col

dataA = [(0, Vectors.dense([1.0, 1.0]),),
         (1, Vectors.dense([1.0, -1.0]),),
         (2, Vectors.dense([-1.0, -1.0]),),
         (3, Vectors.dense([-1.0, 1.0]),)]
dfA = spark.createDataFrame(dataA, ["id", "features"])

dataB = [(4, Vectors.dense([1.0, 0.0]),),
         (5, Vectors.dense([-1.0, 0.0]),),
         (6, Vectors.dense([0.0, 1.0]),),
         (7, Vectors.dense([0.0, -1.0]),)]
dfB = spark.createDataFrame(dataB, ["id", "features"])

key = Vectors.dense([1.0, 0.0])

brp = BucketedRandomProjectionLSH(inputCol="features", outputCol="hashes", bucketLength=2.0,
                                  numHashTables=3)
model = brp.fit(dfA)

# Feature Transformation
print("The hashed dataset where hashed values are stored in the column 'hashes':")
model.transform(dfA).show()

# Compute the locality sensitive hashes for the input rows, then perform approximate
# similarity join.
# We could avoid computing hashes by passing in the already-transformed dataset, e.g.
# `model.approxSimilarityJoin(transformedA, transformedB, 1.5)`
print("Approximately joining dfA and dfB on Euclidean distance smaller than 1.5:")
model.approxSimilarityJoin(dfA, dfB, 1.5, distCol="EuclideanDistance")\
    .select(col("datasetA.id").alias("idA"),
            col("datasetB.id").alias("idB"),
            col("EuclideanDistance")).show()

# Compute the locality sensitive hashes for the input rows, then perform approximate nearest
# neighbor search.
# We could avoid computing hashes by passing in the already-transformed dataset, e.g.
# `model.approxNearestNeighbors(transformedA, key, 2)`
print("Approximately searching dfA for 2 nearest neighbors of the key:")
model.approxNearestNeighbors(dfA, key, 2).show()


MinHash for Jaccard Distance

from pyspark.ml.feature import MinHashLSH
from pyspark.ml.linalg import Vectors
from pyspark.sql.functions import col

dataA = [(0, Vectors.sparse(6, [0, 1, 2], [1.0, 1.0, 1.0]),),
         (1, Vectors.sparse(6, [2, 3, 4], [1.0, 1.0, 1.0]),),
         (2, Vectors.sparse(6, [0, 2, 4], [1.0, 1.0, 1.0]),)]
dfA = spark.createDataFrame(dataA, ["id", "features"])

dataB = [(3, Vectors.sparse(6, [1, 3, 5], [1.0, 1.0, 1.0]),),
         (4, Vectors.sparse(6, [2, 3, 5], [1.0, 1.0, 1.0]),),
         (5, Vectors.sparse(6, [1, 2, 4], [1.0, 1.0, 1.0]),)]
dfB = spark.createDataFrame(dataB, ["id", "features"])

key = Vectors.sparse(6, [1, 3], [1.0, 1.0])

mh = MinHashLSH(inputCol="features", outputCol="hashes", numHashTables=5)
model = mh.fit(dfA)

# Feature Transformation
print("The hashed dataset where hashed values are stored in the column 'hashes':")
model.transform(dfA).show()

# Compute the locality sensitive hashes for the input rows, then perform approximate
# similarity join.
# We could avoid computing hashes by passing in the already-transformed dataset, e.g.
# `model.approxSimilarityJoin(transformedA, transformedB, 0.6)`
print("Approximately joining dfA and dfB on distance smaller than 0.6:")
model.approxSimilarityJoin(dfA, dfB, 0.6, distCol="JaccardDistance")\
    .select(col("datasetA.id").alias("idA"),
            col("datasetB.id").alias("idB"),
            col("JaccardDistance")).show()

# Compute the locality sensitive hashes for the input rows, then perform approximate nearest
# neighbor search.
# We could avoid computing hashes by passing in the already-transformed dataset, e.g.
# `model.approxNearestNeighbors(transformedA, key, 2)`
# It may return less than 2 rows when not enough approximate near-neighbor candidates are
# found.
print("Approximately searching dfA for 2 nearest neighbors of the key:")
model.approxNearestNeighbors(dfA, key, 2).show()
Find full example code at "examples/src/main/python/ml/min_hash_lsh_example.py" in the Spark repo.


风吴痕
关注
专栏目录
pyspark-decision-tree：2.4.6的apsendendo o funcionamento daárvorededecisão
02-12
在PySpark中，我们主要通过`pyspark.ml.classification`模块的`DecisionTreeClassifier`类来实现决策树模型。它基于Gini不纯度或熵作为分裂标准，通过递归地分割数据集构建树状结构，以最小化预测误差。 在开始之前...
pyspark.ml.feature模块详解（持续更新中）
花木兰
08-29 2809
Tokenizer(inputCol=None,outputCol=None) 分词器，把字符串转为小写，并以空格分词 eg: df=spark.createDataFrame([('''Machine learning can be applied to a wide variety of data types, such as vectors, text, images, an...
Pyspark:特征处理(ml.feature包的使用)
读万卷书 行万里路
10-15 1903
ml.feature于分类变量映射有关的类主要有：VectorIndexer、StringIndexer和IndexToString类。ml.feature包中常用归一化的类主要有：MaxAbsScaler、MinMaxScaler、StandardScaler、RobustScaler和Normalizer。除了以上介绍的类之外，ml.feature包中还有其他方法。结合以上对OneHotEncoder的转化结果进行说明：字段中的最大值即为转后的向量的维度，注意不是字段中不同值的总数；
Spark MLlib 特征工程系列—特征提取LSH(BucketedRandomProjectionLSH)
最新发布
2401_84052244的博客
08-18 399
是 Spark 中 Locality Sensitive Hashing (LSH) 的一种实现，专门用于基于欧几里得距离的近似最近邻搜索。它通过随机投影将高维数据映射到低维空间，并利用哈希桶对数据进行分组，确保相似的数据点在哈希空间中的距离更近，从而实现快速的相似度搜索。在大数据环境中，直接计算每个数据点与其他所有点的距离代价极高，而能够有效降低计算复杂度，是处理大规模高维数据的常用工具。是 Spark 中处理高维数据相似度搜索的有效工具。
Pyspark特征工程--Bucketizer
weixin_46408961的博客
03-08 850
Bucketizer：一列连续特性映射到列桶的特性 class pyspark.ml.feature.Bucketizer(splits=None, inputCol=None, outputCol=None, handleInvalid=‘error’) setHandleInvalid(value)：设置handleInvalid的值 handleInvalid = Param(parent=‘undefined’, name=‘handleInvalid’, doc=“how to handle in
pyspark官方文档中的pyspark.ml.feature函数中文简介
YW_Vine的博客
04-28 7368
可以看成对pyspark.ml.feature中的方法的大致中文翻译吧，例子基本都是给予官方文档上的例子，目的就是为了对里面的函数有大致的了解，如果要是看具体的还是看官方文档好，待整理待更新 ，有点乱。 官方文档地址： http://spark.apache.org/docs/latest/api/python/pyspark.ml.html 1、Binarizer(threshold=...
Sentiment-Analysis-Pyspark
04-29
from pyspark.ml.feature import CountVectorizer vectorizer = CountVectorizer(inputCol="value", outputCol="features") vectors = vectorizer.fit(data).transform(data) ``` 接着，使用fit方法训练模型，如...
Windows环境下SparkML程序简单示例
12-14
from pyspark.ml.evaluation import BinaryClassificationEvaluator from pyspark.ml.feature import VectorAssembler ``` 然后，你需要准备训练数据。Spark MLlib支持多种数据格式，如LibSVM、CSV、JSON等。假设你...
k_means&random_forest在Spark框架下的实现
07-05
from pyspark.ml.classification import RandomForestClassifier from pyspark.ml.feature import VectorAssembler from pyspark.sql import SparkSession # 假设我们有DataFrame 'df'，包含特征列'features'和标签...
sample_spark3:如何使用findspark和pyspark使用spark3
03-30
`pyspark`还提供了机器学习库`ml`，可以构建复杂的机器学习模型。例如，你可以用`Pipeline`来组合多个步骤，实现特征工程和模型训练： ```python from pyspark.ml import Pipeline from pyspark.ml.feature import ...
pyspark.ml.feature特征工程常用方法（一）
Black eyes的博客
01-01 2053
本篇博文主要是对pyspark.ml.feature模块的函数进行介绍，也可以直接看官网文档。其中博文的数据皆来自官方文档中例子。官方文档地址：http://spark.apache.org/docs/latest/api/python/pyspark.ml.html pyspark.ml.feature 函数概括： __all__ = ['Binarizer', 'Bucketizer',...
Spark机器学习之特征提取、选择、转换
cheng9981的博客
03-18 1万+
本节介绍了处理特征的算法，大致分为以下几组：      1、提取：从“原始”数据提取特征      2、转换：缩放，转换或修改要素      3、选择：从一组较大的要素中选择一个子集      4、局部敏感哈希（LSH）：这类算法将特征变换的方面与其他算法相结合。 1、特征提取 1.1 TF-IDF（term frequency–inverse document frequen
从0开始学pyspark（七）：pyspark-ML 部分之 转换器（Transformer）学习
泛泛之素
10-16 5810
pyspark.ml.feature.Binarizer(self, threshold=0.0, inputCol=None, outputCol=None) 根据指定的阈值将连续变量转换为对应的二进制 # 创建session from pyspark.sql import SparkSession spark = SparkSession.builder.appName('learn_ml'...
pyspark及Spark报错问题汇总及某些函数用法。
热门推荐
元元的李树专栏
01-07 3万+
此贴，主要记录本人在工作中遇到的某些报错问题，并提出自己的解决办法。 1. spark = SparkSession.builder() TypeError: 'Builder' object is not callable 解决办法：.builder() 改为.builderhttps://issues.apache.org/jira/browse/SPARK-1842...
pyspark.ml特征变换模块
公众号：瑞行AI
10-15 2528
pyspark.ml 2.1 ml模块的算子是基于dataframe构建的： （1）ML Pipeline APIs 快速构建ML pipeline的API （2）pyspark.ml.param module （3）pyspark.ml.feature module （4）pyspark.ml.classification module （5）pyspark.ml.clustering modu...
Pyspark特征工程--Normalizer
weixin_46408961的博客
03-09 395
Normalizer归一化 class pyspark.ml.feature.Normalizer(p=2.0, inputCol=None, outputCol=None) 使用给定的 p 范数将向量形式化为具有单位范数 1-范数：║x║1=│x1│+│x2│+…+│xn│ 2-范数： ∣∣x∣∣2=(∣∣x1∣∣2+∣∣x2∣∣2+...+∣∣xn∣∣2)(1/2) ||x||^2 = (||x1||^2+||x2||^2+...+||xn||^2)^(1/2) ∣∣x∣∣2=(∣∣x1∣∣2+∣∣x2
PySpark ML（转换器）
二哥为啥不像程序员？
11-16 1619
PySpark ML（转换器） 在PySpark中包含了两种机器学习相关的包：MLlib和ML，二者的主要区别在于MLlib包的操作是基于RDD的，ML包的操作是基于DataFrame的。根据之前我们叙述过的DataFrame的性能要远远好于RDD，并且MLlib已经不再被维护了，所以在本专栏中我们将不会讲解MLlib。 ML简介 在ML包中主要包含了三个主要的抽象类：转换器、评估器、管道，本文先来介绍第一种抽象类——转换器。 转换器 在PySpark中，我们通常通过将一个新列附加到DataFrame来转换
Pyspark特征工程--IndexToString
weixin_46408961的博客
03-09 418
IndexToString class pyspark.ml.feature.IndexToString(inputCol=None, outputCol=None, labels=None) 将一列索引映射回对应字符串值的新列的转换器。 索引字符串映射要么来自输入列的 ML 属性，要么来自用户提供的标签（优先于 ML 属性）。 请参阅 StringIndexer 将字符串转换为索引。 01.初始化： from pyspark.sql import SparkSession spark = SparkSes
pyspark 中k-means聚类BSSSE调用函数
05-31
可以使用 pyspark.ml.stat 类中的 chiSquareTest() 函数来计算簇间平方和（BSSSE）。 下面是一个使用 chiSquareTest() 函数计算簇间平方和的示例： ``` python from pyspark.ml.clustering import KMeans from ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值