spark MLlib的 pipeline方式

spark MLlib的 pipeline方式

spark mllib的pipeline，是指将多个机器学习的算法串联到一个工作链中，依次执行各种算法。

在Pipeline中的每个算法被称为“PipelineStage”，表示其中的一个算法。PipelineStage分为两种类型， Estimator和Transformer， 其中 ：

• Transformer将数据转换为两一种形式（例如修改格式），以供后续的Estimator使用，统一的转换函数transform；
• Estimator是由数据得到一个Mode（Mode也是继承于Transformer），有统一触发的函数fit。
  

然后一个“综合”的算法就可以通过pipeline封装起来。这样做的好处是可以很方便的替换算法。例如，我们在应用中往往只是笼统的期望一个“分类”、”拟合“这样的功能，但不知道是用分类或拟合的那个算法效果是最好的，有了这种pipeline机制后，很方便替换各种分类和拟合算法，从而得到最好的效果。

详情：  https://spark.apache.org/docs/latest/ml-guide.html 

/**
 * :: Experimental ::
 * A simple pipeline, which acts as an estimator. A Pipeline consists of a sequence of stages, each
 * of which is either an [[Estimator]] or a [[Transformer]]. When [[Pipeline#fit]] is called, the
 * stages are executed in order. If a stage is an [[Estimator]], its [[Estimator#fit]] method will
 * be called on the input dataset to fit a model. Then the model, which is a transformer, will be
 * used to transform the dataset as the input to the next stage. If a stage is a [[Transformer]],
 * its [[Transformer#transform]] method will be called to produce the dataset for the next stage.
 * The fitted model from a [[Pipeline]] is an [[PipelineModel]], which consists of fitted models and
 * transformers, corresponding to the pipeline stages. If there are no stages, the pipeline acts as
 * an identity transformer.
 */
@Experimental
class Pipeline(override val uid: String) extends Estimator[PipelineModel] {

From WizNote

posted on 2015-07-09 17:24 过雁 阅读( ...) 评论( ...) 编辑 收藏

转载于:https://www.cnblogs.com/zwCHAN/p/4633753.html