pyspark机器学习_机器学习：使用Pyspark进行线性回归

pyspark机器学习

Introduction:

简介 ：

PySpark is the Python API written in python to support Apache Spark. Apache Spark is a distributed framework that can handle Big Data analysis. Spark is written in Scala and can be integrated with Python, Scala, Java, R, SQL languages. Spark is basically a computational engine, that works with huge sets of data by processing them in parallel and batch systems. When you down load spark binaries there will separate folders to support above langauges.

PySpark是用python编写的Python API，用于支持Apache Spark。 Apache Spark是一个分布式框架，可以处理大数据分析。 Spark用Scala编写，可以与Python，Scala，Java，R，SQL语言集成。 Spark基本上是一个计算引擎，通过在并行和批处理系统中处理大量数据来处理它们。 当您下载spark二进制文件时，将有单独的文件夹来支持上述语言。

There are basically two major types of algorithms — transformers : Transforms work with the input datasets and modify it to output datasets using a transform().

基本上有两种主要的算法类型-转换器：转换与输入数据集一起使用，并使用transform()将其修改为输出数据集。

Estimators are the algorithms that take input datasets and produces a trained output model using fit().

估计器是采用输入数据集并使用fit()生成训练后的输出模型的算法。

In this section, I will be showing the machine learning implementation using Spark and Python. I will be focusing here basic ML algorithm Linear Regression implemented in the context of Spark. The program has been executed in the standalone server.

在本节中，我将展示使用Spark和Python的机器学习实现。 我将在这里重点介绍在Spark环境中实现的基本ML算法线性回归。 该程序已在独立服务器中执行。

First, import the libraries as shown below. And it is the most important to give the path of Spark binaries present in your system. Otherwise, you may face issues in executing codes.

首先，如下所示导入库。 给出系统中存在的Spark二进制文件的路径是最重要的。 否则，您可能会在执行代码时遇到问题。

Spark Session :

火花会议 ：

This is the entry point to the programming spark with Dataframe API & dataset. That allows you to perform various tasks using spark. spark context, hive context, SQL context, now all of it is encapsulated in the session. Before spark 2.0, sparkContext was used to access all spark functionality. The spark driver program uses sparkContext to connect to the cluster through a resource manager. sparkConf creates the sparkContext object, which stores configuration parameter like appName (to identify your spark driver), application, number of core, and memory size of executor running on the worker node. After spark 2.0 onwards these two features are encapsulated in spark session. So each time you want to perform tasks using spark you need to create a session and after execution, you must end the session.

这是使用Dataframe API和数据集进行编程火花的切入点。 这使您可以使用spark执行各种任务。 spark上下文，hive上下文，SQL上下文，现在所有这些都封装在会话中。 在spark 2.0之前，sparkContext用于访问所有spark功能。 Spark驱动程序使用sparkContext通过资源管理器连接到集群。 sparkConf创建s