linux spark python,pyspark模块在spark使用的python实例中不可用

当您调用spark submit时，它会为您将自己的python文件夹和py4j库添加到PYTHONPATH中。它们位于spark安装到的文件夹中，它们不是直接安装到您正在使用的python站点包中。例如，如果我pyspark打印出python路径：$ pyspark

Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)

[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

17/10/17 15:34:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/10/17 15:34:04 WARN Utils: Your hostname, rwidmaier resolves to a loopback address: 127.0.0.1; using 10.4.248.126 instead (on interface em1)

17/10/17 15:34:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

/home/rwidmaier/apps/spark/python/pyspark/context.py:195: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0

warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0")

17/10/17 15:34:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0

17/10/17 15:34:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

17/10/17 15:34:09 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

Welcome to

____ __

/ __/__ ___ _____/ /__

_\ \/ _ \/ _ `/ __/ '_/

/__ / .__/\_,_/_/ /_/\_\ version 2.2.0

/_/

Using Python version 2.6.6 (r266:84292, Aug 18 2016 15:13:37)

SparkSession available as 'spark'.

>>> import sys

>>> from pprint import pprint

>>> pprint(sys.path)

['',

u'/tmp/spark-0a08efa7-714a-498b-b5e9-bba48684d52d/userFiles-52012e25-d7af-4599-a214-9637141ed4ec',

'/usr/lib64/python2.6/site-packages/deluge-1.3.11-py2.6-linux-x86_64.egg',

'/home/ryan/apps/spark/python/lib/py4j-0.10.4-src.zip',

'/home/ryan/apps/spark/python',

'/usr/lib64/python26.zip',

'/usr/lib64/python2.6',

'/usr/lib64/python2.6/plat-linux2',

'/usr/lib64/python2.6/lib-tk',

'/usr/lib64/python2.6/lib-old',

'/usr/lib64/python2.6/lib-dynload',

'/usr/lib64/python2.6/site-packages',

'/usr/lib64/python2.6/site-packages/gst-0.10',

'/usr/lib64/python2.6/site-packages/gtk-2.0',

'/usr/lib64/python2.6/site-packages/webkit-1.0',

'/usr/lib/python2.6/site-packages']

您可以在列表中看到它已手动添加：/home/ryan/apps/spark/python/lib/py4j-0.10.4-src.zip文件在

/home/ryan/apps/spark/python

为了确保解释器始终使用正确的库进行设置，只需将PYSPARK_PYTHON设置为指向要使用的PYTHON安装，然后运行“PYSPARK”，而不是直接运行PYTHON。或者，如果你想提供一个驱动程序，你也可以使用spark submit。在