Jupyter与PySpark实现结合spark与python的notebook

Jupyter配置
假设Spark已经配置正常，pyspark也可以正常在shell中使用了，只不过此时使用的python是系统预置的，我们需要改成Anaconda3的IPython实现，为此在当前用户的.bashrc或/etc/profile中增加配置

export PYSPARK_DRIVER_PYTHON=/webdev/app/anaconda3/bin/jupyter-notebook 
export PYSPARK_DRIVER_PYTHON_OPTS=" --ip=0.0.0.0 --port=7777"

如此以来，在启动$SPARK_HOME/bin/pyspark时便可根据环境变量使用Anaconda的jupyter-notebook

注意以上export了PYSPARK_DRIVER_PYTHON与PYSPARK_DRIVER_PYTHON_OPTS两个环境变量后，非shell的pyspark 生怕认可应用也将使用者jupyter-notebook，这必然引起混乱，所以推荐的还是在pyspark的启动命令中当时指定。 
如：

master$ PYSPARK_DRIVER_PYTHON=/webdev/app/anaconda3/bin/jupyter-notebook PYSPARK_DRIVER_PYTHON_OPTS=" --ip=0.0.0.0 --port=7777" pyspark --packages com.databricks:spark-csv_2.11:1.1.0 --master spark://spark_master_hostname:7077 --executor-memory 6400M --driver-memory 6400M

https://www.datacamp.com/community/tutorials/apache-spark-python#PySpark

 

Spark in Jupyter Notebook

Now that you have all that you need to get started, you can launch the Jupyter Notebook Application by typing the following:

PYSPARK_DRIVER_PYTHON="jupyter" PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark