在Spark中使用IPython Notebook

IPython Notebook现已更名为Jupyter Notebook，是一种交互式的笔记本，是可以用来重建和分享包含动态代码、等式、可视化和解释文本的文档的Web程序。Spark提供了python解释器pyspark，可以通过IPython Notebook将Spark的pyspark以笔记本这种交互式更强的方式来访问。

[root@master ~]# IPYTHON_OPTS="notebook --ip=1.2.3.4" pyspark
SPARK_MAJOR_VERSION is set to 1, using Spark
[TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated and will be removed in future versions.
[TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook` in the future
[I 22:28:23.204 NotebookApp] [nb_conda_kernels] enabled, 2 kernels found
[I 22:28:23.269 NotebookApp] ✓ nbpresent HTML export ENABLED
[W 22:28:23.269 NotebookApp] ✗ nbpresent PDF export DISABLED: No module named nbbrowserpdf.exporters.pdf
[I 22:28:23.271 NotebookApp] [nb_conda] enabled
[I 22:28:23.333 NotebookApp] [nb_anacondacloud] enabled
[I 22:28:23.337 NotebookApp] Serving notebooks from local directory: /root
[I 22:28:23.337 NotebookApp] 0 active kernels 
[I 22:28:23.337 NotebookApp] The Jupyter Notebook is running at: http://1.2.3.4:8888/
[I 22:28:23.337 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 22:28:23.337 NotebookApp] No web browser found: could not locate runnable browser.
[I 22:28:34.702 NotebookApp] 302 GET / (172.31.64.222) 0.86ms
[I 22:32:28.357 NotebookApp] Creating new notebook in /Documents
[I 22:32:28.382 NotebookApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[I 22:32:36.049 NotebookApp] Kernel started: 4d304d11-f29f-456e-a9c2-c7dc30204cfd

启动的命令为 IPYTHON_OPTS="notebook --ip=1.2.3.4" pyspark

需要提前安装ipython。推荐使用Anaconda进行安装。

在spark2.0之后的版本，使用上述命令会报错：

[xdwang@dell bin]$ IPYTHON_OPTS="notebook --ip=211.71.76.25" ./pyspark
Error in pyspark startup:
IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

在bashrc中增加环境变量：

vi .bashrc

增加：

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --ip=211.71.76.25"

重新启动：

[xdwang@dell ~]$ pyspark
[I 14:33:18.032 NotebookApp] [nb_conda_kernels] enabled, 2 kernels found
[I 14:33:18.045 NotebookApp] Writing notebook server cookie secret to /home/xdwang/.local/share/jupyter/runtime/notebook_cookie_secret
[I 14:33:18.491 NotebookApp] ✓ nbpresent HTML export ENABLED
[W 14:33:18.491 NotebookApp] ✗ nbpresent PDF export DISABLED: No module named 'nbbrowserpdf'
[I 14:33:18.922 NotebookApp] [nb_anacondacloud] enabled
[I 14:33:18.933 NotebookApp] [nb_conda] enabled
[I 14:33:18.962 NotebookApp] Serving notebooks from local directory: /home/xdwang
[I 14:33:18.962 NotebookApp] 0 active kernels 
[I 14:33:18.962 NotebookApp] The Jupyter Notebook is running at: http://211.71.76.25:8888/
[I 14:33:18.963 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 14:33:18.964 NotebookApp] No web browser found: could not locate runnable browser.
[I 14:33:44.921 NotebookApp] 302 GET / (202.205.97.62) 1.95ms