jupyter通过pip部署并配置spark2

jupyter通过pip部署并配置spark2

以下步骤在HDP03节点操作

1. 创建jupyter用户并创建jupyter数据目录

// 创建用户
[root@HDP03 ~]# useradd  jupyter
[root@HDP03 ~]# passwd jupyter
Changing password for user jupyter.
New password: // 密码：jupyter
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: // 确认密码：jupyter 
passwd: all authentication tokens updated successfully.

// 创建数据目录
[root@HDP03 ~]# mkdir /data/jupyter/data -p
[root@HDP03 ~]# chown  -R jupyter:jupyter /data/jupyter/

2. 通过pip3安装jupyter

[root@HDP03 ~]# su - jupyter
[jupyter@HDP03 ~]$ pip3 install jupyter -i https://mirrors.aliyun.com/pypi/simple/

3. 添加jupyter环境变量

[jupyter@HDP03 ~]$ sudo find / -name "jupyter"
/usr/local/python3/bin/jupyter                  // 命令存放目录

[jupyter@HDP03 ~]$ sudo vim /etc/profile        // 添加环境变量
--------------------------------------------------------
# jupyter_HOME
export JUPYTER_HOME=/usr/local/python3
export PATH=$PATH:$JUPYTER_HOME/bin

[jupyter@HDP03 ~]$ source /etc/profile

4. 生成并修改配置文件,配置一个密码

[jupyter@HDP03 ~]$ jupyter notebook --generate-config
Writing default config to: /home/jupyter/.jupyter/jupyter_notebook_config.py

[jupyter@HDP03 ~]$ vim /home/jupyter/.jupyter/jupyter_notebook_config.py
c.NotebookApp.allow_remote_access = True
c.NotebookApp.ip = '192.168.99.43'
c.NotebookApp.notebook_dir = '/data/jupyter/data'
c.NotebookApp.port = 9999


[jupyter@HDP03 jupyter]$ jupyter notebook password
Enter password:    密码：admin
Verify password:    密码：admin
[NotebookPasswordApp] Wrote hashed password to /home/jupyter/.jupyter/jupyter_notebook_config.json

5. 启动jupyter服务

[jupyter@HDP03 ~]$ cd /data/jupyter/data/
[jupyter@HDP03 jupyter]$ nohup  jupyter notebook >> ./jupyter.log 2>&1 &
[jupyter@HDP03 jupyter]$ ss -tnl | grep 9999
LISTEN     0      128    192.168.99.48:9999                     *:*

jupyter-notebook连接spark2

以下步骤在HDP03节点操作

1. ambari上创建jupyter用户,并在hdfs上创建对应目录

[jupyter@HDP03 ~]$ vim ~/.bashrc
----------------------------------------------------------------
# Spark安装目录
export SPARK_HOME=/usr/hdp/3.1.0.0-78/spark2
export PATH=$PATH:$SPARK_HOME/sbin:$SPARK_HOME/bin:$PATH
#指向spark目录下的python文件夹和py4j包
export PYTHON_PATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHON_PATH

#使用python3
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --ip=192.168.99.48 --no-browser --allow-root"               // 这里ip为服务器ip地址

[jupyter@HDP03 ~]$ source  ~/.bashrc
[jupyter@HDP03 ~]$ nohup pyspark >> ./jupyter.log 2>&1 &        // 后台启动服务