jupyter通过pip部署并配置spark2
以下步骤在HDP03节点操作
- 创建jupyter用户并创建jupyter数据目录
// 创建用户
[root@HDP03 ~]# useradd jupyter
[root@HDP03 ~]# passwd jupyter
Changing password for user jupyter.
New password: // 密码:jupyter
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: // 确认密码:jupyter
passwd: all authentication tokens updated successfully.
// 创建数据目录
[root@HDP03 ~]# mkdir /data/jupyter/data -p
[root@HDP03 ~]# chown -R jupyter:jupyter /data/jupyter/
- 通过pip3安装jupyter
[root@HDP03 ~]# su - jupyter
[jupyter@HDP03 ~]$ pip3 install jupyter -i https://mirrors.aliyun.com/pypi/simple/
- 添加jupyter环境变量
[jupyter@HDP03 ~]$ sudo find / -name "jupyter"
/usr/local/python3/bin/jupyter // 命令存放目录
[jupyter@HDP03 ~]$ sudo vim /etc/profile // 添加环境变量
--------------------------------------------------------
# jupyter_HOME
export JUPYTER_HOME=/usr/local/python3
export PATH=$PATH:$JUPYTER_HOME/bin
[jupyter@HDP03 ~]$ source /etc/profile
- 生成并修改配置文件,配置一个密码
[jupyter@HDP03 ~]$ jupyter notebook --generate-config
Writing default config to: /home/jupyter/.jupyter/jupyter_notebook_config.py
[jupyter@HDP03 ~]$ vim /home/jupyter/.jupyter/jupyter_notebook_config.py
c.NotebookApp.allow_remote_access = True
c.NotebookApp.ip = '192.168.99.43'
c.NotebookApp.notebook_dir = '/data/jupyter/data'
c.NotebookApp.port = 9999
[jupyter@HDP03 jupyter]$ jupyter notebook password
Enter password: 密码:admin
Verify password: 密码:admin
[NotebookPasswordApp] Wrote hashed password to /home/jupyter/.jupyter/jupyter_notebook_config.json
- 启动jupyter服务
[jupyter@HDP03 ~]$ cd /data/jupyter/data/
[jupyter@HDP03 jupyter]$ nohup jupyter notebook >> ./jupyter.log 2>&1 &
[jupyter@HDP03 jupyter]$ ss -tnl | grep 9999
LISTEN 0 128 192.168.99.48:9999 *:*
jupyter-notebook连接spark2
以下步骤在HDP03节点操作
- ambari上创建jupyter用户,并在hdfs上创建对应目录
[jupyter@HDP03 ~]$ vim ~/.bashrc
----------------------------------------------------------------
# Spark安装目录
export SPARK_HOME=/usr/hdp/3.1.0.0-78/spark2
export PATH=$PATH:$SPARK_HOME/sbin:$SPARK_HOME/bin:$PATH
#指向spark目录下的python文件夹和py4j包
export PYTHON_PATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHON_PATH
#使用python3
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --ip=192.168.99.48 --no-browser --allow-root" // 这里ip为服务器ip地址
[jupyter@HDP03 ~]$ source ~/.bashrc
[jupyter@HDP03 ~]$ nohup pyspark >> ./jupyter.log 2>&1 & // 后台启动服务