linux安装pyspark_Linux安装anaconda和集成PySpark - Configuration

Linux安装anaconda和集成PySpark - Configuration

Linux需要安装jdk,spark

使用curl下载Anaconda(这是一个脚本)

1)下载bzip：[root@head42 opt]# yum install bzip2.x86_64

2)运行脚本：[root@head42 opt]# sh Anaconda3-5.1.0-Linux-x86_64.sh (一直enter直到第一个yes,第二个no)

3)运行：ipython

4)输入：from notebook.auth import passwd

passwd()

​ 设置密码

​ 获取sha1值，复制

5)

c.NotebookApp.allow_root = True

c.NotebookApp.ip = '*'

c.NotebookApp.open_browser = False

c.NotebookApp.password = 'sha1:粘贴上一步复制的值'

c.NotebookApp.port = 7070

6)

cd~

vi ~/.bashr

添加以下内容

export PYSPARK_PYTHON=$ANACONDA_HOME/bin/python3

export PYSPARK_DRIVER_PYTHON=$ANACONDA_HOME/bin/jupyter

export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

ipython_opts="notebook -pylab inline"

cd~

source ./.bashrc

7)配置环境变量

export ANACONDA_HOME=/opt/anaconda3

export PATH=$PATH:$ANACONDA_HOME/bin

export PYSPARK_PYTHON=$ANACONDA_HOME/bin/python3

export PYSPARK_DRIVER_PYTHON=$ANACONDA_HOME/bin/jupyter

export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

ipython_opts="notebook -pylab inline"

8)启动pyspark

这样就OK了