python —— linux Anoconda3安装及pyspark搭建

Anoconda3资源获取

python自身缺少numpy、matplotlib、scipy、scikit-learn....等一系列包，需要安装pip来导入这些包才能进行相应运算Anaconda(开源的Python包管理器)是一个python发行版，包含了conda、Python等180多个科学包及其依赖项。包含了大量的包，使用anaconda无需再去额外安装所需包

资源获取：https://pan.baidu.com/s/1PAINsu3EX-fWw1QdsZ6CAA
提取码：t4hh

安装步骤

# 1. 前置安装spark并配置环境变量
[root@jzy1 ~]# vi /etc/profile
    export SPARK_HOME=/opt/soft/spark234
    export SPARK_CONF_DIR=$SPARK_HOME/conf
    export PATH=$PATH:$PARK_HOME/bin
[root@jzy1 ~]# source /etc/profile



# 2. 安装bzip2(此步不执行安装anaconda会报错)
[root@jzy1 ~]# yum install -y bzip2


# 3. 执行下载到linux下的Anaconda3-5.1.0-Linux-x86_64.sh文件
根据提示回车或yes 只有最后安装vcode回答no(不需要安装vcode，下载没有必要)
[root@jzy1 opt]# bash Anaconda3-5.1.0-Linux-x86_64.sh


# 4. 生成jupyter配置文件 进入如下路径,并执行命令
[root@jzy1 ~]# cd /root/anaconda3/bin/
[root@jzy1 ~]# ./jupyter notebook --generate-config


# 5. 按照如下步骤 生成Jupyter登录密码
[root@jzy1 bin]# ./ipython
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from notebook.auth import passwd

In [2]: passwd()
Enter password: 
Verify password: 
Out[2]: 'sha1:297308f26ac6:bc524d2e1afafdbb798456a9161254962968b113'
In [3]: quit()

记事本贴好 （这是我的，每个人生成的不一样）sha1:297308f26ac6:bc524d2e1afafdbb798456a9161254962968b113



# 6. 修改jupyter_notebook_config.py文件
[root@jzy1 bin]# cd /root/.jupyter/
[root@jzy1 .jupyter]# vi jupyter_notebook_config.py 

c.NotebookApp.allow_root=True
c.NotebookApp.ip='*'
c.NotebookApp.open_browser=False
c.NotebookApp.password=u'刚才生成的密码'
c.NotebookApp.port=7070 #此端口为不使用pyspark的端口

# 7. 配置环境变量
vi /etc/profile

export ANACONDA_HOME=/root/anaconda3
export PATH=$PATH:$ANACONDA_HOME/bin
export PYSPARK_DRIVER_PYTHON=jupyter-notebook
export PYSPARK_DRIVER_PYTHON_OPTS="--ip=0.0.0.0 --port=8888"

执行

• 如果要环境加入pyspark 就直接执行命令 pyspark ，端口号8888

• 如果只需要普通python环境 就输入命令 jupyter notebook --allow-root，端口号7070

  然后在web端输入不同的端口号可访问即说明安装完成

[root@jzy1 .jupyter]# pyspark 
[root@jzy1 ~]# pyspark
[I 10:15:59.670 NotebookApp] JupyterLab beta preview extension loaded from /root/anaconda3/lib/python3.6/site-packages/jupyterlab
[I 10:15:59.671 NotebookApp] JupyterLab application directory is /root/anaconda3/share/jupyter/lab
[I 10:15:59.673 NotebookApp] Serving notebooks from local directory: /root
[I 10:15:59.673 NotebookApp] 0 active kernels
[I 10:15:59.673 NotebookApp] The Jupyter Notebook is running at:
[I 10:15:59.673 NotebookApp] http://0.0.0.0:8888/
[I 10:15:59.673 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

#此时可以进入web网页输入连接端口


[root@jzy1 .jupyter]# jupyter notebook --allow-root