本地连接远程Spark(Python环境)

                版权声明：本文为博主原创文章，未经博主允许不得转载。                    https://blog.csdn.net/u013305783/article/details/85243187                </div>
                      <link rel="stylesheet" href="https://csdnimg.cn/release/phoenix/template/css/ck_htmledit_views-cd6c485e8b.css">
                          <div id="content_views" class="markdown_views prism-atom-one-dark">
        <!-- flowchart 箭头图标 勿删 -->
        <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
          <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
        </svg>
        <h1><a name="t0"></a><a id="Python_0" target="_blank"></a>一、Python环境准备</h1>

   远程服务器上Python版本要与本地想匹配，这里本地使用Anaconda来进行安装，远程服务器之间安装。

wget --no-check-certificate https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz
tar -zxvf Python-3.6.7.tgz 
sudo ./configure --prefix=/usr/local/python3
make & make install
建立软连接
mv /usr/bin/python /usr/bin/python.bak
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python

 
 

 
 
1
2
3
4
5
6
7

   本地之间下载Anaconda安装包，会自动安装相关环境。

二、配置PyCharm

   在file->setting->project interpreter中新增一个python环境，

   tool->Deployment->Configuration中新建一个SFTP

   Mappings中Local path路径正斜杠要修改一下改为反斜杠，否则可能会无法识别导致报错

    接下来需要在configuration中更改运行参数

   因为是远程环境，Environment Variables中需要指明服务器端python路径，spark路径，java路径。

   在file->setting->project interpreter下载安装py4j，pyspark等包，否则会报错

   以下是配置中遇到的一些常见错误：


   去上面搜索安装pyspark-stubs

    上面错误表示py4j版本不兼容，重新下载想匹配的版本

   spark FileNotFoundError: [WinError 2] 系统找不到指定的文件，需要查询edit configuration中python interpreter是否选择正确

   python3安装完成后默认没有安装pip，需要自己去安装，否则会出先pip未找到的错误

备份原来的源：输入命令sudo cp /etc/apt/sources.list /etc/apt/sources_init.list
输入命令sudo vi /etc/apt/sources.list
添加如下
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse 

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip
sudo apt-get install python-pip

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18

      </div>
      <link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-e44c3c0e64.css" rel="stylesheet">
              </div>