pycharm连接服务器spark报错

环境:windows+pycharm+pyspark

错误一:OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
  File "/tmp/pycharm_project_744/work/qiedian/data_preprocessing.py", line 28, in <module>
    spark = SparkSession.builder.master("yarn-client").appName("test").getOrCreate()
  File "/home/fxj/miniconda2/lib/python2.7/site-packages/pyspark/sql/session.py", line 173, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/home/fxj/miniconda2/lib/python2.7/site-packages/pyspark/context.py", line 367, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/home/fxj/miniconda2/lib/python2.7/site-packages/pyspark/context.py", line 133, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/home/fxj/miniconda2/lib/python2.7/site-packages/pyspark/context.py", line 316, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/home/fxj/miniconda2/lib/python2.7/site-packages/pyspark/java_gateway.py", line 46, in launch_gateway
    return _launch_gateway(conf)
  File "/home/fxj/miniconda2/lib/python2.7/site-packages/pyspark/java_gateway.py", line 98, in _launch_gateway
    proc = Popen(command, stdin=PIPE, preexec_fn=preexec_func, env=env)
  File "/home/fxj/miniconda2/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/home/fxj/miniconda2/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

此问题的原因是pycharm中java环境未配置。
选择edit Configuration
在这里插入图片描述
配置JAVA_HOME
在这里插入图片描述
在这里插入图片描述JAVA_HOME是服务器中的JAVA信息。
报错二:
Exception: java.lang.Exception: When running with master ‘yarn’ either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. in spark

这个问题的原因是在pycharm中配置的hadoop环境路径信息定义的问题。
首先定义路径信息

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

然后在pycharm中配置一下。

在这里插入图片描述

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值