测试Spark与hive的连通性代码
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("spark sql hive query").enableHiveSupport().getOrCreate()
spark.sql("show databases").show()
PYSPARK_PYTHON=/home/bbders/anaconda3/bin/python3.6 /usr/bin/spark2-submit 123.py
类似这样的格式执行
代码二
import pyspark.sql
SparkSession = pyspark.sql.SparkSession
session = SparkSession.builder.appName('kunlun-indexer').config('spark.port.maxRetries', 1024).enableHiveSupport().getOrCreate()
sparkql = '''
show databases
'''
session.sql(sparkql).show()
#spark2-submit --master yarn test.py(test.py 代码如上)
2、资源问题查看yarn的资源池是否满足需求
3、权限问题