启动Spark-Shell报错:
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’.
解决办法:
将报错的2个参数增大到1434MB
然后重启Yarn服务
启动完毕后,重新运行Spark,报新的错误:
19/01/07 22:07:33 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
于是切换成hdfs用户运行spark,这次没有任何错误,启动成功。
sudo -u hdfs spark-shell