Cloudera Manager启动Spark报内存超出限制的错误：Required executor memory (1024+384 MB) is above the max threshold

启动Spark-Shell报错：
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’.

解决办法：

1. 将报错的2个参数增大到1434MB
   
   

然后重启Yarn服务

3. 启动完毕后，重新运行Spark，报新的错误：
   19/01/07 22:07:33 ERROR spark.SparkContext: Error initializing SparkContext.
   org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
   

4. 于是切换成hdfs用户运行spark，这次没有任何错误，启动成功。

sudo -u hdfs spark-shell

我的博客即将同步至腾讯云+社区，邀请大家一同入驻：https://cloud.tencent.com/developer/support-plan?invite_code=21h11qow6cbo4