启动Spark-Shell报错:
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’.
解决办法:
- 将报错的2个参数增大到1434MB
然后重启Yarn服务
-
启动完毕后,重新运行Spark,报新的错误:
19/01/07 22:07:33 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwx