php提交spark,[Spark] 提交Spark任务报错 return code 3

最新推荐文章于 2024-07-03 07:15:00 发布

吕旸

最新推荐文章于 2024-07-03 07:15:00 发布

阅读量665

点赞数

文章标签： php提交spark

在尝试通过PHP提交Spark任务时遇到了错误，返回代码为3。问题源于执行过程中频繁的Full GC导致内存不足。建议调整Spark执行器的内存设置，例如在客户端通过`set spark.executor.memory=6G`来设定，或者在程序中直接修改参数，或在服务端配置文件中调整并重启服务以生效。

摘要由CSDN通过智能技术生成

环境：百台2U，数据量较大

报错信息：

2017-05-28 00:06:09,997 | ERROR | HiveServer2-Background-Pool: Thread-2033273 | Error running hive query: | org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:248)

org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:339)

at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:169)

at org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)

at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache