【Bug】[42000][30041] Error while processing statement: FAILED: Execution Error,from org.apache.hadoop

• hive on spark执行报错，只要涉及spark就报错（阿里云4核16G）

2024-03-23 17:02:52,843 ERROR [7238bdbe-1441-40e2-8494-19ac6449b0b3 main] spark.SparkTask (SparkTask.java:execute(190)) - Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 6d6070dc-6bbf-4e62-91f7-ca16d6de69f5)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 6d6070dc-6bbf-4e62-91f7-ca16d6de69f5
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getHiveException(SparkSessionImpl.java:215)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:92)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:115)
        at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:136)
        at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:115)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:128)
        at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:88)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:105)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:101)
        at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:76)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:87)
        ... 24 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:106)
        ... 29 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
        at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:172)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)

根据上面报错，再jps查看进程只有一个RunJar(hivesever2),可以得知spark没有启动，可能没有内存启动！

1. hive/conf/hive-env.sh(2048→4096,不够就接着往上加，目前加到5G了)
   

2. hive/conf/spark-defaults.conf（1g➡5g跟据服务器情况酌情考量）
   

3. spark 是在yarn上的，因而要把yarn-site.xml中的内存开大一点(不知道有没有用，根据自己的情况更改大小！)

 <!--yarn单个容器允许分配的最大最小内存 -->
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>16384</value>
    </property>

    <!-- yarn容器允许管理的物理内存大小 -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>16384</value>
    </property>

    <!-- 关闭yarn对物理内存和虚拟内存的限制检查 -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

4. kill -9 杀掉hivesever2或者Hive Metastore进程重启hive（反正就是鲨掉所有RunJar进程)

# 二选一，我是用DG，因此重启hiveserver2
# 启动元数据库，后台启动更佳
nohup /opt/module/hive/bin/hive --service metastore &
# 启动hiveserver2服务，后台启动
nohup /opt/module/hive/bin/hiveserver2 1>/dev/null 2>/dev/null &

看到这个就启动成功了！hive on spark 启动！

还是会报错，重启hive就好了

• 重启hive脚本（改下hive路径）

#!/bin/bash
case $1 in
"start"){
                echo ---------- hive $i 启动 ------------
                nohup /opt/module/hive/bin/hiveserver2 1>/dev/null 2>/dev/null & ;jps
};;
"stop"){
        echo ---------- hive $i 停止 ------------ 
        ps -ef | grep hive | awk '{print $2}' | xargs kill -9;jps
};;
"restart"){
        echo ---------- hive $i 重启 ------------ 
        hive.sh stop;
                hive.sh start;jps
};;
esac

求助大佬，还有没有更好的解决方案（目前是重启就能用，偶尔不能用）