spark
头顶榴莲树
持续分享大数据技术经验,欢迎交流~
展开
-
There is no available StoreManager of type “rdbms“-HDP-3.1.5集成Spark-3.1.1报错
详细日志:22/03/29 09:02:47 ERROR Datastore: Exception thrown creating StoreManager. See the nested exceptionThere is no available StoreManager of type "rdbms". Make sure that you have put the relevant DataNucleus store plugin in your CLASSPATH and if defin原创 2022-03-29 17:05:32 · 2071 阅读 · 0 评论 -
Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration
HDP3使用apache版Spark2.4查询hive时报错:Caused by: java.lang.RuntimeException: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration at org.apache.hadoop.hive.ql.session.SessionState.st原创 2022-03-24 16:22:09 · 2556 阅读 · 0 评论 -
Spark作业一直处于ACCEPTED状态
原因1:集群处于不健康状态打开yarn监控页面,查看nodes 状态如果Unhealthy Nodes列不为0,说明有的节点不健康,可以点进去查看详情,处理完并且集群处于健康状态后可尝试再次提交作业。原创 2022-02-25 13:36:27 · 3159 阅读 · 0 评论 -
Spark提交报错:1 node(s) are excluded in this operation
提交spark on yarn作业报错:主要错误信息“There are 1 datanode(s) running and 1 node(s) are excluded in this operation”,有一个datanode被排除1088 [main] INFO org.apache.spark.deploy.yarn.Client - Verifying our application has not requested more than the maximum memory ca原创 2022-02-25 10:59:39 · 2764 阅读 · 0 评论 -
IDEA本地执行Spark报错:is not a valid DFS filename
本地执行spark structured streaming 报错,程序代码:def main(args: Array[String]): Unit = { val spark = SparkSession .builder .master("local[2]") .appName("sparkStream2hudi") .getOrCreate() //消费kafka import spark.implicits._原创 2022-02-21 10:31:52 · 2423 阅读 · 2 评论 -
HDP-3.1.5.0-152环境与hudi-0.9.0兼容性问题
集群环境:HDP 3.1.5.0-152Spark 2.3.0(hudi官网要求2.4+)Hudi 0.9.0当前环境执行hudi官网测试代码报错“NoSuchFieldError: NULL_VALUE”,代码如下:def main(args: Array[String]): Unit = { val spark = SparkSession.builder .master("local[2]") .appName("hudi-test")原创 2022-02-18 14:58:02 · 1012 阅读 · 1 评论 -
hdp集成apache spark提交任务时报错:bad substitution
__spark_conf__/__hadoop_conf__: bad substitution原创 2022-02-18 14:48:32 · 1736 阅读 · 2 评论