- 博客(12)
- 收藏
- 关注
RSS订阅原创 java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object
spark程序报错：java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)原因为打包时本地所用的scala版本是2.12 ，而集群中的scala版本是2.11，将本地的scala版本改成2.11后在打包运行， 运行成功...
2022-03-09 20:28:15
5016
原创 SparkException: Dynamic allocation of executors requires the external shuffle service解决办法
详细报错： ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled....ERROR ApplicationMaster: U...
2022-03-09 16:38:46
2279
1
原创 sqoop ERROR tool.ExportTool: Error during export: Export job failed!
sqoop导入数据报错：21/06/09 17:31:19 ERROR tool.ExportTool: Error during export:Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:442) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)...
2021-06-09 18:00:17
849
原创 FAILED: SemanticException 0:0 Error creating temporary folder on: Error encountered near token
执行hive sql任务时报错：FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://ShareSdkHadoop/user/hive/warehouse/dm_di_master.db. Error encountered near token 'TOK_TMP_FILE'报错原因：操作线上库，当前的测试用户没有线上库的写入权限解决办法：调用shell脚本时，在hive前加上HADOOP_USER.
2021-06-08 12:26:07
3762
原创 hive grouping sets使用
1、数据准备企业一 部门一 张三1企业一 部门二 张三2企业一 部门三 张三3企业一 部门四 张三4企业二 部门一 李四1企业二 部门二 李四2企业二 部门三 李四3企业二 部门四 李四4企业三 部门一 王二1企业三 部门二 王二2企业三 部门三 王二3企业三 部门四 王二42、建测试表CREATE TABLE qc_test.dsw_test_group ( company string, department string, `employee` str
2021-05-28 17:15:10
443
原创 Presto常见报错
报错：Error:Query failed (#20210526_084418_01979_zqazj): Hive views are not supported: 'dm_sdk_mapping.dim_transform_full_par_secview'解决办法：presto查询不支持视图表，切换为对应的原表即可报错：Error:Query failed (#20210526_085736_02043_zqazj): SQL array indices start at 1...
2021-05-26 17:11:36
13363
原创 Application application_id failed 2 times due to ApplicationMaster for attempt xx timed out解决办法
spark-submit 提交任务失败，报错详情：Application application_1609129417468_1602993 failed 2 times due to ApplicationMaster for attempt appattempt_1609129417468_1602993_000002 timed out. Failing the application.排查为ApplicationMaster 为子任务尝试次数超时，于是尝试添加以下参数：--conf s.
2021-05-25 09:56:22
6039
2
原创 shell 中 报错 integer expression expected 解决办法
执行shell 报错：排查为 -ge -le 支持判断整数，不支持判断小数大小解决方法：使用bc命令比较小数大小，再用 -eq 1 判断，返回1 比较true 返回0表示false 如：echo "0.1>= 0.05" | bca=`echo "0.1 >= 0.05" | bc`if [ $a -eq 1 ];then echo trueelse echo falsefi...
2021-04-16 21:45:15
8407
原创 FAILED: SemanticException The abstract syntax tree is null 解决办法
hive中sql报错：FAILED: SemanticException The abstract syntax tree is null执行sql为：with t as(select 1 id,'k1' key,123 value union allselect 1 id,'k2' key,124 value union allselect 2 ,'k1' key,234 value )select id, max(case when key='k1' then ...
2021-04-14 12:08:41
1067
原创 Cannot create a table having a column whose name contains commas in Hive metastore. 解决办法
spark 报错：Cannot create a table having a column whose name contains commas in Hive metastore. Table: `xxx`.`xxx`; Column: get_geohash(43.834269, 87.605044, 8);解决办法：sql 中方法后写别名即可，如下：get_geohash(43.834269, 87.605044, 8) as b...
2021-03-26 20:13:33
1677
原创 Detected yarn cluster mode, but isn‘t running on a cluster. Deployment to YARN is not supported 解决办法
spark-submit报如下错：org.apache.spark.SparkException: Detected yarn cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.排查为 将--deploy-mode cluster 配置写入jar包时，使用spark-submit 提...
2021-03-16 17:43:34
1267
原创 Exception in thread "main"java.lang. .UnsupportedClassVersionError: 的解决办法
最近在linux上运行windows上打jar包时报错：Exception in thread "main" java.lang.UnsupportedClassVersionError: wc/WordCountMain : Unsupported major.minor version 52.0 ， 截图如下：
2018-07-07 22:45:42
5531
1
空空如也
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人