-- Spark
踩坑Spark
海若[MATRIX]
大数据全栈
展开
-
Spark源码阅读_1:Spark2.4.3 win10 IDEA 源码编译调试踩坑
1.安装包准备jdk1.8maven3.5.4scala2.11.2git2.32.0spark2.4.3【spark-2.4.3.tgz 】注意:安装过程,自行百度,版本要一致,我是认真看了pom文件确定的版本。2.报错在源码根目录运行cmd,执行./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.5 -DskipTests clean package报错如下:[INFO] BUILD FAILURE[INFO] ---..原创 2021-07-05 00:46:40 · 186 阅读 · 0 评论 -
Spark报错: Permission denied: user=HaiRuo, access=WRITE
1.场景spark在hive上运行时报错2.报错Caused by: `org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=atguigu, access=WRITE, inode="/user/hive/warehouse/dwd.db/dwd_qz_chapter":root:supergroup:drwxr-xr-x..原创 2020-06-23 19:28:48 · 891 阅读 · 0 评论 -
Spark报错:Table or view not found
1.场景 def etlQzChapter(ssc: SparkContext, sparkSession: SparkSession) = { import sparkSession.implicits._ //隐式转换 ssc.textFile("/user/atguigu/ods/QzChapter.log").filter(item => { val obj = ParseJsonData.getJsonData(item) obj.isInst..原创 2020-06-23 17:00:08 · 7565 阅读 · 2 评论 -
运行Spark-shell报错:File does not exist: hdfs://mycluster/spark_historylog
1.场景执行spark-shell报错[root@hadoop101 conf]# spark-shell2.报错Setting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).2020-06-19 22:42:16,335 ERROR spark.SparkContext: Error initi..原创 2020-06-19 23:58:33 · 4324 阅读 · 1 评论 -
Spark连接Hive报错:1 字节的 UTF-8 序列的字节 1 无效 Error while instantiating
1.场景:使用Spark的Java API连接Hivejava代码 public static void main(String[] args) { ThreadLocal<SparkSession> sessionPool = new ThreadLocal<>(); // 先判断会话池中是否有session,如果有就直接用,没有再创建 if (sessionPool.get() != null) { ...原创 2020-06-10 11:47:34 · 361 阅读 · 0 评论