qq_37597249的博客

一、spark on yarn 提交任务有两种模式1、cluster模式 :ApplicationMaster进程进行资源分配和executor的启动提交命令：--master yarn \--deploy-mode client \--driver-memory 5G \ applicationMaster 所在的容器分配的内存--driver-cores 5 \ a...

import org.apache.spark.sql.{SaveMode, SparkSession}import org.apache.spark.sql.jdbc.JdbcDialectsimport org.apache.spark.sql.jdbc._import org.apache.spark.sql.types._一、读Oracle 表// 参数自己传进来1、定...

session.udf.register("toMMS",(str:String) =>{ val sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS") val sdf2 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss") var st ="" if(str.length>1...

报错信息： java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 61, Column 30: Index expression of ...

错误信息：06/13/18 13:17:50  06/14/18 13:17:50  krbtgt/HADOOP.COM@HADOOP.COM18/06/13 13:17:59 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under S...

spark 报错 [Stage 45:(482 + 46) / 528][Stage 46:(64 + 148) / 528][Stage 55:>(27 + 0) / 528]18/06/07 16:07:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceedin...

1.执行：vi /etc/inputrc    #set bell-style none 把这一行的#去掉2、执行：vi ~/.bashrc   添加：setterm –blength 03、执行：reboot 声音关闭

直接操作表的时候用自增这个方式会报错因为spark over里必须有字段  在hive中使用    直接用SQL : create table table2Name as select row_number() over () as row_id,* from table1NameUUID主键：也可以操作表 也可以操作RDD 可以同时 这种在增加UUID之后必须cache 他会一直到触发a...