![](https://img-blog.csdnimg.cn/20201014180756918.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
scala
北镒
只争朝夕,不负韶华!
展开
-
Spark 中的UDF
对于非匿名函数写法spark.udf.register("is_yg_dabiao", isYGDaBiao(_: String, _: String): String)对于非匿名函数的注册,一般是函数名需要与我们的自定义函数一致,参数列表中的形参名称可以用下划线代替对于匿名函数写法spark .udf.register("is_yg_dabiao", (a: String, b: String) => a.toInt + b.toInt)对于匿名函数的注册,".原创 2021-09-26 10:14:59 · 464 阅读 · 0 评论 -
几种文件读取方式
1、Spark读取HDFS文件import org.apache.spark.SparkContextimport org.apache.spark.rdd.RDDimport org.apache.spark.sql.SparkSession/** 环境及参数 */ val spark: SparkSession = SparkSession.builder().getOrCreate() val sc: SparkContext = spark.sparkContext原创 2021-07-20 10:12:31 · 132 阅读 · 0 评论 -
Some keys were defined with the same name but different types解决办法
Some keys were defined with the same name but different types: 'ssExtractBuild'报这个错误,是和sbt插件冲突有关,到.sbt目录下把plugins目录剪切出去(或者删除),然后重新导入sbt配置原创 2021-07-19 10:00:43 · 245 阅读 · 0 评论 -
java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream
本期是一个spark内部jar冲突的问题shangdiAllResult.repartition(1).write.mode("overwrite").text(selectFrameOut)name := "bj_traffic_laboratory"lazy val commonSettings = Seq( organization := "com.xwsoft", version := "0.1.0", scalaVersion := "2.11.12", up原创 2021-05-25 11:29:44 · 375 阅读 · 0 评论 -
sbt管理项目时build.sbt文件的编写
无子项目情况name := "temp_test"version := "0.1"scalaVersion := "2.11.12"updateOptions := updateOptions.value.withCachedResolution(true)libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.11" % "2.3.2", "org.apache.spark" % "spark-sql_2原创 2021-04-29 18:03:23 · 645 阅读 · 0 评论 -
scala数据类型报错
先上报错Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field:- field (class: "scala.Long", name: "f_cell_id")- root class: "com.xwsoft.structure.Bts"If the schema is inferred from a Scala tuple/case class, or a Java bean,原创 2021-03-08 15:52:54 · 838 阅读 · 0 评论