spark
晴々明雅
这个作者很懒,什么都没留下…
展开
-
clickhouse unexpected packet data received from client,stack strace
clickhouse unexpected packet data received from client 亲测解决方式原创 2023-02-10 14:09:20 · 740 阅读 · 0 评论 -
com.typesafe.config.ConfigException$BugOrBroken: com.typesafe.config.impl.SerializedConfigValue shou
spark写图片数据到sftp报错:com.typesafe.config.ConfigException$BugOrBroken: com.typesafe.config.impl.SerializedConfigValue should not exist outside of serialization原创 2022-04-07 20:50:51 · 1491 阅读 · 0 评论 -
spark + sftp
1,读取本地图片,并上传至SFTP val file = new File("/mnt/data/hdd4/yarn/local/usercache/IT_bigdata/appcache/tag1200.jpg") val img = ImageIO.read(file) val baos = new ByteArrayOutputStream() ImageIO.write(img, "jpg", baos) val byte = baos.toByteArray原创 2022-04-02 08:27:20 · 802 阅读 · 0 评论 -
Failed to execute user defined function(anonfun$concatStr$1: (map<string,string>, string) => string)
自定义spark UDF函数如下: def concatStr: UserDefinedFunction = udf((list: Map[String, String], key: String) => { val param = list.getOrElse(key, "") key.concat(param) })今天出现报错信息:org.apache.spark.SparkException: Failed to execute user defined fu原创 2020-10-07 10:15:57 · 3913 阅读 · 0 评论 -
java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names()[Ljava/lang/String;
原jar包版本2.4.3,现在集群运行环境是2.3.2,代码中有一段要调用StructType.names()方法,结果报错java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names()[Ljava/lang/String;使用如下文章中的方法:https://blog.csdn.net/wuhuimin9...原创 2020-04-10 11:09:46 · 721 阅读 · 0 评论 -
spark 中生成包含元组(1,“a") (2,"b”) (3,"c”)...(25,"y") (26,"z")的RDD
需求:把26个字母与数字组合在一起,并形成RDD算子将字母和数字放一起有很多形式val strings = 'a' to 'z'for (i <- 0 to 25) {(i + 1, strings(i).toString)}这样就可以打印出需要的格式但是怎么生成RDD呢首先RDD中的是一系列的元组,类似与Array((Int,String))=((1,“a") (2,"...原创 2019-06-25 14:05:43 · 565 阅读 · 0 评论