楔子
spark 运行提示
io.netty.buffer.PooledByteBufAllocator.metric()
查找提示netty 版本 冲突
问题1 io.netty.buffer.PooledByteBufAllocator.metric()
pom解决冲突
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.17.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>3.9.9.Final</version>
</dependency>
</dependencies>
</dependencyManagement>
问题2 .NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/Inpu
kafka-client·依赖,exclude掉lz4的依赖
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.11.0.2</version>
<exclusions>
<exclusion>
<groupId>net.jpountz.lz4</groupId>
<artifactId>lz4</artifactId>
</exclusion>
</exclusions>
</dependency>
参考 https://blog.csdn.net/wuhuimin900/article/details/82902902
问题3 java.lang.IllegalArgumentException: Illegal pattern component: XXX
读取csv文件出错 java.lang.IllegalArgumentException: Illegal pattern component: XXX
def main(args: Array[String]): Unit = {
val rdd = spark.read
.option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ")
.csv("d://2.csv")
rdd.show()
}
参考 https://blog.csdn.net/weixin_36714575/article/details/81943540
https://stackoverflow.com/questions/46429616/spark-2-2-illegal-pattern-component-xxx-java-lang-illegalargumentexception-ill