Spark maven运行出错

楔子

spark 运行提示io.netty.buffer.PooledByteBufAllocator.metric()
查找提示netty 版本 冲突

问题1 io.netty.buffer.PooledByteBufAllocator.metric()

pom解决冲突

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.17.Final</version>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty</artifactId>
            <version>3.9.9.Final</version>
        </dependency>
    </dependencies>
</dependencyManagement>

问题2 .NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/Inpu

kafka-client·依赖，exclude掉lz4的依赖

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.11.0.2</version>
    <exclusions>
        <exclusion>
            <groupId>net.jpountz.lz4</groupId>
            <artifactId>lz4</artifactId>
        </exclusion>
    </exclusions>
</dependency>

参考 https://blog.csdn.net/wuhuimin900/article/details/82902902

问题3 java.lang.IllegalArgumentException: Illegal pattern component: XXX

读取csv文件出错 java.lang.IllegalArgumentException: Illegal pattern component: XXX

def main(args: Array[String]): Unit = {

  val rdd = spark.read
    .option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ")
    .csv("d://2.csv")
  rdd.show()

}

参考 https://blog.csdn.net/weixin_36714575/article/details/81943540
https://stackoverflow.com/questions/46429616/spark-2-2-illegal-pattern-component-xxx-java-lang-illegalargumentexception-ill

参考

https://blog.csdn.net/u011669700/article/details/80195886