structed streaming基础---跳过的坑

1.spark structed streaming 报如图下面的错误

在pom.xml加这个配置即可

<spark.version>2.3.0</spark.version>
 <scala.version>2.11</scala.version>
 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql-kafka-0-10_${scala.version}</artifactId>
  <version>${spark.version}</version>
 </dependency>

或在maven官网搜:spark－sql－kafka  也可以

第二个问题是运行structed streaming就无缘无故报空指针异常，看下报错日志

18/09/20 18:59:46 ERROR StreamMetadata: Error writing stream metadata StreamMetadata(ea90df83-a9ab-4e23-b03f-90bc3f8581ca) to file:/C:/Users/iwen/AppData/Local/Temp/temporary-c0117371-5f2e-419b-b45c-1dfd7609c644/metadata
java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
......
Exception in thread "main" java.lang.NullPointerException

解决方法

下载这个，放到这个位置(https://github.com/srccodes/hadoop-common-2.2.0-bin)

代码最上面加一句这个即可

System.setProperty("hadoop.home.dir", "C:\\hadoop-common-2.2.0")

 

转载于:https://www.cnblogs.com/kaiwen03/p/9682726.html