Spark Streaming Demo报错 NettyRpcEndpointRef; local class incompatible: stream classdesc serialVersion

最新推荐文章于 2024-04-05 13:00:00 发布

Sail__

最新推荐文章于 2024-04-05 13:00:00 发布

阅读量1.8k

点赞数

分类专栏： Spark 文章标签： Spark InvalidClassException serialVersionUID NettyRpcEndpointRef scala版本

本文链接：https://blog.csdn.net/diaobatian/article/details/102916435

版权

Spark Streaming demo运行时遇到Java和Linux端的NullPointerException及InvalidClassException，原因是Scala版本不一致导致序列化问题。解决办法是确保所有环境使用相同版本的Scala，例如统一为2.11.12。

摘要由CSDN通过智能技术生成

关键词

wordCount Demo运行报错关键词：

java.lang.NullPointerException

org.apache.spark.storage.BlockManagerMaster.registerBlockManager

java.io.InvalidClassException

org.apache.spark.rpc.netty.NettyRpcEndpointRef; local class incompatible: stream classdesc serialVersionUID = -4186747031772874359, local class serialVersionUID = 6257082371135760434

scala 版本

spark-streaming 版本

现象

Java端报错

19/11/05 15:06:05 INFO SparkEnv: Registering OutputCommitCoordinator
19/11/05 15:06:06 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/11/05 15:06:06 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://DD-HP5500:4040
19/11/05 15:06:06 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://axe1:7077...
19/11/05 15:06:06 INFO TransportClientFactory: Successfully created connection to axe1/192.168.86.101:7077 after 46 ms (0 ms spent in bootstraps)
19/11/05 15:06:26 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://axe1:7077...
19/11/05 15:06:46 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://axe1:7077...
19/11/05 15:07:06 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
19/11/05 15:07:06 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
19/11/05 15:07:06 INFO SparkUI: Stopped Spark web UI at http://DD-HP5500:4040
19/11/05 15:07:06 INFO StandaloneSchedulerBackend: Shutting down all executors
19/11/05 15:07:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56271.
19/11/05 15:07:06 INFO NettyBlockTransferService: Server created on DD-HP5500:56271
19/11/05 15:07:06 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/11/05 15:07:06 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/11/05 15:07:06 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
19/11/05 15:07:06 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/11/05 15:07:06 INFO MemoryStore: MemoryStore cleared
19/11/05 15:07:06 INFO BlockManager: BlockManager stopped
19/11/05 15:07:06 INFO BlockManagerMaster: BlockManagerMaster stopped
19/11/05 15:07:06 WARN MetricsSystem: Stoppi