Spark Streamning 程序调试

最新推荐文章于 2024-05-03 16:38:30 发布
最新推荐文章于 2024-05-03 16:38:30 发布
阅读量1.4w 收藏 4
点赞数
分类专栏： spark 文章标签： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wankunde/article/details/50843574
版权

OOM

16/03/11 17:19:40 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
16/03/11 17:19:40 ERROR executor.Executor: Exception in task 0.0 in stage 147.0 (TID 343)
java.lang.OutOfMemoryError: Java heap space
    at org.apache.spark.network.netty.NettyBlockTransferService.uploadBlock(NettyBlockTransferService.scala:133)
    at org.apache.spark.network.BlockTransferService.uploadBlockSync(BlockTransferService.scala:118)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$replicate(BlockManager.scala:955)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:860)
    at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645)
    at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:153)
    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
16/03/11 17:19:40 INFO storage.DiskBlockManager: Shutdown hook called
ExecutorLostFailure (executor 10 exited caused by one of the running tasks) Reason: Container marked as failed: container_e02_1457595842637_0021_02_000019 on host: datanode4. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143

Task在执行任务的时候OOM，OOM重试4次仍然失败的话，Application会重启，重启仍然失败，JOB直接FAIL退出。

Receiver 启动失败

16/03/09 15:30:58 INFO receiver.BlockGenerator: Pushed block input-0-1457508633200
16/03/09 15:30:58 INFO storage.MemoryStore: Block input-0-1457508633400 stored as bytes in memory (estimated size 14.9 MB, free 437.9 MB)
16/03/09 15:31:00 INFO receiver.BlockGenerator: Pushed block input-0-1457508633400
16/03/09 15:31:05 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 6690ms for sessionid 0x2519f45091a19f6, closing socket connection and attempting reconnect
16/03/09 15:31:05 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
16/03/09 15:31:05 WARN storage.BlockManager: Putting block input-0-1457508633600 failed
16/03/09 15:31:05 ERROR util.SparkUncaughtExceptionHandler: [Container in shutdown] Uncaught exception in thread Thread[Thread-5,5,main]
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
    at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
    at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
    at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1196)
    at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1202)
    at org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:136)
    at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:173)
    at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:147)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:798)
    at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645)
    at org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:77)
    at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:157)
    at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushArrayBuffer(ReceiverSupervisorImpl.scala:128)
    at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl$$anon$3.onPushBlock(ReceiverSupervisorImpl.scala:109)
	at org.apache.spark.streaming.receiver.BlockGenerator.pushBlock(BlockGenerator.scala:296)
	at org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks(BlockGenerator.scala:268)
    at org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:109)

Receiver 启动的时候OOM了,40s 写了内存440M数据。从日志来看，Receiver 接收数据后开始Push Block,push的时候进行data serializer，这个时候OOM，后期需要考虑使用kryo进行序列化。

Excutor丢失

16/03/09 15:26:57 INFO storage.BlockManagerInfo: Added input-0-1457508369000 on disk on datanode2:40023 (size: 2.8 MB)
16/03/09 15:26:57 INFO cluster.YarnClusterSchedulerBackend: Disabling executor 2.
16/03/09 15:26:57 INFO scheduler.DAGScheduler: Executor lost: 2 (epoch 1)
16/03/09 15:26:57 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
16/03/09 15:26:57 INFO storage.BlockManagerMasterEndpoint: Removing block manager BlockManagerId(2, datanode5, 57965)
16/03/09 15:26:57 INFO storage.BlockManagerMaster: Removed 2 successfully in removeExecutor
16/03/09 15:26:57 INFO yarn.YarnAllocator: Completed container container_1457053973969_0047_01_000003 on host: datanode5 (state: COMPLETE, exit status: 143)
16/03/09 15:26:57 WARN yarn.YarnAllocator: Container marked as failed: container_1457053973969_0047_01_000003 on host: datanode5. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/03/09 15:26:57 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1457053973969_0047_01_000003 on host: datanode5. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/03/09 15:26:57 ERROR cluster.YarnClusterScheduler: Lost executor 2 on datanode5: Container marked as failed: container_1457053973969_0047_01_000003 on host: datanode5. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/03/09 15:26:57 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 2.0 (TID 70, datanode5): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container marked as failed: container_1457053973969_0047_01_000003 on host: datanode5. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

16/03/09 15:26:57 INFO cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 2

这个问题和上面是同一问题，Receiver 丢失导致

Executor申请内存不够

16/03/09 15:28:57 WARN scheduler.TaskSetManager: Lost task 8.0 in stage 8.0 (TID 234, datanode2): java.lang.OutOfMemoryError: Unable to acquire 262144 bytes of memory, got 139413
    at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:91)
    at org.apache.spark.unsafe.map.BytesToBytesMap.allocate(BytesToBytesMap.java:735)
    at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:197)
    at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:212)
    at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.<init>(UnsafeFixedWidthAggregationMap.java:103)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:483)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

16/03/09 15:28:57 INFO storage.BlockManagerInfo: Added input-0-1457508343400 on disk on datanode2:40023 (size: 2.7 MB)
16/03/09 15:28:57 INFO storage.BlockManagerInfo: Added input-0-1457508343600 on disk on datanode2:40023 (size: 3.0 MB)
16/03/09 15:28:57 INFO scheduler.TaskSetManager: Starting task 8.1 in stage 8.0 (TID 236, datanode2, partition 8,NODE_LOCAL, 2006 bytes)
16/03/09 15:28:57 WARN scheduler.TaskSetManager: Lost task 9.0 in stage 8.0 (TID 235, datanode2): java.io.FileNotFoundException: /data/yarn/local/usercache/deployop/appcache/application_1457053973969_0047/blockmgr-e75070dd-691b-43ba-8df6-34a8de5549ce/02/input-0-1457508343800 (No such file or directory)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
    at org.apache.spark.storage.DiskStore.putBytes(DiskStore.scala:47)
    at org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1050)
    at org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1009)
    at org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:456)
	at org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:445)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.storage.MemoryStore.evictBlocksToFreeSpace(MemoryStore.scala:445)
    at org.apache.spark.memory.StorageMemoryPool.shrinkPoolToFreeSpace(StorageMemoryPool.scala:133)
    at org.apache.spark.memory.UnifiedMemoryManager.org$apache$spark$memory$UnifiedMemoryManager$$maybeGrowExecutionPool$1(UnifiedMemoryManager.scala:102)
	at org.apache.spark.memory.UnifiedMemoryManager$$anonfun$acquireExecutionMemory$1.apply$mcVJ$sp(UnifiedMemoryManager.scala:127)
    at org.apache.spark.memory.ExecutionMemoryPool.acquireMemory(ExecutionMemoryPool.scala:114)
    at org.apache.spark.memory.UnifiedMemoryManager.acquireExecutionMemory(UnifiedMemoryManager.scala:126)
    at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:140)
    at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:244)
    at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:112)
    at org.apache.spark.unsafe.map.BytesToBytesMap.acquireNewPage(BytesToBytesMap.java:707)
    at org.apache.spark.unsafe.map.BytesToBytesMap.access$1800(BytesToBytesMap.java:64)
    at org.apache.spark.unsafe.map.BytesToBytesMap$Location.putNewKey(BytesToBytesMap.java:662)
    at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.getAggregationBufferFromUnsafeRow(UnsafeFixedWidthAggregationMap.java:132)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:517)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:686)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

16/03/09 15:28:57 INFO storage.BlockManagerInfo: Added input-0-1457508537400 in memory on datanode2:40169 (size: 4.0 MB, free: 215.6 MB)
16/03/09 15:28:57 INFO scheduler.TaskSetManager: Starting task 9.1 in stage 8.0 (TID 237, datanode2, partition 9,NODE_LOCAL, 2006 bytes)
16/03/09 15:28:57 WARN scheduler.TaskSetManager: Lost task 8.1 in stage 8.0 (TID 236, datanode2): java.lang.AssertionError: assertion failed
    at scala.Predef$.assert(Predef.scala:165)
    at org.apache.spark.memory.UnifiedMemoryManager.acquireExecutionMemory(UnifiedMemoryManager.scala:80)
    at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:140)
    at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:244)
    at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:83)
    at org.apache.spark.unsafe.map.BytesToBytesMap.allocate(BytesToBytesMap.java:735)
    at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:197)
    at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:212)
    at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.<init>(UnsafeFixedWidthAggregationMap.java:103)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:483)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

16/03/09 15:28:57 INFO scheduler.TaskSetManager: Lost task 9.1 in stage 8.0 (TID 237) on executor datanode2: java.lang.AssertionError (assertion failed) [duplicate 1]
16/03/09 15:28:57 INFO scheduler.TaskSetManager: Starting task 9.2 in stage 8.0 (TID 238, datanode2, partition 9,NODE_LOCAL, 2006 bytes)

yarn再次在其他节点上进行重试，仍然失败：

16/03/09 15:28:58 ERROR executor.Executor: Exception in task 9.3 in stage 8.0 (TID 240)
java.lang.AssertionError: assertion failed
    at scala.Predef$.assert(Predef.scala:165)
    at org.apache.spark.memory.UnifiedMemoryManager.acquireExecutionMemory(UnifiedMemoryManager.scala:80)
    at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:140)
    at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:244)
    at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:83)
    at org.apache.spark.unsafe.map.BytesToBytesMap.allocate(BytesToBytesMap.java:735)
    at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:197)
    at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:212)
    at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.<init>(UnsafeFixedWidthAggregationMap.java:103)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:483)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
    at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
16/03/09 15:28:58 INFO util.ShutdownHookManager: Shutdown hook called

RDD在computeOrReadCheckpoint方法中计算方法mapPartitions的时候OOM.

Block input-xxx not found

16/03/09 15:29:10 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 10.0 (TID 242, datanode4): java.lang.Exception: Could not compute split, block input-0-1457508327800 not found
    at org.apache.spark.rdd.BlockRDD.compute(BlockRDD.scala:51)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

yarn不断重试，确认数据丢失后，程序终止

16/03/09 15:29:13 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 24 (MapPartitionsRDD[419] at json at AppBolt.java:77), which has no missing parents
Exception in thread "streaming-job-executor-0" java.lang.Error: java.lang.InterruptedException
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:612)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1952)
    at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1025)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.reduce(RDD.scala:1007)
	at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1.apply(RDD.scala:1136)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.treeAggregate(RDD.scala:1113)
    at org.apache.spark.sql.execution.datasources.json.InferSchema$.infer(InferSchema.scala:65)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4.apply(JSONRelation.scala:114)
	at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4.apply(JSONRelation.scala:109)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation.dataSchema$lzycompute(JSONRelation.scala:109)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation.dataSchema(JSONRelation.scala:108)
    at org.apache.spark.sql.sources.HadoopFsRelation.schema$lzycompute(interfaces.scala:636)
    at org.apache.spark.sql.sources.HadoopFsRelation.schema(interfaces.scala:635)
    at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:37)
    at org.apache.spark.sql.SQLContext.baseRelationToDataFrame(SQLContext.scala:442)
    at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:288)
    at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:275)
    at com.paic.data.cep.app.AppBolt$1.call(AppBolt.java:77)
    at com.paic.data.cep.app.AppBolt$1.call(AppBolt.java:54)
    at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$4.apply(JavaDStreamLike.scala:343)
	at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$4.apply(JavaDStreamLike.scala:343)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
	at scala.util.Try$.apply(Try.scala:161)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    ... 2 more
16/03/09 15:29:13 INFO scheduler.JobScheduler: Stopped JobScheduler

GC overhead limit exceeded

16/03/09 15:34:15 INFO storage.BlockManager: Dropping block input-0-1457508827000 from memory
16/03/09 15:34:15 INFO storage.BlockManager: Writing block input-0-1457508827000 to disk
16/03/09 15:34:15 INFO storage.BlockManager: Dropping block input-0-1457508827200 from memory
16/03/09 15:34:15 INFO storage.BlockManager: Writing block input-0-1457508827200 to disk
16/03/09 15:34:16 INFO storage.MemoryStore: Block input-0-1457508850400 stored as bytes in memory (estimated size 54.6 MB, free 497.9 MB)
16/03/09 15:34:29 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 4140ms for sessionid 0x2519f45091a1a6d, closing socket connection and attempting reconnect
16/03/09 15:34:38 INFO zkclient.ZkClient: zookeeper state changed (Disconnected)
16/03/09 15:34:38 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
16/03/09 15:34:38 WARN server.TransportChannelHandler: Exception in connection from /10.20.9.17:55968
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
    at org.apache.spark.storage.BlockManager$$anonfun$doGetLocal$9.apply(BlockManager.scala:530)
	at org.apache.spark.storage.BlockManager$$anonfun$doGetLocal$9.apply(BlockManager.scala:525)
    at org.apache.spark.storage.MemoryStore.bytes$lzycompute$1(MemoryStore.scala:112)
    at org.apache.spark.storage.MemoryStore.org$apache$spark$storage$MemoryStore$$bytes$1(MemoryStore.scala:112)
	at org.apache.spark.storage.MemoryStore$$anonfun$3.apply(MemoryStore.scala:114)
    at org.apache.spark.storage.MemoryStore$$anonfun$3.apply(MemoryStore.scala:114)
	at org.apache.spark.storage.MemoryStore.tryToPut(MemoryStore.scala:386)
	at org.apache.spark.storage.MemoryStore.putBytes(MemoryStore.scala:114)
	at org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:525)
	at org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:293)
	at org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:58)
    at org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:58)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
	at org.apache.spark.network.netty.NettyBlockRpcServer.receive(NettyBlockRpcServer.scala:58)
	at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:149)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:102)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
16/03/09 15:34:42 INFO zookeeper.ClientCnxn: Opening socket connection to server cnsz141659.app.paic.com.cn/10.25.161.128:2181. Will not attempt to authenticate using SASL (unknown error)
16/03/09 15:34:43 INFO zookeeper.ClientCnxn: Socket connection established to cnsz141659.app.paic.com.cn/10.25.161.128:2181, initiating session
16/03/09 15:34:50 ERROR util.SparkUncaughtExceptionHandler: [Container in shutdown] Uncaught exception in thread Thread[Thread-5,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded
16/03/09 15:34:50 INFO storage.MemoryStore: 1 blocks selected for dropping
16/03/09 15:34:50 INFO storage.BlockManager: Dropping block input-0-1457508827400 from memory
16/03/09 15:34:50 INFO storage.BlockManager: Writing block input-0-1457508827400 to disk
16/03/09 15:34:50 INFO zookeeper.ClientCnxn: Client session timed out, have not heard from server in 7620ms for sessionid 0x2519f45091a1a6d, closing socket connection and attempting reconnect
16/03/09 15:34:50 WARN util.ShutdownHookManager: ShutdownHook '$anon$2' failed, java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at sun.net.www.protocol.jar.Handler.parseContextSpec(Handler.java:206)
	at sun.net.www.protocol.jar.Handler.parseURL(Handler.java:152)
	at java.net.URL.<init>(URL.java:614)
	at java.net.URL.<init>(URL.java:482)
	at sun.misc.URLClassPath$JarLoader.checkResource(URLClassPath.java:757)
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:842)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:239)
    at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:239)
	at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:239)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:239)
    at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:218)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
16/03/09 15:34:50 INFO consumer.SimpleConsumer: Reconnect due to socket error: java.lang.OutOfMemoryError: GC overhead limit exceeded

Oralce Explain :

Exception in thread thread_name: java.lang.OutOfMemoryError: GC Overhead limit exceeded
Cause: The detail message “GC overhead limit exceeded” indicates that the garbage collector is running all the time and Java program is making very slow progress. After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 (compile time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown. This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations.
Action: Increase the heap size. The java.lang.OutOfMemoryError exception for GC Overhead limit exceeded can be turned off with the command line flag -XX:-UseGCOverheadLimit.

我使用的是JDK1.6.0_37和JDK_1.7.0_60版本，这2个版本中JVM默认启动的时候-XX:+UseGCOverheadLimit，即启用了该特性。这其实是JVM的一种推断，如果垃圾回收耗费了98%的时间，但是回收的内存还不到2%，那么JVM会认为即将发生OOM，让程序提前结束。当然我们可以使用-XX:-UseGCOverheadLimit，关掉这个特性。

不太明白为什么JDK要提供这么个参数。当我们遇到这个错误只能说明：要么是内存空间不足，要么是存在内存泄露。这个时候对我们有意义的，其实就是heap dump，我们可以分析堆内存的情况，从而诊断出是否代码存在问题。

我们知道如果在启动JVM的时候设置了如下的参数，那么JVM崩溃的时候会打印出heap dump。

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:/

个人感觉，如果真的发生了”GC overhead limit exceeded”错误，那么其实离实际的OOM已经不远了，所以让JVM做个预测提前结束，感觉意义不大。

这里应该是Receiver在将数据保存到Disk的时候，接收数据的速度过快导致的OOM。

Driver 自动重启

16/03/10 13:28:55 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 416.0 (TID 45932, datanode2, partition 2,NODE_LOCAL, 2017 bytes)
16/03/10 13:28:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 416.0 (TID 45931) in 1868 ms on datanode2 (2/4)
Exception in thread "streaming-job-executor-0" java.lang.Error: java.lang.InterruptedException
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:612)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1314)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.take(RDD.scala:1288)
	at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1416)
    at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1416)
	at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1416)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1415)
    at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:501)
    at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:46)
    at com.paic.data.cep.app.AppBolt$1.call(AppBolt.java:57)
    at com.paic.data.cep.app.AppBolt$1.call(AppBolt.java:54)
    at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$4.apply(JavaDStreamLike.scala:343)
	at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$4.apply(JavaDStreamLike.scala:343)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
	at scala.util.Try$.apply(Try.scala:161)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    ... 2 more
16/03/10 13:28:55 INFO scheduler.JobScheduler: Stopped JobScheduler

在2016/03/10 13:28:00 集中有4个Job失败，失败日志：

java.io.IOException: java.lang.AssertionError: assertion failed
    at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
    at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.AssertionError: assertion failed
    at scala.Predef$.assert(Predef.scala:165)
    at org.apache.spark.memory.UnifiedMemoryManager.acquireStorageMemory(UnifiedMemoryManager.scala:140)
    at org.apache.spark.storage.MemoryStore.tryToPut(MemoryStore.scala:383)
    at org.apache.spark.storage.MemoryStore.tryToPut(MemoryStore.scala:342)
    at org.apache.spark.storage.MemoryStore.putBytes(MemoryStore.scala:99)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:803)
    at org.apache.spark.storage.BlockManager.putBytes(BlockManager.scala:690)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$anonfun$$getRemote$1$1.apply(TorrentBroadcast.scala:130)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$anonfun$$getRemote$1$1.apply(TorrentBroadcast.scala:127)
    at scala.Option.map(Option.scala:145)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.org$apache$spark$broadcast$TorrentBroadcast$$anonfun$$getRemote$1(TorrentBroadcast.scala:127)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$1.apply(TorrentBroadcast.scala:137)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$1.apply(TorrentBroadcast.scala:137)
    at scala.Option.orElse(Option.scala:257)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
    at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
    at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1219)
    ... 12 more

查看Executors，Executor 7 的Address CANNOT FIND ADDRESS，无语了……

wankunde
关注
  • 0
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
专栏目录
ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
DayOneMore的博客
03-06 2745
问题描述 spark分布式运行的过程中，自己有两个worker，但是运行的时候，只有一台worker正常运行，另一台报了右边的错，如下图 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM 把正常的机器停掉，只运行有问题的worker（dl21,dl23）,发现能出结果，但是两台机器的stderr仍然是右边的错，所以奇怪的是，结果怎么出来的？ 网上说是因为资源分配不够，但是我分配了executor 10G资源，反而出现下面这种问题 然后有些
spark常见问题处理
我的博客
08-24 3935
1、spark thriftserver报以下错误，其他诸如hive/sparksql等方式均正常 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriverActorSystem-akka.actor.default-dispatcher-379] shutting down ActorSystem [sparkDriverActorSystem] java.lang.OutOfMemoryError: Java heap
正确解决java.lang.InterruptedException异常的有效解决方法
最新发布
wbajsjhhhhh的博客
05-03 2955
正确解决java.lang.InterruptedException异常的有效解决方法
Spark源码(启动ApplicationMaster和Driver线程)-第二期
Airy的博客
08-23 1257
上一期指路 上一期我们分析到了YarnClusterApplication的start，我们继续从这里分析。 override def start(args: Array[String], conf: SparkConf): Unit = { // SparkSubmit would use yarn cache to distribute files & jars in yarn mode, // so remove them from sparkConf here
ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
qq_33915826的博客
12-05 5663
Spark提交任务执行遇到如下错误： 1.命令行错误： 2.yarn 日志错误：（yarn日志不知如何获取的请参考上一篇博客 yarn日志打印） ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM 19/12/05 11:03:53 INFO netty.NettyBlockTransferService...
hive sql 报错后继续执行_Hive SQL 报错
weixin_36028243的博客
12-31 260
错误SQL:select * from ((select main_brand_code, third_cate_code, user_pinfrom adm.adm_dm_4a_ad_fission_view_midwhere dt = '2019-04-25'union allselect main_brand_code, 999999 as third_cate_code, user_pin...
运行Spark程序的方法
01-07
在启动时便会自动生成一个驱动器程序，所以只需要在Spark shell中输入项运行的操作就可以了 pyspark –master MasterURL –name NAME 在Spark shell中运行语句sc可直接得到驱动器程序的相关信息 jupyter Notebook 是...
spark的远程调试
04-28
本篇文档详细的介绍了spark的远程调试，带你一步一步认识
Java操作Spark入门程序
10-15
使用Java语言编写操作Spark的入门程序，包括Java连接Spark，所需的信息配置开发，连接目标配置Spark集群详细说明
idea远程调试spark的步骤讲解
08-26
"Idea远程调试Spark的步骤讲解" 远程调试Spark是指在集群上运行的Spark项目出现问题，但是在本地无法重现时，通过远程调试的方式来一步一步跟踪代码的过程。下面是远程调试Spark的四步骤： 第一步：jar包拷贝到...
Spark调优 Spark应用程序性能优化
09-21
写完了一个复杂的spark作业之后，进行性能调优的时候，首先第一步，就是要来调节最优的资源配置；在这个基础之上，如果说你的spark作业，能够分配的资源达到了你的能力范围的顶端之后，无法再分配更多的资源了，公司...
hive sql 报错后继续执行_hive insert sql 报错，如何处理？
weixin_32371243的博客
12-31 452
环境：linux centos7hadoop-3.1.3(伪集群部署)hive-3.1.2tez-0.9.2问题：执行insert sql时报错：控制台直接反馈：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskhadoop日志：2019-10-30 17:14:55,529 ...
工作中报错故障集合
qq_43801878的博客
03-15 1580
OOM常见报错排查之堆外内存溢出 ●报错：ExecutorLostFailure (executor xxx exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 12.4 GB of 12 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or d
端口多次重试失败异常
sunlang的博客
06-25 1065
11-08-2020 11:38:48 CST t_uac_user INFO - Starting job t_uac_user at 1597117128685 11-08-2020 11:38:48 CST t_uac_user INFO - job JVM args: '-Dazkaban.flowid=order' '-Dazkaban.execid=130' '-Dazkaban.jobid=t_uac_user' 11-08-2020 11:38:48 CST t_uac_user INFO.
spark error
xugen12的专栏
07-28 2350
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 16/07/28 05:42:58 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT] 16/07/28 05:42:59 W
java shutdown hook_解析Runtime中shutdown hook的使用详解
weixin_32223631的博客
02-16 511
根据 Java API, 所谓 shutdown hook 就是已经初始化但尚未开始执行的线程对象。在Runtime 注册后，如果 jvm 要停止前，这些 shutdown hook 便开始执行。声明：Runtime.addShutdownHook(Thread t)举例如下：package john2;/*** test shutdown hook* All rights released an...
Spark学习：ShutdownHookManager虚拟机关闭钩子管理器
weixin_30782331的博客
01-23 1496
　　Java程序经常也会遇到进程挂掉的情况，一些状态没有正确的保存下来，这时候就需要在JVM关掉的时候执行一些清理现场的代码。 　　JAVA中的ShutdownHook提供了比较好的方案。 　　JDK提供了Java.Runtime.addShutdownHook(Thread hook)方法，可以注册一个JVM关闭的钩子，这个钩子可以在一下几种场景中被调用： １.　程序正常退出 ２.　使用S...
jvm called wls shutdown hook问题
chicha3483的博客
06-21 612
最近因为工作原因接手了另外一个项目组。刚过来就遇到了一个生产运维反馈的问题，环境莫名其妙的断掉。启动了之后，过几个小时或者十几个小时就会断掉。很诡异的一个情况。上去看了下，weblogic进程是没有了。也没看出来是什么问题。就把日志拿下来看了下，日志报错大概如下： <Notice>...
spark sql运行出错（Container killed on request. Exit code is 143）
热门推荐
yijichangkong的专栏
05-06 5万+
错误描述：SQL三张表做join运行出错；用hive执行报错如下： Diagnostic Messages for this Task: Container [pid=27756,containerID=container_1460459369308_5864_01_000570] is running beyond physical memory limits. Current usag
使用IDEA开发Scala程序.docx
06-05
"使用IDEA开发Scala程序，包括创建Spark应用程序，本地及分布式部署" 本文主要介绍了如何使用IntelliJ IDEA开发Spark应用程序，特别是在Scala环境下进行编程。实验目标旨在让学习者掌握IDEA的使用，以及如何开发和...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值