spark No more replicas available for rdd异常

原创 2018年04月17日 17:52:02

运行一个客户端模式spark程序出现No more replicas available for rdd异常,如下:

18-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 507), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 594), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 406), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 525), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Resubmitted ShuffleMapTask(11, 43), so marking it as still running
2018-04-16 18:01:53,750 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Executor lost: 34 (epoch 34)
2018-04-16 18:01:53,750 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Trying to remove executor 34 from BlockManagerMaster.
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_180 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_183 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_184 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_181 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_182 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_188 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_185 !
2018-04-16 18:01:53,751 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_186 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_178 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_299 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_179 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_194 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_195 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_192 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_193 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_198 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_199 !
2018-04-16 18:01:53,752 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_196 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_189 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_53 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_54 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_55 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_56 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_57 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_50 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_51 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_52 !
2018-04-16 18:01:53,753 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_43 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_44 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_45 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_46 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_47 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_49 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_40 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_75 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_76 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_79 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_71 !
2018-04-16 18:01:53,754 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_72 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_74 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_64 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_65 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_66 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_67 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_69 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_60 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_63 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_2 !
2018-04-16 18:01:53,755 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_17 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_18 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_19 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_5 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_6 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_7 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_8 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_9 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_10 !
2018-04-16 18:01:53,756 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_11 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_12 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_14 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_16 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_602 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_603 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_600 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_601 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_606 !
2018-04-16 18:01:53,757 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_607 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_604 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_0 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_608 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_1 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_609 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_39 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_31 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_32 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_33 !
2018-04-16 18:01:53,758 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_34 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_35 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_36 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_37 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_38 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_29 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_20 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_21 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_22 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_23 !
2018-04-16 18:01:53,759 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_24 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_25 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_510 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_511 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_514 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_515 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_513 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_518 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_519 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_516 !
2018-04-16 18:01:53,760 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_517 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_521 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_400 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_401 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_404 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_525 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_526 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_405 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_523 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_524 !
2018-04-16 18:01:53,761 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_403 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_408 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_529 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_409 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_406 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_527 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_407 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_610 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_614 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_612 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_615 !
2018-04-16 18:01:53,762 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_616 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_500 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_503 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_504 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_502 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_507 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_508 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_509 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_97 !
2018-04-16 18:01:53,763 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_98 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_99 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_561 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_562 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_441 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_560 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_312 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_554 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_434 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_431 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_311 !
2018-04-16 18:01:53,764 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_432 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_316 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_93 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_437 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_558 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_94 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_559 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_317 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_556 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_95 !
2018-04-16 18:01:53,765 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_96 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_557 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_315 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_436 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_439 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_318 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_319 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_86 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_87 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_89 !
2018-04-16 18:01:53,766 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_451 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_572 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_330 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_331 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_573 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_210 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_452 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_571 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_450 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_323 !
2018-04-16 18:01:53,767 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_202 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_445 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_324 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_566 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_203 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_200 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_321 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_564 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_322 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_81 !
2018-04-16 18:01:53,768 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_82 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_327 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_569 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_206 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_449 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_328 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_83 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_84 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_567 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_205 !
2018-04-16 18:01:53,769 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_326 !
2018-04-16 18:01:53,770 WARN [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:66] No more replicas available for rdd_329_447 !
....
available for rdd_329_146 !
2018-04-16 18:01:53,794 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Removing block manager BlockManagerId(34, 10.42.123.220, 41352, None)
2018-04-16 18:01:53,795 INFO [dag-scheduler-event-loop] o.a.s.s.BlockManagerMaster [Logging.scala:54] Removed 34 successfully in removeExecutor
2018-04-16 18:01:53,795 INFO [dag-scheduler-event-loop] o.a.s.s.DAGScheduler [Logging.scala:54] Shuffle files lost for executor: 34 (epoch 34)
2018-04-16 18:01:53,795 INFO [dag-scheduler-event-loop] o.a.s.s.ShuffleMapStage [Logging.scala:54] ShuffleMapStage 11 is now unavailable on executor 34 (0/1320, false)
2018-04-16 18:01:53,955 INFO [dispatcher-event-loop-11] o.a.s.d.c.StandaloneAppClient$ClientEndpoint [Logging.scala:54] Executor updated: app-20180416092531-0001/34 is now EXITED (Command exited with code 137)
2018-04-16 18:01:53,956 INFO [dispatcher-event-loop-11] o.a.s.s.c.StandaloneSchedulerBackend [Logging.scala:54] Executor app-20180416092531-0001/34 removed: Command exited with code 137
2018-04-16 18:01:53,956 INFO [dispatcher-event-loop-11] o.a.s.s.BlockManagerMaster [Logging.scala:54] Removal of executor 34 requested
2018-04-16 18:01:53,957 INFO [dispatcher-event-loop-9] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Trying to remove executor 34 from BlockManagerMaster.
2018-04-16 18:01:53,957 INFO [dispatcher-event-loop-11] o.a.s.s.c.CoarseGrainedSchedulerBackend$DriverEndpoint [Logging.scala:54] Asked to remove non-existent executor 34
2018-04-16 18:01:53,958 INFO [dispatcher-event-loop-0] o.a.s.d.c.StandaloneAppClient$ClientEndpoint [Logging.scala:54] Executor added: app-20180416092531-0001/35 on worker-20180416090258-10.42.123.220-35640 (10.42.123.220:35640) with 8 cores
2018-04-16 18:01:53,958 INFO [dispatcher-event-loop-0] o.a.s.s.c.StandaloneSchedulerBackend [Logging.scala:54] Granted executor ID app-20180416092531-0001/35 on hostPort 10.42.123.220:35640 with 8 cores, 4.0 GB RAM
2018-04-16 18:01:54,143 INFO [dispatcher-event-loop-1] o.a.s.d.c.StandaloneAppClient$ClientEndpoint [Logging.scala:54] Executor updated: app-20180416092531-0001/35 is now RUNNING
2018-04-16 18:02:41,526 INFO [dispatcher-event-loop-0] o.a.s.s.c.CoarseGrainedSchedulerBackend$DriverEndpoint [Logging.scala:54] Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.42.123.220:46230) with ID 35
2018-04-16 18:02:41,529 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 190.19 in stage 11.0 (TID 14490, 10.42.123.220, executor 35, partition 190, PROCESS_LOCAL, 213702 bytes)
2018-04-16 18:02:41,531 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 102.16 in stage 11.0 (TID 14491, 10.42.123.220, executor 35, partition 102, PROCESS_LOCAL, 209783 bytes)
2018-04-16 18:02:41,534 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 577.10 in stage 11.0 (TID 14492, 10.42.123.220, executor 35, partition 577, PROCESS_LOCAL, 259277 bytes)
2018-04-16 18:02:41,536 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 302.18 in stage 11.0 (TID 14493, 10.42.123.220, executor 35, partition 302, PROCESS_LOCAL, 260706 bytes)
2018-04-16 18:02:41,538 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 592.5 in stage 11.0 (TID 14494, 10.42.123.220, executor 35, partition 592, PROCESS_LOCAL, 256686 bytes)
2018-04-16 18:02:41,541 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 310.19 in stage 11.0 (TID 14495, 10.42.123.220, executor 35, partition 310, PROCESS_LOCAL, 261343 bytes)
2018-04-16 18:02:41,543 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 343.13 in stage 11.0 (TID 14496, 10.42.123.220, executor 35, partition 343, PROCESS_LOCAL, 257803 bytes)
2018-04-16 18:02:41,545 INFO [dispatcher-event-loop-0] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 191.13 in stage 11.0 (TID 14497, 10.42.123.220, executor 35, partition 191, PROCESS_LOCAL, 211660 bytes)
2018-04-16 18:02:41,862 INFO [dispatcher-event-loop-7] o.a.s.s.BlockManagerMasterEndpoint [Logging.scala:54] Registering block manager 10.42.123.220:35262 with 2004.6 MB RAM, BlockManagerId(35, 10.42.123.220, 35262, None)
2018-04-16 18:02:45,758 INFO [dispatcher-event-loop-3] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added broadcast_11_piece0 in memory on 10.42.123.220:35262 (size: 22.9 KB, free: 2004.6 MB)
2018-04-16 18:02:50,493 INFO [dispatcher-event-loop-10] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_310 in memory on 10.42.123.220:35262 (size: 128.1 KB, free: 2004.5 MB)
2018-04-16 18:02:50,506 INFO [dispatcher-event-loop-8] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_102 in memory on 10.42.123.220:35262 (size: 91.9 KB, free: 2004.4 MB)
2018-04-16 18:02:50,509 INFO [dispatcher-event-loop-8] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_191 in memory on 10.42.123.220:35262 (size: 95.6 KB, free: 2004.3 MB)
2018-04-16 18:02:50,513 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_592 in memory on 10.42.123.220:35262 (size: 121.8 KB, free: 2004.2 MB)
2018-04-16 18:02:50,517 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_343 in memory on 10.42.123.220:35262 (size: 124.4 KB, free: 2004.0 MB)
2018-04-16 18:02:50,517 INFO [dispatcher-event-loop-4] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_190 in memory on 10.42.123.220:35262 (size: 96.0 KB, free: 2003.9 MB)
2018-04-16 18:02:50,521 INFO [dispatcher-event-loop-0] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_302 in memory on 10.42.123.220:35262 (size: 127.6 KB, free: 2003.8 MB)
2018-04-16 18:02:50,523 INFO [dispatcher-event-loop-5] o.a.s.s.BlockManagerInfo [Logging.scala:54] Added rdd_329_577 in memory on 10.42.123.220:35262 (size: 123.6 KB, free: 2003.7 MB)
2018-04-16 18:02:55,474 INFO [dispatcher-event-loop-9] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 43.29 in stage 11.0 (TID 14498, 10.42.123.220, executor 35, partition 43, PROCESS_LOCAL, 213324 bytes)
2018-04-16 18:02:55,475 INFO [task-result-getter-3] o.a.s.s.TaskSetManager [Logging.scala:54] Finished task 343.13 in stage 11.0 (TID 14496) in 13934 ms on 10.42.123.220 (executor 35) (1/1320)
2018-04-16 18:02:55,480 INFO [dispatcher-event-loop-9] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 525.27 in stage 11.0 (TID 14499, 10.42.123.220, executor 35, partition 525, PROCESS_LOCAL, 258797 bytes)
2018-04-16 18:02:55,480 INFO [task-result-getter-1] o.a.s.s.TaskSetManager [Logging.scala:54] Finished task 577.10 in stage 11.0 (TID 14492) in 13949 ms on 10.42.123.220 (executor 35) (2/1320)
2018-04-16 18:02:55,485 INFO [dispatcher-event-loop-9] o.a.s.s.TaskSetManager [Logging.scala:54] Starting task 406.23 in stage 11.0 (TID 14500, 10.42.123.220, executor 35, partition 406, PROCESS_LOCAL, 259253 bytes)
2018-04-16 18:02:55,486 INFO [task-result-getter-0] o.a.s.s.TaskSetManager [Loggin

主要是driver日志中出现了
No more replicas available for rdd 、Resubmitted ShuffleMapTask(任务反复提交)、
Shuffle files lost for executor
等异常信息而且job一直不出结果。

查看spark UI
Running Applications > 某个应用 > Applcation Detail UI > stages > tasks 某个status为失败的任务

ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

原因:
这里遇到的问题主要是因为数据源数据量过大,而机器的内存无法满足需求,导致长时间执行超时断开的情况,数据无法有效进行交互计算,因此有必要增加内存

具体解决办法:spark应用宿主机上部署了多个虚拟机,关停其中空闲的虚拟机,给运行spark的虚拟机腾出内存空间

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/zhujianlin1990/article/details/79978221

CBuilder中使用管道技术实例

废话不多说,大家来看代码bool __fastcall TForm1::RunCmd(AnsiString cmd,TStringList *stringlist){    TMemoryStream...
  • pp616
  • pp616
  • 2003-07-02 15:59:00
  • 1107

spark 2.1 BlockManagerMasterEndpoint

Member fields.class BlockManagerMasterEndpoint( override val rpcEnv: RpcEnv, val isLocal: Bo...
  • houzhizhen
  • houzhizhen
  • 2017-05-11 14:54:39
  • 211

spark学习-75-源代码:Endpoint模型介绍(6)-Endpoint的消息的接收(2)

关于Endpoint如何处理消息的,我一直找不到初始调用点 请问这个图中消息接受这个点在哪里?我想从这里看,但是找不到最初的起点 1。消息的产生点 没找到,消息是怎么产生的还不清楚,知道...
  • qq_21383435
  • qq_21383435
  • 2018-01-08 16:51:03
  • 99

BlockManagerMaster对BlockManager的管理

BlockManagerMaster是在SparkEnv中创建的,负责对Block的管理和协调,具体操作依赖于BlockManagerMasterEndpoint。Drive和Executor处理Bl...
  • u010110208
  • u010110208
  • 2016-09-12 06:50:50
  • 1256

Spark中的错误处理

从网路冷眼的微博上看到这一篇文: Try again, Apache Spark!, 主要解释了为何Spark的函数式和异步使得错误处理过程更加复杂,读罢受益匪浅,简单翻译并加入自己的见解, 分享一...
  • zrc199021
  • zrc199021
  • 2016-09-30 15:24:59
  • 3253

spark内核揭秘-09-RDD的count操作 触发Job全生命周期-01

spark内核揭秘-09-RDD的count操作 触发Job全生命周期-01 spark内核揭秘-09-RDD的count操作 触发Job全生命周期-01 spark内核揭秘-09-RDD的count...
  • stark_summer
  • stark_summer
  • 2015-01-20 16:10:51
  • 17441

spark RDD算子(九)之基本的Action操作 first, take, collect, count, countByValue, reduce, aggregate, fold,top

first, take , collect, count, countByValue, reduce, aggregate , fold, top, takeOrdered, foreach 等Ac...
  • T1DMzks
  • T1DMzks
  • 2017-04-24 23:37:22
  • 1597

Spark入门:在Intellij IDEA上单机运行Spark的RRD的map和filter

map是对RDD数据集里面每个数据都进行括号里面的操作。 filter是对RDD数据集进行过滤,符合括号里面的操作的,留下来...
  • xuyaoqiaoyaoge
  • xuyaoqiaoyaoge
  • 2015-07-14 21:01:40
  • 1853

Spark RDD基础(二)之常见的转化操作和行动操作及持久化

1.基本RDD操作1.1元素转化操作map() map接受一个函数,把这个函数用于RDD的每个元素,将函数的返回结果作为结果RDD中对应元素的值,map的返回值类型不需要和输入类型一样#计算RDD中...
  • JavaMoo
  • JavaMoo
  • 2017-08-27 20:05:30
  • 584

Spark RDD使用详解1--RDD原理

在集群背后,有一个非常重要的分布式数据架构,即弹性分布式数据集(Resilient Distributed Dataset,RDD)。它是一种有容错机制的特殊集合,可以分布在集群的节点上,以函数式编操...
  • guohecang
  • guohecang
  • 2016-06-22 19:34:07
  • 16424
收藏助手
不良信息举报
您举报文章:spark No more replicas available for rdd异常
举报原因:
原因补充:

(最多只允许输入30个字)