记一次pyspark运行错误，报YarnClientSchedulerBackend失败
他这个错误主要是RPC消息lost，然后executor lost。
我打开yarn管理界面8088，看到主节点是不健康的节点，看他为什么不健康，进入到webui告诉我的log里面，看到可能是因为磁盘太小。
然后我清空了yum缓存，删除了mysql，扩张到了5G的大小。
然后就一切顺利了。
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
21/09/29 11:33:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
21/09/29 11:33:06 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
21/09/29 11:33:25 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state UNDEFINED!
Check the YARN application logs for more details.
21/09/29 11:33:25 ERROR YarnClientSchedulerBackend: Diagnostics message: Shutdown hook called before final status was reported.
21/09/29 11:33:25 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:745)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.(SparkContext.scala:560)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
py4j.Gateway.invoke(Gateway.java:238)
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
py4j.GatewayConnection.run(GatewayConnection.java:238)
java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from /192.168.87.112:38424 is closed
21/09/29 11:33:25 ERROR YarnSchedulerBackend
Y
a
r
n
S
c
h
e
d
u
l
e
r
E
n
d
p
o
i
n
t
:
S
e
n
d
i
n
g
R
e
q
u
e
s
t
E
x
e
c
u
t
o
r
s
(
0
,
0
,
M
a
p
(
)
,
S
e
t
(
)
)
t
o
A
M
w
a
s
u
n
s
u
c
c
e
s
s
f
u
l
j
a
v
a
.
i
o
.
I
O
E
x
c
e
p
t
i
o
n
:
C
o
n
n
e
c
t
i
o
n
f
r
o
m
/
192.168.87.112
:
38424
c
l
o
s
e
d
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
c
l
i
e
n
t
.
T
r
a
n
s
p
o
r
t
R
e
s
p
o
n
s
e
H
a
n
d
l
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
T
r
a
n
s
p
o
r
t
R
e
s
p
o
n
s
e
H
a
n
d
l
e
r
.
j
a
v
a
:
146
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
s
e
r
v
e
r
.
T
r
a
n
s
p
o
r
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
T
r
a
n
s
p
o
r
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
.
j
a
v
a
:
108
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
j
a
v
a
:
75
)
a
t
i
o
.
n
e
t
t
y
.
h
a
n
d
l
e
r
.
t
i
m
e
o
u
t
.
I
d
l
e
S
t
a
t
e
H
a
n
d
l
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
I
d
l
e
S
t
a
t
e
H
a
n
d
l
e
r
.
j
a
v
a
:
277
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
j
a
v
a
:
75
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
j
a
v
a
:
75
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
u
t
i
l
.
T
r
a
n
s
p
o
r
t
F
r
a
m
e
D
e
c
o
d
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
T
r
a
n
s
p
o
r
t
F
r
a
m
e
D
e
c
o
d
e
r
.
j
a
v
a
:
182
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
D
e
f
a
u
l
t
C
h
a
n
n
e
l
P
i
p
e
l
i
n
e
YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful java.io.IOException: Connection from /192.168.87.112:38424 closed at org.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146) at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.DefaultChannelPipeline
YarnSchedulerEndpoint:SendingRequestExecutors(0,0,Map(),Set())toAMwasunsuccessfuljava.io.IOException:Connectionfrom/192.168.87.112:38424closedatorg.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146)atorg.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atorg.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.DefaultChannelPipelineHeadContext.channelInactive(DefaultChannelPipeline.java:1354)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor
5.
r
u
n
(
S
i
n
g
l
e
T
h
r
e
a
d
E
v
e
n
t
E
x
e
c
u
t
o
r
.
j
a
v
a
:
858
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
T
h
r
e
a
d
F
a
c
t
o
r
y
5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory
5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 ERROR Utils: Uncaught exception in thread YARN application state monitor
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils
.
a
w
a
i
t
R
e
s
u
l
t
(
T
h
r
e
a
d
U
t
i
l
s
.
s
c
a
l
a
:
226
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
r
p
c
.
R
p
c
T
i
m
e
o
u
t
.
a
w
a
i
t
R
e
s
u
l
t
(
R
p
c
T
i
m
e
o
u
t
.
s
c
a
l
a
:
75
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
s
c
h
e
d
u
l
e
r
.
c
l
u
s
t
e
r
.
C
o
a
r
s
e
G
r
a
i
n
e
d
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
r
e
q
u
e
s
t
T
o
t
a
l
E
x
e
c
u
t
o
r
s
(
C
o
a
r
s
e
G
r
a
i
n
e
d
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
c
a
l
a
:
574
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
s
c
h
e
d
u
l
e
r
.
c
l
u
s
t
e
r
.
Y
a
r
n
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
t
o
p
(
Y
a
r
n
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
c
a
l
a
:
98
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
s
c
h
e
d
u
l
e
r
.
c
l
u
s
t
e
r
.
Y
a
r
n
C
l
i
e
n
t
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
t
o
p
(
Y
a
r
n
C
l
i
e
n
t
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
c
a
l
a
:
164
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
s
c
h
e
d
u
l
e
r
.
T
a
s
k
S
c
h
e
d
u
l
e
r
I
m
p
l
.
s
t
o
p
(
T
a
s
k
S
c
h
e
d
u
l
e
r
I
m
p
l
.
s
c
a
l
a
:
581
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
s
c
h
e
d
u
l
e
r
.
D
A
G
S
c
h
e
d
u
l
e
r
.
s
t
o
p
(
D
A
G
S
c
h
e
d
u
l
e
r
.
s
c
a
l
a
:
2040
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
S
p
a
r
k
C
o
n
t
e
x
t
.awaitResult(ThreadUtils.scala:226) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:574) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:98) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164) at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:581) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2040) at org.apache.spark.SparkContext
.awaitResult(ThreadUtils.scala:226)atorg.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)atorg.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:574)atorg.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:98)atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164)atorg.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:581)atorg.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2040)atorg.apache.spark.SparkContext
a
n
o
n
f
u
n
anonfun
anonfunstop
6.
a
p
p
l
y
6.apply
6.applymcV
s
p
(
S
p
a
r
k
C
o
n
t
e
x
t
.
s
c
a
l
a
:
1949
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
u
t
i
l
.
U
t
i
l
s
sp(SparkContext.scala:1949) at org.apache.spark.util.Utils
sp(SparkContext.scala:1949)atorg.apache.spark.util.Utils.tryLogNonFatalError(Utils.scala:1340)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1948)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend
M
o
n
i
t
o
r
T
h
r
e
a
d
.
r
u
n
(
Y
a
r
n
C
l
i
e
n
t
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
c
a
l
a
:
121
)
C
a
u
s
e
d
b
y
:
j
a
v
a
.
i
o
.
I
O
E
x
c
e
p
t
i
o
n
:
C
o
n
n
e
c
t
i
o
n
f
r
o
m
/
192.168.87.112
:
38424
c
l
o
s
e
d
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
c
l
i
e
n
t
.
T
r
a
n
s
p
o
r
t
R
e
s
p
o
n
s
e
H
a
n
d
l
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
T
r
a
n
s
p
o
r
t
R
e
s
p
o
n
s
e
H
a
n
d
l
e
r
.
j
a
v
a
:
146
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
s
e
r
v
e
r
.
T
r
a
n
s
p
o
r
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
T
r
a
n
s
p
o
r
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
.
j
a
v
a
:
108
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
j
a
v
a
:
75
)
a
t
i
o
.
n
e
t
t
y
.
h
a
n
d
l
e
r
.
t
i
m
e
o
u
t
.
I
d
l
e
S
t
a
t
e
H
a
n
d
l
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
I
d
l
e
S
t
a
t
e
H
a
n
d
l
e
r
.
j
a
v
a
:
277
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
j
a
v
a
:
75
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
C
h
a
n
n
e
l
I
n
b
o
u
n
d
H
a
n
d
l
e
r
A
d
a
p
t
e
r
.
j
a
v
a
:
75
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
u
t
i
l
.
T
r
a
n
s
p
o
r
t
F
r
a
m
e
D
e
c
o
d
e
r
.
c
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
T
r
a
n
s
p
o
r
t
F
r
a
m
e
D
e
c
o
d
e
r
.
j
a
v
a
:
182
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
245
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
i
n
v
o
k
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
231
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
f
i
r
e
C
h
a
n
n
e
l
I
n
a
c
t
i
v
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
224
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
D
e
f
a
u
l
t
C
h
a
n
n
e
l
P
i
p
e
l
i
n
e
MonitorThread.run(YarnClientSchedulerBackend.scala:121) Caused by: java.io.IOException: Connection from /192.168.87.112:38424 closed at org.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146) at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.DefaultChannelPipeline
MonitorThread.run(YarnClientSchedulerBackend.scala:121)Causedby:java.io.IOException:Connectionfrom/192.168.87.112:38424closedatorg.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146)atorg.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atorg.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.DefaultChannelPipelineHeadContext.channelInactive(DefaultChannelPipeline.java:1354)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor
5.
r
u
n
(
S
i
n
g
l
e
T
h
r
e
a
d
E
v
e
n
t
E
x
e
c
u
t
o
r
.
j
a
v
a
:
858
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
T
h
r
e
a
d
F
a
c
t
o
r
y
5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory
5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
21/09/29 11:33:37 ERROR Inbox: Ignoring error
java.lang.NullPointerException
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
D
r
i
v
e
r
E
n
d
p
o
i
n
t
DriverEndpoint
DriverEndpoint
a
n
o
n
f
u
n
anonfun
anonfunreceiveAndReply
1.
a
p
p
l
y
O
r
E
l
s
e
(
C
o
a
r
s
e
G
r
a
i
n
e
d
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
c
a
l
a
:
232
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
r
p
c
.
n
e
t
t
y
.
I
n
b
o
x
1.applyOrElse(CoarseGrainedSchedulerBackend.scala:232) at org.apache.spark.rpc.netty.Inbox
1.applyOrElse(CoarseGrainedSchedulerBackend.scala:232)atorg.apache.spark.rpc.netty.Inbox
a
n
o
n
f
u
n
anonfun
anonfunprocess
1.
a
p
p
l
y
1.apply
1.applymcV
s
p
(
I
n
b
o
x
.
s
c
a
l
a
:
105
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
r
p
c
.
n
e
t
t
y
.
I
n
b
o
x
.
s
a
f
e
l
y
C
a
l
l
(
I
n
b
o
x
.
s
c
a
l
a
:
205
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
r
p
c
.
n
e
t
t
y
.
I
n
b
o
x
.
p
r
o
c
e
s
s
(
I
n
b
o
x
.
s
c
a
l
a
:
101
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
r
p
c
.
n
e
t
t
y
.
D
i
s
p
a
t
c
h
e
r
sp(Inbox.scala:105) at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205) at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101) at org.apache.spark.rpc.netty.Dispatcher
sp(Inbox.scala:105)atorg.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)atorg.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)atorg.apache.spark.rpc.netty.DispatcherMessageLoop.run(Dispatcher.scala:221)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor
W
o
r
k
e
r
.
r
u
n
(
T
h
r
e
a
d
P
o
o
l
E
x
e
c
u
t
o
r
.
j
a
v
a
:
624
)
a
t
j
a
v
a
.
l
a
n
g
.
T
h
r
e
a
d
.
r
u
n
(
T
h
r
e
a
d
.
j
a
v
a
:
748
)
21
/
09
/
2911
:
33
:
37
W
A
R
N
T
r
a
n
s
p
o
r
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
:
E
x
c
e
p
t
i
o
n
i
n
c
o
n
n
e
c
t
i
o
n
f
r
o
m
/
192.168.87.113
:
59216
j
a
v
a
.
i
o
.
I
O
E
x
c
e
p
t
i
o
n
:
C
o
n
n
e
c
t
i
o
n
r
e
s
e
t
b
y
p
e
e
r
a
t
s
u
n
.
n
i
o
.
c
h
.
F
i
l
e
D
i
s
p
a
t
c
h
e
r
I
m
p
l
.
r
e
a
d
0
(
N
a
t
i
v
e
M
e
t
h
o
d
)
a
t
s
u
n
.
n
i
o
.
c
h
.
S
o
c
k
e
t
D
i
s
p
a
t
c
h
e
r
.
r
e
a
d
(
S
o
c
k
e
t
D
i
s
p
a
t
c
h
e
r
.
j
a
v
a
:
39
)
a
t
s
u
n
.
n
i
o
.
c
h
.
I
O
U
t
i
l
.
r
e
a
d
I
n
t
o
N
a
t
i
v
e
B
u
f
f
e
r
(
I
O
U
t
i
l
.
j
a
v
a
:
223
)
a
t
s
u
n
.
n
i
o
.
c
h
.
I
O
U
t
i
l
.
r
e
a
d
(
I
O
U
t
i
l
.
j
a
v
a
:
192
)
a
t
s
u
n
.
n
i
o
.
c
h
.
S
o
c
k
e
t
C
h
a
n
n
e
l
I
m
p
l
.
r
e
a
d
(
S
o
c
k
e
t
C
h
a
n
n
e
l
I
m
p
l
.
j
a
v
a
:
380
)
a
t
i
o
.
n
e
t
t
y
.
b
u
f
f
e
r
.
P
o
o
l
e
d
U
n
s
a
f
e
D
i
r
e
c
t
B
y
t
e
B
u
f
.
s
e
t
B
y
t
e
s
(
P
o
o
l
e
d
U
n
s
a
f
e
D
i
r
e
c
t
B
y
t
e
B
u
f
.
j
a
v
a
:
288
)
a
t
i
o
.
n
e
t
t
y
.
b
u
f
f
e
r
.
A
b
s
t
r
a
c
t
B
y
t
e
B
u
f
.
w
r
i
t
e
B
y
t
e
s
(
A
b
s
t
r
a
c
t
B
y
t
e
B
u
f
.
j
a
v
a
:
1106
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
s
o
c
k
e
t
.
n
i
o
.
N
i
o
S
o
c
k
e
t
C
h
a
n
n
e
l
.
d
o
R
e
a
d
B
y
t
e
s
(
N
i
o
S
o
c
k
e
t
C
h
a
n
n
e
l
.
j
a
v
a
:
343
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
n
i
o
.
A
b
s
t
r
a
c
t
N
i
o
B
y
t
e
C
h
a
n
n
e
l
Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21/09/29 11:33:37 WARN TransportChannelHandler: Exception in connection from /192.168.87.113:59216 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) at io.netty.channel.nio.AbstractNioByteChannel
Worker.run(ThreadPoolExecutor.java:624)atjava.lang.Thread.run(Thread.java:748)21/09/2911:33:37WARNTransportChannelHandler:Exceptioninconnectionfrom/192.168.87.113:59216java.io.IOException:Connectionresetbypeeratsun.nio.ch.FileDispatcherImpl.read0(NativeMethod)atsun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)atsun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)atsun.nio.ch.IOUtil.read(IOUtil.java:192)atsun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)atio.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)atio.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)atio.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)atio.netty.channel.nio.AbstractNioByteChannelNioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor
5.
r
u
n
(
S
i
n
g
l
e
T
h
r
e
a
d
E
v
e
n
t
E
x
e
c
u
t
o
r
.
j
a
v
a
:
858
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
T
h
r
e
a
d
F
a
c
t
o
r
y
5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory
5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:43 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state FAILED! Check the YARN application logs for more details.
21/09/29 11:33:43 ERROR YarnClientSchedulerBackend: Diagnostics message: Application application_1632884832055_0005 failed 2 times due to AM Container for appattempt_1632884832055_0005_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://Spark11:8089/proxy/application_1632884832055_0005/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=28319,containerID=container_1632884832055_0005_02_000001] is running beyond virtual memory limits. Current usage: 265.0 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1632884832055_0005_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 28325 28319 28319 28319 (java) 433 138 2359644160 67542 /home/wang/env/jdk1.8.0_191/bin/java -server -Xmx512m -Djava.io.tmpdir=/mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/tmp -Dspark.yarn.app.container.log.dir=/home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg Spark11:40079 --properties-file /mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/spark_conf/spark_conf.properties
|- 28319 28317 28319 28319 (bash) 0 0 115896320 307 /bin/bash -c /home/wang/env/jdk1.8.0_191/bin/java -server -Xmx512m -Djava.io.tmpdir=/mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/tmp -Dspark.yarn.app.container.log.dir=/home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg ‘Spark11:40079’ --properties-file /mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/spark_conf/spark_conf.properties 1> /home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001/stdout 2> /home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
21/09/29 11:33:43 ERROR TransportClient: Failed to send RPC RPC 4670801397555361713 to /192.168.87.112:56042: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel
A
b
s
t
r
a
c
t
U
n
s
a
f
e
.
w
r
i
t
e
(
.
.
.
)
(
U
n
k
n
o
w
n
S
o
u
r
c
e
)
21
/
09
/
2911
:
33
:
43
E
R
R
O
R
Y
a
r
n
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
AbstractUnsafe.write(...)(Unknown Source) 21/09/29 11:33:43 ERROR YarnSchedulerBackend
AbstractUnsafe.write(...)(UnknownSource)21/09/2911:33:43ERRORYarnSchedulerBackendYarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC RPC 4670801397555361713 to /192.168.87.112:56042: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient
R
p
c
C
h
a
n
n
e
l
L
i
s
t
e
n
e
r
.
h
a
n
d
l
e
F
a
i
l
u
r
e
(
T
r
a
n
s
p
o
r
t
C
l
i
e
n
t
.
j
a
v
a
:
357
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
c
l
i
e
n
t
.
T
r
a
n
s
p
o
r
t
C
l
i
e
n
t
RpcChannelListener.handleFailure(TransportClient.java:357) at org.apache.spark.network.client.TransportClient
RpcChannelListener.handleFailure(TransportClient.java:357)atorg.apache.spark.network.client.TransportClientStdChannelListener.operationComplete(TransportClient.java:334)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
at io.netty.channel.AbstractChannel
A
b
s
t
r
a
c
t
U
n
s
a
f
e
.
s
a
f
e
S
e
t
F
a
i
l
u
r
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
.
j
a
v
a
:
987
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
AbstractUnsafe.safeSetFailure(AbstractChannel.java:987) at io.netty.channel.AbstractChannel
AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)atio.netty.channel.AbstractChannelAbstractUnsafe.write(AbstractChannel.java:869)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.access
1900
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
38
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext
1900(AbstractChannelHandlerContext.java:38)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
at io.netty.channel.AbstractChannelHandlerContext
W
r
i
t
e
A
n
d
F
l
u
s
h
T
a
s
k
.
w
r
i
t
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
1128
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) at io.netty.channel.AbstractChannelHandlerContext
WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor
5.
r
u
n
(
S
i
n
g
l
e
T
h
r
e
a
d
E
v
e
n
t
E
x
e
c
u
t
o
r
.
j
a
v
a
:
858
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
T
h
r
e
a
d
F
a
c
t
o
r
y
5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory
5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel
A
b
s
t
r
a
c
t
U
n
s
a
f
e
.
w
r
i
t
e
(
.
.
.
)
(
U
n
k
n
o
w
n
S
o
u
r
c
e
)
21
/
09
/
2911
:
33
:
43
E
R
R
O
R
U
t
i
l
s
:
U
n
c
a
u
g
h
t
e
x
c
e
p
t
i
o
n
i
n
t
h
r
e
a
d
Y
A
R
N
a
p
p
l
i
c
a
t
i
o
n
s
t
a
t
e
m
o
n
i
t
o
r
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
S
p
a
r
k
E
x
c
e
p
t
i
o
n
:
E
x
c
e
p
t
i
o
n
t
h
r
o
w
n
i
n
a
w
a
i
t
R
e
s
u
l
t
:
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
u
t
i
l
.
T
h
r
e
a
d
U
t
i
l
s
AbstractUnsafe.write(...)(Unknown Source) 21/09/29 11:33:43 ERROR Utils: Uncaught exception in thread YARN application state monitor org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils
AbstractUnsafe.write(...)(UnknownSource)21/09/2911:33:43ERRORUtils:UncaughtexceptioninthreadYARNapplicationstatemonitororg.apache.spark.SparkException:ExceptionthrowninawaitResult:atorg.apache.spark.util.ThreadUtils.awaitResult(ThreadUtils.scala:226)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:574)
at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:98)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164)
at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:581)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2040)
at org.apache.spark.SparkContext$
a
n
o
n
f
u
n
anonfun
anonfunstop
6.
a
p
p
l
y
6.apply
6.applymcV
s
p
(
S
p
a
r
k
C
o
n
t
e
x
t
.
s
c
a
l
a
:
1949
)
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
u
t
i
l
.
U
t
i
l
s
sp(SparkContext.scala:1949) at org.apache.spark.util.Utils
sp(SparkContext.scala:1949)atorg.apache.spark.util.Utils.tryLogNonFatalError(Utils.scala:1340)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1948)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend
M
o
n
i
t
o
r
T
h
r
e
a
d
.
r
u
n
(
Y
a
r
n
C
l
i
e
n
t
S
c
h
e
d
u
l
e
r
B
a
c
k
e
n
d
.
s
c
a
l
a
:
121
)
C
a
u
s
e
d
b
y
:
j
a
v
a
.
i
o
.
I
O
E
x
c
e
p
t
i
o
n
:
F
a
i
l
e
d
t
o
s
e
n
d
R
P
C
R
P
C
4670801397555361713
t
o
/
192.168.87.112
:
56042
:
j
a
v
a
.
n
i
o
.
c
h
a
n
n
e
l
s
.
C
l
o
s
e
d
C
h
a
n
n
e
l
E
x
c
e
p
t
i
o
n
a
t
o
r
g
.
a
p
a
c
h
e
.
s
p
a
r
k
.
n
e
t
w
o
r
k
.
c
l
i
e
n
t
.
T
r
a
n
s
p
o
r
t
C
l
i
e
n
t
MonitorThread.run(YarnClientSchedulerBackend.scala:121) Caused by: java.io.IOException: Failed to send RPC RPC 4670801397555361713 to /192.168.87.112:56042: java.nio.channels.ClosedChannelException at org.apache.spark.network.client.TransportClient
MonitorThread.run(YarnClientSchedulerBackend.scala:121)Causedby:java.io.IOException:FailedtosendRPCRPC4670801397555361713to/192.168.87.112:56042:java.nio.channels.ClosedChannelExceptionatorg.apache.spark.network.client.TransportClientRpcChannelListener.handleFailure(TransportClient.java:357)
at org.apache.spark.network.client.TransportClient
S
t
d
C
h
a
n
n
e
l
L
i
s
t
e
n
e
r
.
o
p
e
r
a
t
i
o
n
C
o
m
p
l
e
t
e
(
T
r
a
n
s
p
o
r
t
C
l
i
e
n
t
.
j
a
v
a
:
334
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
0
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
507
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
s
N
o
w
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
481
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
n
o
t
i
f
y
L
i
s
t
e
n
e
r
s
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
420
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
t
r
y
F
a
i
l
u
r
e
(
D
e
f
a
u
l
t
P
r
o
m
i
s
e
.
j
a
v
a
:
122
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
StdChannelListener.operationComplete(TransportClient.java:334) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at io.netty.channel.AbstractChannel
StdChannelListener.operationComplete(TransportClient.java:334)atio.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)atio.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)atio.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)atio.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)atio.netty.channel.AbstractChannelAbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
at io.netty.channel.AbstractChannel
A
b
s
t
r
a
c
t
U
n
s
a
f
e
.
w
r
i
t
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
.
j
a
v
a
:
869
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
D
e
f
a
u
l
t
C
h
a
n
n
e
l
P
i
p
e
l
i
n
e
AbstractUnsafe.write(AbstractChannel.java:869) at io.netty.channel.DefaultChannelPipeline
AbstractUnsafe.write(AbstractChannel.java:869)atio.netty.channel.DefaultChannelPipelineHeadContext.write(DefaultChannelPipeline.java:1316)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.access
1900
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
38
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext
1900(AbstractChannelHandlerContext.java:38)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
at io.netty.channel.AbstractChannelHandlerContext
W
r
i
t
e
A
n
d
F
l
u
s
h
T
a
s
k
.
w
r
i
t
e
(
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
.
j
a
v
a
:
1128
)
a
t
i
o
.
n
e
t
t
y
.
c
h
a
n
n
e
l
.
A
b
s
t
r
a
c
t
C
h
a
n
n
e
l
H
a
n
d
l
e
r
C
o
n
t
e
x
t
WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) at io.netty.channel.AbstractChannelHandlerContext
WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor
5.
r
u
n
(
S
i
n
g
l
e
T
h
r
e
a
d
E
v
e
n
t
E
x
e
c
u
t
o
r
.
j
a
v
a
:
858
)
a
t
i
o
.
n
e
t
t
y
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
D
e
f
a
u
l
t
T
h
r
e
a
d
F
a
c
t
o
r
y
5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory
5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(…)(Unknown Source)
21/09/29 11:33:43 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:745)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.(SparkContext.scala:560)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/shell.py:45: UserWarning: Failed to initialize Spark session.
warnings.warn(“Failed to initialize Spark session.”)
Traceback (most recent call last):
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/shell.py”, line 41, in
spark = SparkSession._create_shell_session()
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/sql/session.py”, line 583, in _create_shell_session
return SparkSession.builder.getOrCreate()
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/sql/session.py”, line 173, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 349, in getOrCreate
SparkContext(conf=conf or SparkConf())
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 118, in init
conf, jsc, profiler_cls)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 180, in _do_init
self._jsc = jsc or self._initialize_context(self._conf._jconf)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 288, in _initialize_context
return self._jvm.JavaSparkContext(jconf)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 1525, in call
answer, self._gateway_client, None, self._fqn)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py”, line 328, in get_return_value
format(target_id, “.”, name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:745)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.(SparkContext.scala:560)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
3万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
