java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read

环境:

- Spark 2.11-2.0.2

- Hadoop 2.6.5


使用Spark连接HDFS,在高并发的时候,偶尔会遇到以下异常:


2017-01-26 10:30:29,079 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: xxx:50010:DataXceiver error processing WRITE_BLOCK operation  src: /xxx:52827 dst: /xxx:50010
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xxx:50010 remote=/xxx:52827]
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    at java.io.DataInputStream.read(DataInputStream.java:132)
    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:496)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:891)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:239)
    at java.lang.Thread.run(Thread.java:662)


出现以上错误是因为datanode的服务线程连接数都被占用,导致Spark等待超时。可以通过以下方法解决:

  • 根据datanode的资源情况适当增加的服务线程数:在hdfs-default.xml配置文件里面新增或修改dfs.datanode.handler.count,默认是10
  • 增加客户端的超时时间dfs.client.socket-timeout,默认是60000ms,注意,旧版本的Hadoop用的是dfs.socket.timeout,可参考http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
例如以下Java代码设置超时时间为300000ms=300s=5分钟:

try (JavaSparkContext context = new JavaSparkContext(new SparkConf().setMaster("local[*]").setAppName("test"))) {
    context.sc().hadoopConfiguration().set("dfs.client.socket-timeout", "300000");
    ......

  • 2
    点赞
  • 13
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值