com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message

1、可能是遇到了查询时间过长的Read,不妨看看profiling或者server log日志的信息,找到查询时间比较长的语句,例如超过10秒的语句,看是否能优化;
2、适当提高socketTimeout一些,看是否能缓解这种出错的情形。
供参考。

业务需求:将mongo库里的全量数据相关信息跑出来写入文件中,供合作方拉取。

业务思路:用DBCursor取得库的连接,移动游标每读5w条数据进行多线程异步写文件操作。

测试库1500多w数据运行没有异常,可是到了正式库3000多w就报如题错误。

后来根据参考1方案优化了Read语句,解决问题。

优化前代码:

 public List<FileInfo> handle(long pageSize, String collectionName, BasicDBObject query, BasicDBObject fields,Date time,String path) {
        DBCursor dbCursor = this.mongoTemplate.getCollection(collectionName).find(query,fields).sort(new BasicDBObject("_id", 1)).limit((int)pageSize);
        List<FileInfo> fileInfos = new LinkedList<FileInfo>();
        String filename = Constants.FILE_NAME;
        int nThreads = Constants.nThreads;
        ExecutorService executorService = Executors.newFixedThreadPool(nThreads);
        while (dbCursor.hasNext()){
            FileInfo fileInfo = new FileInfo();
            fileInfos.add(JavaBeanUtils.dbObject2Bean(dbCursor.next(),fileInfo));
            if (fileInfos.size() == Constants.FLAG_VALUE){//每到5w条写一次到文件
                final  List<FileInfo> fileInfos2 = new LinkedList<FileInfo>(fileInfos);
                fileInfos.clear();
                threadPoolHandler(executorService,fileInfos2,path,filename,time);
            }
        }
        threadPoolHandler(executorService,fileInfos,path,filename,time);
        //判断任务是否结束
        executorService.shutdown();
        while (true){
            if (executorService.isTerminated()){
                endingFlag = true;
                break;
            }
        }

        return null;
    }

发现Read语句中的sort limit字段没有,之前设计是为了分页用,现在没用上(一次取异步写入文件),大大降低查询效率,去掉效率高了很多。
优化后代码:

public List<FileInfo> handle(String collectionName, BasicDBObject query, BasicDBObject fields,Date time,String path) {
        DBCursor dbCursor = this.mongoTemplate.getCollection(collectionName).find(query,fields);
        List<FileInfo> fileInfos = new LinkedList<FileInfo>();
        String filename = Constants.FILE_NAME;
        int nThreads = Constants.nThreads;
        ExecutorService executorService = Executors.newFixedThreadPool(nThreads);
        while (dbCursor.hasNext()){
            FileInfo fileInfo = new FileInfo();
            fileInfos.add(JavaBeanUtils.dbObject2Bean(dbCursor.next(),fileInfo));
            if (fileInfos.size() == Constants.FLAG_VALUE){//每到5w条写一次到文件
                final  List<FileInfo> fileInfos2 = new LinkedList<FileInfo>(fileInfos);
                fileInfos.clear();
                threadPoolHandler(executorService,fileInfos2,path,filename,time);
            }
        }
        threadPoolHandler(executorService,fileInfos,path,filename,time);
        //判断任务是否结束
        executorService.shutdown();
        while (true){
            if (executorService.isTerminated()){
                endingFlag = true;
                break;
            }
        }

        return null;
    }

最终方案:
最终查询mongo添加了有索引能缩小范围的条件createtime和batchSize(如果cursor空闲一定时间后(10分钟),server端是否将其移除,默认为false,即server会将空闲10分钟的cursor移除以节约内存。如果为true,则表示server端不需要移除空闲的cursor,而是等待用户手动关闭),循环createtime多线程获取DBCursor,FLAG_VALUE调整到5000,保证数据及时写入文件

 private void  handle(String collectionName, BasicDBObject query, BasicDBObject fields, boolean isFull) {
        DBCursor dbCursor = this.mongoTemplate.getCollection(collectionName).find(query,fields).batchSize(30);
        List<FileInfo> fileInfos = new CopyOnWriteArrayList<>();
        try {
            while (dbCursor.hasNext()){
                FileInfo fileInfo = new FileInfo();
                fileInfos.add(JavaBeanUtils.dbObject2Bean(dbCursor.next(),fileInfo));
                if (fileInfos.size()%100==0){
                    logger.info("=========我还活着,已经遍历了{}条数据",fileInfos.size());
                }
                if (fileInfos.size() == searchConfig.getFlag_value()){//每到5k条写一次到文件
                    final  List<FileInfo> fileInfos2 = new CopyOnWriteArrayList<FileInfo>(fileInfos);
                    fileInfos.clear();
                    task(fileInfos2,isFull);
                }
            }
            if (fileInfos.size()>0){
                task(fileInfos,isFull);
            }
        } finally {
            //关闭游标
            dbCursor.close();
        }
    }

搞定!!!

  • 2
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
项目突然跑不起来报这个错 Exception in monitor thread while connecting to server 10.18.21.79:27117 com.mongodb.MongoSocketReadException: Exception receiving message at com.mongodb.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:536) at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:421) at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:290) at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) at com.mongodb.connection.CommandHelper.sendAndReceive(CommandHelper.java:84) at com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:34) at com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:91) at com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:51) at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:127) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:114) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.mongodb.connection.SocketStream.read(SocketStream.java:84) at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:547) at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:418)
06-09

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值