解决springboot2.x整合mongodb下载50MB文件pending6秒

问题描述：从线上服务器下载50MB文件需要pending6秒，无法忍受

场景：LZ正欢快的敲着代码，同事过来说springboot1.x整合mongodb响应很快，大概50MB pending100ms左右，但是2.x之后就pending6秒

Springboot2.xmongodb下载文件简要代码

public void download(String address, String fileName, HttpServletResponse response) {
        try {
            GridFSFile pdfFile = fsTemplate.findOne(new Query(GridFsCriteria.whereFilename().is(address)));
            GridFsResource convert = convert(pdfFile);
            InputStream inputStream = convert.getInputStream();
            byte[] buf = new byte[1024];
            int len;
            while ((len = inputStream.read(buf)) != -1) {
                out.write(buf, 0, len);
            }
            contentType = one.getContentType();
        } catch (Exception e) {
            out.println("该文件没有contentType");
        }
            }

可以看到最终从mongodb里面拿出流，然后循环读流把文件写出去，很慢，但是springboot1.X时很快就可以响应

Spring boot1.x整合mongodb下载文件简要代码

private void sendFile(String id, String type, HttpServletResponse httpServletResponse) {

    GridFSDBFile gridFSDBFile = MongoFileUtil.findFileById(id);

    if (gridFSDBFile != null) {

        try {

            if (StringUtils.isNotEmpty(gridFSDBFile.getContentType())) {

                httpServletResponse.setContentType(MediaType.parseMediaType(gridFSDBFile.getContentType()).toString());

            } else {

                httpServletResponse.setContentType(MediaType.APPLICATION_OCTET_STREAM_VALUE);

            }

            if (StringUtils.isEmpty(type) || !type.equals("show") || StringUtils.isEmpty(gridFSDBFile.getContentType())) {

                httpServletResponse.setHeader("Connection", "close");

                //强制下载配置

                httpServletResponse.addHeader("Content-Disposition", "attachment;filename=" + URLEncoder.encode(gridFSDBFile.getFilename().trim(), "utf-8"));

            }



            if (StringUtils.isNotEmpty(type) && type.equals("show") && gridFSDBFile.getContentType().contains("image")) {

                httpServletResponse.setHeader("Cache-Control", "max-age=600");

            }



            ServletOutputStream out = httpServletResponse.getOutputStream();

            gridFSDBFile.writeTo(out);

            out.flush();

            out.close();

        } catch (IOException e) {

            e.printStackTrace();

        }

    }

}

可以看到是从mongodb拿到一个GridFSDBFile对象写出去为什么1.x很快就能响应但是升级2.x之后就那么慢呢？

我们看看2.x读数组所用的方法：

GridFSDownloadStreamImpl 的read方法

public int read(final byte[] b) {

    return read(b, 0, b.length);

}



public int read(final byte[] b, final int off, final int len) {

    checkClosed();



    if (currentPosition == length) {

        return -1;


      //这就是为什么第一次很慢 以后就很快的原因  buffer不为null就表示之前已经把数组读到byte数组了
    } else if (buffer == null) {

        buffer = getBuffer(chunkIndex);

    } else if (bufferOffset == buffer.length) {

        chunkIndex += 1;

        buffer = getBuffer(chunkIndex);

        bufferOffset = 0;

    }



    int r = Math.min(len, buffer.length - bufferOffset);
    //把需要的 off - r复制出去

    System.arraycopy(buffer, bufferOffset, b, off, r);

    bufferOffset += r;

    currentPosition += r;

    return r;

}

可以看到大致流程是：从buffer数组里面coppy从开始到结束

System.arraycopy方式是native方法那么每读取一个byte数组是很快的，从debug看第一次循环的时候时间久，
以后每次都很快，而且第一次读之后接口就会有响应就是会有保存窗口弹出，说明服务器有响应了。那么问题出在哪里了呢？
从流程分析第一次读很慢之后很快，那么有可能第一次获取buffer数组的方法比较慢，
我们来研究一下buffer = getBuffer(chunkIndex);

private byte[] getBuffer(final int chunkIndexToFetch) {

    return getBufferFromChunk(getChunk(chunkIndexToFetch), chunkIndexToFetch);

}

private byte[] getBufferFromChunk(final Document chunk, final int expectedChunkIndex) {



    if (chunk == null || chunk.getInteger("n") != expectedChunkIndex) {

        throw new MongoGridFSException(format("Could not find file chunk for file_id: %s at chunk index %s.",

                fileId, expectedChunkIndex));

    }



    if (!(chunk.get("data") instanceof Binary)) {

        throw new MongoGridFSException("Unexpected data format for the chunk");

    }

    byte[] data = chunk.get("data", Binary.class).getData();



    long expectedDataLength = 0;

    boolean extraChunk = false;

    if (expectedChunkIndex + 1 > numberOfChunks) {

        extraChunk = true;

    } else if (expectedChunkIndex + 1 == numberOfChunks) {

        expectedDataLength = length - (expectedChunkIndex * (long) chunkSizeInBytes);

    } else {

        expectedDataLength = chunkSizeInBytes;

    }



    if (extraChunk && data.length > expectedDataLength) {

        throw new MongoGridFSException(format("Extra chunk data for file_id: %s. Unexpected chunk at chunk index %s."

                + "The size was %s and it should be %s bytes.", fileId, expectedChunkIndex, data.length, expectedDataLength));

    } else if (data.length != expectedDataLength) {

        throw new MongoGridFSException(format("Chunk size data length is not the expected size. "

                        + "The size was %s for file_id: %s chunk index %s it should be %s bytes.",

                data.length, fileId, expectedChunkIndex, expectedDataLength));

    }

    return data;

}



public byte[] getData() {

    return (byte[])this.data.clone();

}

private Document getChunk(final int startChunkIndex) {

    if (cursor == null) {

        cursor = getCursor(startChunkIndex);

    }

    Document chunk = null;

    if (cursor.hasNext()) {

        chunk = cursor.next();

        if (batchSize == 1) {

            discardCursor();

        }

        if (chunk.getInteger("n") != startChunkIndex) {

            throw new MongoGridFSException(format("Could not find file chunk for file_id: %s at chunk index %s.",

                    fileId, startChunkIndex));

        }

    }



    return chunk;

}

 

private MongoCursor<Document> getCursor(final int startChunkIndex) {

    FindIterable<Document> findIterable;

    Document filter = new Document("files_id", fileId).append("n", new Document("$gte", startChunkIndex));

    if (clientSession != null) {

        findIterable = chunksCollection.find(clientSession, filter);

    } else {

        findIterable = chunksCollection.find(filter);

    }

    return findIterable.batchSize(batchSize).sort(new Document("n", 1)).iterator();

}

getBuffer/getBufferFromChunk方法没有什么东西 只是把byte数组拿出来，最多是数组进行了clone也是本地方法

下面继续看getCursor方法漏出来了

看到这里我们就需要大致了解一下mongodb怎么存储文件的了，分块(Chunk)，就是把文件弄成一块，一块的，至于每块的大小mongodb有默认值

private static final int DEFAULT_CHUNKSIZE_BYTES = 255 * 1024;

getCursor可以大致看下把这个文件所有的‘块’组织成一个buffer数组(byte数组)。哦，所以第一次很慢，以后每次buffer不为null就直接coppy出来就ok了。

那么spring1.x为什么响应很快呢?

我们看writeTo方法

public long writeTo(final OutputStream out) throws IOException {

    int nc = numChunks();

    for (int i = 0; i < nc; i++) {

        out.write(getChunk(i));

    }

    return length;

}

private byte[] getChunk(final int chunkNumber) {

    if (fs == null) {

        throw new IllegalStateException("No GridFS instance defined!");

    }


    //每次把一块拿出来
    DBObject chunk = fs.getChunksCollection().findOne(new BasicDBObject("files_id", id).append("n", chunkNumber));

    if (chunk == null) {

        throw new MongoException("Can't find a chunk!  file id: " + id + " chunk: " + chunkNumber);

    }



    return (byte[]) chunk.get("data");

}

每次把每一块的byte数组写出去就ok了，so响应嘎嘎的

 

那么springboot2.x之后就没有类似1.x这样的写法了吗？并不是，只是我们没有找打而已

@Autowired
private GridFS gridFS;

GridFSDBFile one = gridFS.findOne(new ObjectId(address));

one.writeTo(outputStream);

 

只是获取GridFS对象

@Configuration
public class MongoDbConfig {

    @Resource
    private MongoDbFactory mongoDbFactory;

    @Bean
    public GridFS getGridFS(){
        DB db = mongoDbFactory.getLegacyDb();
        return new GridFS(db);
    }
}