SequenceInputStream使用笔记

最近在解决一个程序性能上的一个问题,先大致说一下需求:

将数据库中查询出来的n条明细信息,经过一定的格式生成文件的记录,将记录的流上传到file server. 原来的处理方式是1)将详细信息全部查询出来进行遍历 2) 将其中的每一条进行格式组装,然后将其发送到file server.

思考: 发现如果详细信息数据较多,这样在逐行遍历每一条进行组装会有out of memery 的问题, 于是在想分页读取详细信息,并分页生成格式数据和input stream, 这样的话在送给file server的时候需要送一个input stream,而不是多个,于是我需要将多个input stream 按照顺序聚合成一个input stream, 方法如下:

public static ByteArrayInputStream writeLines(Collection<?> lines) {
        if (CollectionUtils.isEmpty(lines)) {
            return null;
        }
        StringBuilder builder = new StringBuilder();
        builder.append(new String(new byte[]{(byte) 0xEF, (byte) 0xBB, (byte) 0xBF}, StandardCharsets.UTF_8));
        for (final Object line : lines) {
            builder.append(line.toString()).append("\n");
        }
        return new ByteArrayInputStream(builder.toString().getBytes(StandardCharsets.UTF_8));
    }

    public static void main(String[] args) throws IOException {

        //ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(5, 10, 10L, TimeUnit.SECONDS, new ArrayBlockingQueue(100));

        List lines = Lists.newArrayList();
        long begin = System.currentTimeMillis();
        for (int i = 0; i < 1000000; i++) {
            lines.add("aaaaaaa");
            lines.add("bbbbbbb");
        }
        System.out.println("放入的时间是 " + (System.currentTimeMillis() - begin));
        List partition = Lists.partition(lines, 100000);
        System.out.println("begin time is " + begin);

        Vector<InputStream> vector = new Vector<>();
        for (int i = 0; i < partition.size(); i++) {
            List lst = (List) partition.get(i);
            vector.add(writeLines(lst));
        }
        long vectorEnd = System.currentTimeMillis();
        System.out.println("vectorEnd - begin = " + (vectorEnd - begin));

        SequenceInputStream sequenceInputStream = new SequenceInputStream(vector.elements());
        System.out.println("parse stream end = " + (System.currentTimeMillis() - vectorEnd));

        File logFile = new File("C:\\logs\\msg.log");
        if (!logFile.exists()) {
            logFile.createNewFile();
        }
        FileOutputStream fileOutputStream = new FileOutputStream(logFile);
        byte[] b = new byte[2048];
        while ((sequenceInputStream.read(b)) != -1) {
            fileOutputStream.write(b);
        }

        sequenceInputStream.close();
        fileOutputStream.close();

        System.out.println(sequenceInputStream.toString());
        long end = System.currentTimeMillis();
        System.out.println("end - begin :" + (end - begin));
    }

大致测试了一下性能还可以.

这里面原生的用法用到了vector,在java8中,此用法被作为sonar issue, 解决办法是用linkList替换掉vector,如下:

List<InputStream> fileStreams = new LinkedList<>();

fileStreams.add(inputStream);

final Iterator<InputStream> streamIterable = fileStreams.iterator();
            file.setInputStream(new SequenceInputStream(new Enumeration<InputStream>() {
                @Override
                public boolean hasMoreElements() {
                    return streamIterable.hasNext();
                }

                @Override
                public InputStream nextElement() {
                    return streamIterable.next();
                }
            }));

这样写就不会有sonar issues.