Flink 写入 OSS问题记录

问题描述 Caused by: java.io.IOException: No space left on device

这种问题，一般就是没有存储空间导致的。可以根据这个去找下。

Caused by: java.io.IOException: No space left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:326)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:95)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSOutputStream.write(AliyunOSSOutputStream.java:107)
        at java.io.OutputStream.write(OutputStream.java:116)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.flink.fs.osshadoop.common.HadoopDataOutputStream.write(HadoopDataOutputStream.java:47)
        at java.io.OutputStream.write(OutputStream.java:75)
        at org.apache.flink.core.fs.FSDataOutputStreamWrapper.write(FSDataOutputStreamWrapper.java:60)
        at org.apache.flink.api.java.io.TextOutputFormat.writeRecord(TextOutputFormat.java:106)
        at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.invoke(OutputFormatSinkFunction.java:87)
        at org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:49)
        at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)

Flink读写流程

下图是一个简单的读写流程， 在流程中不难发现，在写入到OSS的过程中，经历了一个缓冲区，这个缓冲区的内存耗尽，导致问题产生。

在读源码的过程中，发现在写入的时候， 会根据配置的 ‘fs.oss.buffer.dir’ 去申请本地的空间。

    public AliyunOSSOutputStream(Configuration conf, AliyunOSSFileSystemStore store, String key, Progressable progress, Statistics statistics) throws IOException {
        this.store = store;
        this.key = key;
        this.progress = progress;
        this.statistics = statistics;
        this.partSizeThreshold = conf.getLong("fs.oss.multipart.upload.threshold", 20971520L);
        if (conf.get("fs.oss.buffer.dir") == null) {
            conf.set("fs.oss.buffer.dir", conf.get("hadoop.tmp.dir") + "/oss");
        }

        this.dirAlloc = new LocalDirAllocator("fs.oss.buffer.dir");
        this.tmpFile = this.dirAlloc.createTmpFileForWrite("output-", -1L, conf);
        this.backupStream = new BufferedOutputStream(new FileOutputStream(this.tmpFile));
        this.closed = false;
    }

解决办法

在flink-conf.yaml 中加入以下配置即可

fs.oss.buffer.dir: /alidata1/oss/tmp/