源码来自:github spark/NioBufferedFileInputStream
NioBufferedFileInputStream是spark实现的一种新的字节流,它既支持内部缓冲区,又支持nio读取文件,使用direct buffer避免java堆与native内存之间的数据拷贝。在Java jdk中没有可供直接使用的具备以上2个功能的字节流。sun.nio.ch.ChannelInputStream虽然支持使用nio读取一个文件,但是不支持缓冲。
NioBufferedFileInputStream的实现方式与BufferedInputStream有点类似,都具有:
fill方法(spark为refill方法)
read方法
skip方法
available方法
NioBufferedFileInputStream不支持BufferedInputStream的markPos标记。使用markPos标记回复到已读数据的某个位点。
可以结合我对BufferedInputStream源码分析一起阅读。
NioBufferedFileInputStream的源码如下:
package org.apache.spark.io;
import org.apache.spark.storage.StorageUtils;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
/**
* {@link InputStream} implementation which uses direct buffer
* to read a file to avoid extra copy of data between Java and
* native memory which happens when using {@link java.io.BufferedInputStream}.
* Unfortunately, this is not something already available in JDK,
* {@link sun.nio.ch.ChannelInputStream} supports reading a file using nio,
* but does not support buffering.
*/
public final class NioBufferedFileInputStream extends InputStream {
private static final int DEFAULT_BUFFER_SIZE_BYTES = 8192;
private final ByteBuffer byteBuffer;
private final FileChannel fileChannel;
public NioBufferedFileInputStream(File file, int bufferSizeInBytes) throws IOException {
byteBuffer = ByteBuffer.allocateDirect(bufferSizeInBytes);
fileChannel = FileChannel.open(file.toPath(), StandardOpenOption.READ);
byteBuffer.flip();
}
public NioBufferedFileInputStream(File file) throws IOException {
this(file, DEFAULT_BUFFER_SIZE_BYTES);
}
/**
* Checks weather data is left to be read from the input stream.
* @return true if data is left, false otherwise
* @throws IOException
*/
private boolean refill() throws IOException {
if (!byteBuffer.hasRemaining()) {
byteBuffer.clear();
int nRead = 0;
while (nRead == 0) {
nRead = fileChannel.read(byteBuffer);
}
if (nRead < 0) {
return false;
}
byteBuffer.flip();
}
return true;
}
@Override
public synchronized int read() throws IOException {
if (!refill()) {
return -1;
}
return byteBuffer.get() & 0xFF;
}
@Override
public synchronized int read(byte[] b, int offset, int len) throws IOException {
if (offset < 0 || len < 0 || offset + len < 0 || offset + len > b.length) {
throw new IndexOutOfBoundsException();
}
if (!refill()) {
return -1;
}
len = Math.min(len, byteBuffer.remaining());
byteBuffer.get(b, offset, len);
return len;
}
@Override
public synchronized int available() throws IOException {
return byteBuffer.remaining();
}
@Override
public synchronized long skip(long n) throws IOException {
if (n <= 0L) {
return 0L;
}
if (byteBuffer.remaining() >= n) {
// The buffered content is enough to skip
byteBuffer.position(byteBuffer.position() + (int) n);
return n;
}
long skippedFromBuffer = byteBuffer.remaining();
long toSkipFromFileChannel = n - skippedFromBuffer;
// Discard everything we have read in the buffer.
byteBuffer.position(0);
byteBuffer.flip();
return skippedFromBuffer + skipFromFileChannel(toSkipFromFileChannel);
}
private long skipFromFileChannel(long n) throws IOException {
long currentFilePosition = fileChannel.position();
long size = fileChannel.size();
if (n > size - currentFilePosition) {
fileChannel.position(size);
return size - currentFilePosition;
} else {
fileChannel.position(currentFilePosition + n);
return n;
}
}
@Override
public synchronized void close() throws IOException {
fileChannel.close();
StorageUtils.dispose(byteBuffer);
}
@Override
protected void finalize() throws IOException {
close();
}
}
refill()方法
处理方式如下:
如果bytebuffer已满,没有剩余字节数,清空bytebuffer,然后从fileChannel中读取数据填充到bytebuffer中。
如果到达流的末端,返回false,否则返回true。是否像BufferedInputStream填满缓冲区那样填满bytebuffer,未明。
read()方法
执行一次refill()方法,再调用bytebuffer的get方法。返回bytebuffer的当前position的对应字节,并将position前移一位。
处理方式同样可以简称为:没剩先填充,有剩就读取。与BufferedInputStream不同的只是,判断是否有剩这个逻辑放到了refill方法中。
read(byte[] b, int offset, int len)方法
执行一次refill()方法,再调用bytebuffer的get(byte[] b, int offset, int len)方法。不一定能读取目标字节数,当bytebuffer中的剩余字节数不够时,读取剩余字节数。方法返回实际读取字节数。
available()方法
直接调用bytebuffer的remaining()方法,返回bytebuffer的剩余字节数。
skip(int n)方法
如果bytebuffer的剩余字节数大于目标跳过字节数,直接将bytebuffer的position前移n位。
如果bytebuffer的剩余字节数不够大,不足以跳过目标字节数,分两步来完成跳过目标字节数:
1、处理bytebuffer。把bytebuffer的剩余字节数都跳完,重置position,丢弃在bytebuffer中已读的所有内容。此时仍留有一部分未跳目标字节数,假设为m。
2、处理fileChannel。将fileChannel的position前移m位。同样,如果fileChannel的size不够大,直接将position前移到末尾的size位置。
最后,返回实际跳过的字节数。