I am reading a very large (500mb) file with Files.lines(...).
It reads a part of the file but at some point it breaks with java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1
I think the file has lines with different charsets. Is there a way to skip these broken lines? I know that the stream returned is backed by a Reader and with the reader I know how to skip, but don't know how to get the Reader from the stream to set it up as I like.
List lines = new ArrayList<>();
try (Stream stream = Files.lines(Paths.get(getClass().getClassLoader().getResource("bigtest.txt").toURI()), Charset.forName("UTF-8"))) {
stream
.filter(s -> s.substring(0, 2).equalsIgnoreCase("aa"))
.forEach(lines::add);
} catch (final IOException e) {
// catch
}
解决方案
You can’t filter lines with invalid characters after the decoding when the preconfigured decoder already stops the decoding with an exception. You have to configure a CharsetDecoder manually to tell it to ignore invalid input or replace that input with a special character.
CharsetDecoder dec=StandardCharsets.UTF_8.newDecoder()
.onMalformedInput(CodingErrorAction.IGNORE);
Path path=Paths.get(getClass().getClassLoader().getResource("bigtest.txt").toURI());
List lines;
try(Reader r=Channels.newReader(FileChannel.open(path), dec, -1);
BufferedReader br=new BufferedReader(r)) {
lines=br.lines()
.filter(s -> s.regionMatches(true, 0, "aa", 0, 2))
.collect(Collectors.toList());
}
This simply ignores charset decoding errors, skipping the characters. To skip entire lines containing errors, you can let the decoder insert a replacement character (defaults to '\ufffd') for errors and filter out lines containing that character:
CharsetDecoder dec=StandardCharsets.UTF_8.newDecoder()
.onMalformedInput(CodingErrorAction.REPLACE);
Path path=Paths.get(getClass().getClassLoader().getResource("bigtest.txt").toURI());
List lines;
try(Reader r=Channels.newReader(FileChannel.open(path), dec, -1);
BufferedReader br=new BufferedReader(r)) {
lines=br.lines()
.filter(s->!s.contains(dec.replacement()))
.filter(s -> s.regionMatches(true, 0, "aa", 0, 2))
.collect(Collectors.toList());
}