我仍然是一个相对较新的程序员,我在Java中遇到的一个问题是Out of Memory Errors.我不想使用-Xmx来增加内存,因为我觉得错误是由于编程不好造成的,我想改进编码而不是依靠更多的内存.
我所做的工作涉及处理大量文本文件,每个文本文件在压缩时大约为1GB.我在这里的代码是为了遍历一个新的压缩文本文件被删除的目录.它打开了第二个最新的文本文件(不是最新的,因为它仍然被写入),并使用Jsoup库来解析文本文件中的某些字段(字段用自定义分隔符分隔:“| nTa |”表示一个新列,“| nLa |”表示一个新行.
我觉得应该没有理由使用大量内存.我打开一个文件,扫描它,解析相关位,将解析后的版本写入另一个文件,关闭文件,然后移动到下一个文件.我不需要将整个文件存储在内存中,我当然不需要存储已经在内存中处理过的文件.
当我开始解析第二个文件时,我遇到错误,这表明我没有处理垃圾收集.请查看代码,看看你是否能发现我正在做的事情,这意味着我使用的内存比我应该的多.我想学习如何做到这一点,所以我不会出现内存错误!
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Scanner;
import java.util.TreeMap;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
import org.jsoup.Jsoup;
public class ParseHTML {
public static int commentExtractField = 3;
public static int contentExtractField = 4;
public static int descriptionField = 5;
public static void main(String[] args) throws Exception {
File directoryCompleted = null;
File filesCompleted[] = null;
while(true) {
// find second most recent file in completed directory
directoryCompleted = new File(args[0]);
filesCompleted = directoryCompleted.listFiles();
if (filesCompleted.length > 1) {
TreeMap timeStamps = new TreeMap(Collections.reverseOrder());
for (File f : filesCompleted) {
timeStamps.put(getTimestamp(f), f);
}
File fileToProcess = null;
int counter = 0;
for (Long l : timeStamps.keySet()) {
fileToProcess = timeStamps.get(l);
if (counter == 1) {
break;
}
counter++;
}
// start processing file
GZIPInputStream gzipInputStream = null;
if (fileToProcess != null) {
gzipInputStream = new GZIPInputStream(new FileInputStream(fileToProcess));
}
else {
System.err.println("No file to process!");
System.exit(1);
}
Scanner scanner = new Scanner(gzipInputStream);
scanner.useDelimiter("\|nLa\|");
GZIPOutputStream output = new GZIPOutputStream(new FileOutputStream("parsed/" + fileToProcess.getName()));
while (scanner.hasNext()) {
Scanner scanner2 = new Scanner(scanner.next());
scanner2.useDelimiter("\|nTa\|");
ArrayList row = new ArrayList();
while(scanner2.hasNext()) {
row.add(scanner2.next());
}
for (int index = 0; index < row.size(); index++) {
if (index == commentExtractField ||
index == contentExtractField ||
index == descriptionField) {
output.write(jsoupParse(row.get(index)).getBytes("UTF-8"));
}
else {
output.write(row.get(index).getBytes("UTF-8"));
}
String delimiter = "";
if (index == row.size() - 1) {
delimiter = "|nLa|";
}
else {
delimiter = "|nTa|";
}
output.write(delimiter.getBytes("UTF-8"));
}
}
output.finish();
output.close();
scanner.close();
gzipInputStream.close();
}
}
}
public static Long getTimestamp(File f) {
String name = f.getName();
String removeExt = name.substring(0, name.length() - 3);
String timestamp = removeExt.substring(7, removeExt.length());
return Long.parseLong(timestamp);
}
public static String jsoupParse(String s) {
if (s.length() == 4) {
return s;
}
else {
return Jsoup.parse(s).text();
}
}
}
我怎样才能确保当我完成对象时,它们被销毁并且不使用任何资源?例如,每次我关闭GZIPInputStream,GZIPOutputStream和Scanner时,我怎样才能确保它们被彻底销毁?
为了记录,我得到的错误是:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
at java.lang.StringBuilder.append(StringBuilder.java:203)
at org.jsoup.parser.TokeniserState$47.read(TokeniserState.java:1171)
at org.jsoup.parser.Tokeniser.read(Tokeniser.java:42)
at org.jsoup.parser.TreeBuilder.runParser(TreeBuilder.java:101)
at org.jsoup.parser.TreeBuilder.parse(TreeBuilder.java:53)
at org.jsoup.parser.Parser.parse(Parser.java:24)
at org.jsoup.Jsoup.parse(Jsoup.java:44)
at ParseHTML.jsoupParse(ParseHTML.java:125)
at ParseHTML.main(ParseHTML.java:81)