最近在烦恼用java操作大文件的问题——如果我有一大堆数据在内存中,想要写入文件,如何提速?
我们都知道 Java 中的 RandomAccessFile 可以访问文件的任意位置 , 另外 , java 中想要提速的方法有缓存,多线程操作等。而在这一环节中我首先想到的就是多线程写入文件,这样可以提升速度的吧?
但是我们知道,无论何时文件的写操作,底层都会加锁(这会影响性能就不解释了),为了验证这个锁对多线程的影响有多大,下面做了一个实验,首先准备要写入文件的字符内容(500个字符):
int len = 500;
int dimen1 = 5;
int dimen2 = len / dimen1;
char data[] = new char[len];
char dd[][] = new char[dimen1][dimen2];
for (int i = 0; i < dimen1; i++) {
for (int j = 0; j < dimen2; j++) {
dd[i][j] = (char) ('a' + i);
data[i * dimen2 + j] = (char) ('a' + i);
}
}
为了能够得到一个比较好的结果,统计十次:
for (int k = 0; k < 10; k++) {
CountDownLatch cl = new CountDownLatch(dd.length);
long start = System.currentTimeMillis();
for (int i = 0; i < dd.length; i++) {
new Thread(new RwThread(new RandomAccessFile(f, "rws"),
i * dimen2*2, i * dimen2*2 + dimen2*2, dd[i], cl)).start();
}
cl.await();
System.out.println("duration: "
+ (System.currentTimeMillis() - start));
}
多线程谢文件的代码如下:
import java.io.RandomAccessFile;
import java.util.concurrent.CountDownLatch;
public static class RwThread implements Runnable {
private RandomAccessFile raf;
private int start;
private int end;
private char[] data;
private CountDownLatch latch;
public RwThread(RandomAccessFile raf, int start, int end, char[] data,
CountDownLatch cl) {
this.raf = raf;
this.start = start;
this.end = end;
this.data = data;
latch = cl;
}
@Override
public void run() {
try {
raf.seek(start);
for (int i = 0; i < data.length; i++) {
raf.writeChar(data[i]);
}
} catch (Exception e) {
} finally {
try {
if (raf != null) {
raf.close();
raf = null;
}
} catch (Exception e) {
}
}
latch.countDown();
}
}
得到耗时输出结果如下:
duration: 130
duration: 114
duration: 115
duration: 127
duration: 124
duration: 117
duration: 114
duration: 119
duration: 118
duration: 119
首先,看到这个结果我就惊呆了 , 写 500 个字符 尽然花费了 100多 ms ........
那他和单线程操作的差距有多大呢? 请继续往下看:
同样统计十次,代码如下:
for (int k = 0; k < 10; k++) {
CountDownLatch cl = new CountDownLatch(1);
long start = System.currentTimeMillis();
new Thread(new RwThread(new RandomAccessFile(f, "rw"), 0,
len * 2, data, cl)).start();
cl.await();
System.out.println("duration: "
+ (System.currentTimeMillis() - start));
}
得到耗时输出结果:
duration: 5
duration: 2
duration: 3
duration: 2
duration: 3
duration: 2
duration: 2
duration: 2
duration: 2
duration: 2
天啦,结果居然相差了几十倍呢
今天的内容就到这里了,下面是完整的代码 ......
import java.io.File;
import java.io.RandomAccessFile;
import java.util.concurrent.CountDownLatch;
/**
* @author edwin
*
*/
public class CreateTmpFile {
public static void main(String[] args) throws Exception {
String root = "F:" + File.separator + "tmpLog";
File f = new File(root, "ttt.txt");
File ddd = new File(root, "ttt1.txt");
if (!f.exists()) {
f.createNewFile();
}
if (!ddd.exists()) {
ddd.createNewFile();
}
System.out.println(f.getAbsolutePath());
int len = 500;
int dimen1 = 5;
int dimen2 = len / dimen1;
char data[] = new char[len];
char dd[][] = new char[dimen1][dimen2];
for (int i = 0; i < dimen1; i++) {
for (int j = 0; j < dimen2; j++) {
dd[i][j] = (char) ('a' + i);
data[i * dimen2 + j] = (char) ('a' + i);
}
}
for (int k = 0; k < 10; k++) {
CountDownLatch cl = new CountDownLatch(1);
long start = System.currentTimeMillis();
new Thread(new RwThread(new RandomAccessFile(f, "rw"), 0, len * 2,
data, cl)).start();
cl.await();
System.out.println("duration: "
+ (System.currentTimeMillis() - start));
}
// for (int k = 0; k < 10; k++) {
// CountDownLatch cl = new CountDownLatch(dd.length);
// long start = System.currentTimeMillis();
// for (int i = 0; i < dd.length; i++) {
// new Thread(new RwThread(new RandomAccessFile(f, "rws"),
// i * dimen2*2, i * dimen2*2 + dimen2*2, dd[i], cl)).start();
// }
// cl.await();
// System.out.println("duration: "
// + (System.currentTimeMillis() - start));
// }
}
public static class RwThread implements Runnable {
private RandomAccessFile raf;
private int start;
private int end;
private char[] data;
private CountDownLatch latch;
public RwThread(RandomAccessFile raf, int start, int end, char[] data,
CountDownLatch cl) {
this.raf = raf;
this.start = start;
this.end = end;
this.data = data;
latch = cl;
}
@Override
public void run() {
try {
raf.seek(start);
for (int i = 0; i < data.length; i++) {
raf.writeChar(data[i]);
}
} catch (Exception e) {
} finally {
try {
if (raf != null) {
raf.close();
raf = null;
}
} catch (Exception e) {
}
}
latch.countDown();
}
}
}