最近,我需要将一些文件(每个文件都有JSON格式的对象列表(数组))转换为每个文件都具有相同数据(对象)的分隔行的文件。 这是一次性的任务,很简单。 我使用Java nio的某些功能进行了读写。 我以最简单的方式使用了GSON。 一个线程在文件上运行,转换和写入。 整个操作在几秒钟内完成。 但是,我想并发一点。 因此,我增强了可同时工作的工具。
线程数
可运行以读取文件。
阅读器线程将提交到ExecutorService。 输出是对象列表(示例中为User),将被放入BlockingQueue。
可运行以写入文件。
每个可运行对象将从阻塞队列中轮询。 它将数据行写入文件。 我没有将编写器Runnable添加到ExecutorService,而是仅使用它启动了一个线程。 Runnable具有while(some boolen is true) {...}
模式。 有关以下内容的更多信息...
同步一切
BlockingQueue是这两种线程的接口。 由于writer的runnable在while循环(消费者)中运行,我希望能够使其停止,以便该工具终止。 因此,我为此使用了两个对象:
信号
读取输入文件的循环会增加一个计数器。 完成遍历输入文件并提交编写器后,我在主线程中初始化了一个信号灯: semaphore.acquire(numberOfFiles);
在每个可运行的阅读器中,我释放了信号量: semaphore.release();
原子布尔
作者的while循环使用AtomicBoolean。 只要AtomicBoolean == true,编写器将继续。 在主线程中,在获取信号量之后,我将AtomicBoolean设置为false。 这使编写器线程可以终止。
使用Java NIO
为了扫描,读取和写入文件系统,我使用了Java NIO的某些功能。
扫描: Files.newDirectoryStream(inputFilesDirectory, "*.json");
开始之前删除输出目录: Files.walkFileTree...
BufferedReader和BufferedWriter: Files.newBufferedReader(filePath);
Files.newBufferedWriter(fileOutputPath, Charset.defaultCharset());
一注。 为了生成此示例的随机文件,我使用了apache commons lang: RandomStringUtils.randomAlphabetic
GitHub中的所有代码。
public class JsonArrayToJsonLines {
private final static Path inputFilesDirectory = Paths.get("src\\main\\resources\\files");
private final static Path outputDirectory = Paths
.get("src\\main\\resources\\files\\output");
private final static Gson gson = new Gson();
private final BlockingQueue<EntitiesData> entitiesQueue = new LinkedBlockingQueue<>();
private AtomicBoolean stillWorking = new AtomicBoolean(true);
private Semaphore semaphore = new Semaphore(0);
int numberOfFiles = 0;
private JsonArrayToJsonLines() {
}
public static void main(String[] args) throws IOException, InterruptedException {
new JsonArrayToJsonLines().process();
}
private void process() throws IOException, InterruptedException {
deleteFilesInOutputDir();
final ExecutorService executorService = createExecutorService();
DirectoryStream<Path> directoryStream = Files.newDirectoryStream(inputFilesDirectory, "*.json");
for (int i = 0; i < 2; i++) {
new Thread(new JsonElementsFileWriter(stillWorking, semaphore, entitiesQueue)).start();
}
directoryStream.forEach(new Consumer<Path>() {
@Override
public void accept(Path filePath) {
numberOfFiles++;
executorService.submit(new OriginalFileReader(filePath, entitiesQueue));
}
});
semaphore.acquire(numberOfFiles);
stillWorking.set(false);
shutDownExecutor(executorService);
}
private void deleteFilesInOutputDir() throws IOException {
Files.walkFileTree(outputDirectory, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
Files.delete(file);
return FileVisitResult.CONTINUE;
}
});
}
private ExecutorService createExecutorService() {
int numberOfCpus = Runtime.getRuntime().availableProcessors();
return Executors.newFixedThreadPool(numberOfCpus);
}
private void shutDownExecutor(final ExecutorService executorService) {
executorService.shutdown();
try {
if (!executorService.awaitTermination(120, TimeUnit.SECONDS)) {
executorService.shutdownNow();
}
if (!executorService.awaitTermination(120, TimeUnit.SECONDS)) {
}
} catch (InterruptedException ex) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
private static final class OriginalFileReader implements Runnable {
private final Path filePath;
private final BlockingQueue<EntitiesData> entitiesQueue;
private OriginalFileReader(Path filePath, BlockingQueue<EntitiesData> entitiesQueue) {
this.filePath = filePath;
this.entitiesQueue = entitiesQueue;
}
@Override
public void run() {
Path fileName = filePath.getFileName();
try {
BufferedReader br = Files.newBufferedReader(filePath);
User[] entities = gson.fromJson(br, User[].class);
System.out.println("---> " + fileName);
entitiesQueue.put(new EntitiesData(fileName.toString(), entities));
} catch (IOException | InterruptedException e) {
throw new RuntimeException(filePath.toString(), e);
}
}
}
private static final class JsonElementsFileWriter implements Runnable {
private final BlockingQueue<EntitiesData> entitiesQueue;
private final AtomicBoolean stillWorking;
private final Semaphore semaphore;
private JsonElementsFileWriter(AtomicBoolean stillWorking, Semaphore semaphore,
BlockingQueue<EntitiesData> entitiesQueue) {
this.stillWorking = stillWorking;
this.semaphore = semaphore;
this.entitiesQueue = entitiesQueue;
}
@Override
public void run() {
while (stillWorking.get()) {
try {
EntitiesData data = entitiesQueue.poll(100, TimeUnit.MILLISECONDS);
if (data != null) {
try {
String fileOutput = outputDirectory.toString() + File.separator + data.fileName;
Path fileOutputPath = Paths.get(fileOutput);
BufferedWriter writer = Files.newBufferedWriter(fileOutputPath, Charset.defaultCharset());
for (User user : data.entities) {
writer.append(gson.toJson(user));
writer.newLine();
}
writer.flush();
System.out.println("=======================================>>>>> " + data.fileName);
} catch (IOException e) {
throw new RuntimeException(data.fileName, e);
} finally {
semaphore.release();
}
}
} catch (InterruptedException e1) {
}
}
}
}
private static final class EntitiesData {
private final String fileName;
private final User[] entities;
private EntitiesData(String fileName, User[] entities) {
this.fileName = fileName;
this.entities = entities;
}
}
}
翻译自: https://www.javacodegeeks.com/2014/12/playing-with-java-concurrency.html