springbatch读取文件_使用Spring Batch从HDFS读取文件

I've to write a Spring batch which will read a file from HDFS and will update the data in MySQL DB.

The source file in HDFS contains some report data, in CSV format.

Can someone point me to an example of reading a file from HDFS?

Thanks.

解决方案

The FlatFileItemReader in Spring Batch works with any Spring Framework Resource implementation:

@Bean

public FlatFileItemReader itemReader() {

Resource resource; // get (or autowire) resource

return new FlatFileItemReaderBuilder()

.resource(resource)

// set other reader properties

.build();

}

So if you manage to have a Resource handle pointing to a HDFS file, your are done.

Now in order to have a HDFS resource, you can:

Use Spring for Hadoop. Once the HDFS file system is configured, you would be able to get the resource from the application context with applicationContext.getResource("hdfs:data.csv");

Implement your own Resource using Hadoop APIs (like shown in the answer by Michael Simons). I see that some folks already did this here

Hope this helps.