java读写parquet,是否可以在不依赖Hadoop和HDFS的情况下使用Java来读写Parquet？

I've been hunting around for a solution to this question.

It appears to me that there is no way to embed reading and writing Parquet format in a Java program without pulling in dependencies on HDFS and Hadoop. Is this correct?

I want to read and write on a client machine, outside of a Hadoop cluster.

I started to get excited about Apache Drill, but it appears that it must run as a separate process. What I need is an in-process ability to read and write a file using the Parquet format.

解决方案

You can write parquet format out side hadoop cluster using java Parquet Client API.

Here is a sample code in java which writes parquet format to local disk.

{

final String schemaLocation = "/tmp/avro_format.json";

final Schema avroSchema = new Schema.Parser().parse(new File(schemaLocation));

final MessageType parquetSchema = new AvroSchemaConverter().convert(avroSchema);

final WriteSupport writeSupport = new AvroWriteSupport(parquetSchema, avroSchema);

final String parquetFile = "/tmp/parquet/data.parquet";

final Path path = new Path(parquetFile);

ParquetWriter parquetWriter = new ParquetWriter(path, writeSupport, CompressionCodecName.SNAPPY, BLOCK_SIZE, PAGE_SIZE);

final GenericRecord record = new GenericData.Record(avroSchema);

record.put("id", 1);

record.put("age", 10);

record.put("name", "ABC");

record.put("place", "BCD");

parquetWriter.write(record);

parquetWriter.close();

}

avro_format.json,

{

"type":"record",

"name":"Pojo",

"namespace":"com.xx.test",

"fields":[

{

"name":"id",

"type":[

"int",

"null"

]

},

{

"name":"age",

"type":[

"int",

"null"

]

},

{

"name":"name",

"type":[

"string",

"null"

]

},

{

"name":"place",

"type":[

"string",

"null"

]

}

]

}

Hope this helps.