java读写parquet_是否可以在不依赖Hadoop和HDFS的情况下使用Java读写Parquet？

我一直在寻找这个问题的解决方案.

在我看来,如果不依赖HDFS和Hadoop,就无法在Java程序中嵌入读写Parquet格式.这个对吗？

我想在Hadoop集群之外的客户端计算机上进行读写.

我开始对Apache Drill感到兴奋,但是看来它必须作为单独的进程运行.我需要的是一种使用Parquet格式读写文件的过程中功能.

解决方法:

您可以使用java Parquet Client API在hadoop集群外部编写Parquet格式.

这是Java中的示例代码,该示例代码将镶木地板格式写入本地磁盘.

{

final String schemaLocation = "/tmp/avro_format.json";

final Schema avroSchema = new Schema.Parser().parse(new File(schemaLocation));

final MessageType parquetSchema = new AvroSchemaConverter().convert(avroSchema);

final WriteSupport writeSupport = new AvroWriteSupport(parquetSchema, avroSchema);

final String parquetFile = "/tmp/parquet/data.parquet";

final Path path = new Path(parquetFile);

ParquetWriter parquetWriter = new ParquetWriter(path, writeSupport, CompressionCodecName.SNAPPY, BLOCK_SIZE, PAGE_SIZE);

final GenericRecord record = new GenericData.Record(avroSchema);

record.put("id", 1);

record.put("age", 10);

record.put("name", "ABC");

record.put("place", "BCD");

parquetWriter.write(record);

parquetWriter.close();

}

avro_format.json,

{

"type":"record",

"name":"Pojo",

"namespace":"com.xx.test",

"fields":[

{

"name":"id",

"type":[

"int",

"null"

]

},

{

"name":"age",

"type":[

"int",

"null"

]

},

{

"name":"name",

"type":[

"string",

"null"

]

},

{

"name":"place",

"type":[

"string",

"null"

]

}

]

}

希望这可以帮助.

标签：hadoop,apache-drill,java,parquet,data-formats

来源： https://codeday.me/bug/20191111/2022069.html