Flink读取HDFS上的Parquet文件生成DataSet

打酱油的葫芦娃

已于 2023-11-01 14:54:12 修改

阅读量7.3k

点赞数 4

分类专栏： Flink 文章标签： flink

于 2020-03-27 18:17:12 首次发布

本文链接：https://blog.csdn.net/WenWu_Both/article/details/105147281

版权

Flink 专栏收录该内容

19 篇文章 2 订阅 ¥49.90 ¥99.00

订阅专栏

超级会员免费看

本文介绍了如何在Flink中通过泛型模式读取HDFS上的Parquet文件，重点讨论了构建ParquetInputFormat的参数，包括从Parquet文件Footer获取Schema和使用AvroSchemaConverter工具类进行转换的方法。

摘要由CSDN通过智能技术生成

首先打开Flink的官方网站，查看一下DataSet已支持的数据源。

File-based

readTextFile(path) / TextInputFormat - Reads files line wise and returns them as Strings.

readTextFileWithValue(path) / TextValueInputFormat - Reads files line wise and returns them as StringValues. StringValues are mutable strings.

readCsvFile(path) / CsvInputFormat - Parses files of comma (or another char) delimited fields. Returns a DataSet of tuples or POJOs. Supports the basic java types and their Value counterparts as field types.

readFileOfPrimitives(path, Class) / PrimitiveInputFormat - Parses files of new-line (or another char sequence) delimited primitive data types such as String or Integer.

readFileOfPrimitives(path, delimiter, Class) / PrimitiveInputFormat - Parses files of new-line

了解本专栏