本文介绍了如何利用Hive的Streaming API通过Python脚本高效地将大量数据插入到Hive表中，避免了标准INSERT INTO操作的性能问题。通过示例展示了如何编写Python脚本处理输入数据并将其转换为Hive可读格式，以及在Linux和Windows HDInsight上执行此操作的方法。
Hive is a data warehouse designed for querying and aggregating large datasets that reside on HDFS.
The standard INSERT INTO syntax performs poorly because:
Each statement required a Map/Reduce process to be executed.
Each statement will result in a new file being added to HDFS - over time this will lead to very poor performance when reading from the table.
With that said, there is now a Streaming API for Hive / HCatalog, as detailed here.
I am faced with the need to insert data at velocity into Hive, using Python. I am aware of the pyhive and pyhs2 libraries, but neither of them appears to make use of the Streaming API.
Has anyone successfully managed to get Python to insert many rows into Hive using the Streaming API, and how was this done?
I look forward to your insights!
解决方案
Hive user can stream table through script to tra
最低0.47元/天 解锁文章
889
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
