Hive is a data warehouse designed for querying and aggregating large datasets that reside on HDFS.
The standard INSERT INTO syntax performs poorly because:
Each statement required a Map/Reduce process to be executed.
Each statement will result in a new file being added to HDFS - over time this will lead to very poor performance when reading from the table.
With that said, there is now a Streaming API for Hive / HCatalog, as detailed here.
I am faced with the need to insert data at velocity into Hive, using Python. I am aware of the pyhive and pyhs2 libraries, but neither of them appears to make use of the Streaming API.
Has anyone successfully managed to get Python to insert many rows into Hive using the Streaming API, and how was this done?
I look forward to your insights!
解决方案
Hive user can stream table through script to tra