【Python】使用pyhive提取数据时同时显示日志

To simultaneously display logs and extract data using PyHive, you need to establish a connection with your Hive server and then fetch the data and logs concurrently. Here is a detailed step-by-step guide for achieving this task using Python and the PyHive library:

1. First, install the required Python libraries. You can do this using pip:

pip install pyhive
pip install thrift
pip install sasl
pip install thrift-sasl

2. Import the required libraries in your Python script:

from pyhive import hive
from TCLIService.ttypes import TOperationState

3. Define the connection parameters and establish the connection to your Hive server using hive.Connection class:

conn = hive.Connection(
    host='your_host',
    port=your_port,
    username='your_username',
    password='your_password',
    auth='CUSTOM',  # Use 'LDAP' or 'KERBEROS' if applicable
    database='your_database'
)

4. Now, create a function to simultaneously fetch logs and data:

def execute_hive_query_and_fetch_logs(conn, query):
    cursor = conn.cursor()
    cursor.execute(query, async_=True)

    status = cursor.poll().operationState

    while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNING_STATE):
        logs = cursor.fetch_logs()

        for message in logs:
            print(message)  # Print log messages

        status = cursor.poll().operationState

    result = cursor.fetchall()
    cursor.close()
    return result

5. Use the above function to execute your Hive query and fetch logs and data concurrently:

query = "SELECT * FROM your_table;"
result = execute_hive_query_and_fetch_logs(conn, query)

print("Result:")
for row in result:
    print(row)


Replace 'your_host', 'your_port', 'your_username', 'your_database', and 'your_table' with appropriate values as per your setup.

This code will fetch logs and data simultaneously while executing the Hive query.