AWS-EMR & Airflow 调度hivesql 异常thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

最新推荐文章于 2024-09-15 08:19:50 发布

Norris Huang

最新推荐文章于 2024-09-15 08:19:50 发布

阅读量1.3k

点赞数

分类专栏： Hadoop AWS EMR 文章标签： aws python 云计算

本文链接：https://blog.csdn.net/turk/article/details/127310634

版权

Hadoop 同时被 3 个专栏收录

14 篇文章 1 订阅

订阅专栏

AWS

2 篇文章 0 订阅

订阅专栏

EMR

1 篇文章 0 订阅

订阅专栏

通过Airflow 调用 EMR 中的hive 执行sql，遇到错误异常

 hh = HiveServer2Hook(
        hiveserver2_conn_id="hive_connect_emr"
    )
    res = hh.get_pandas_df("show tables")

[2022-10-13, 21:46:21 ] {{taskinstance.py:1703}} ERROR - Task failed with exception
Traceback (most recent call last):
File “/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py”, line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File “/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py”, line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File “/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py”, line 1514, in _execute_task
result = execute_callable(context=context)
File “/usr/local/lib/python3.7/site-packages/airflow/operators/python.py”, line 151, in execute
return_value = self.execute_callable()
File “/usr/local/lib/python3.7/site-packages/airflow/operators/python.py”, line 162, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File “/usr/local/airflow/dags/netease_hiev_hook.py”, line 29, in execSQLDWD
res = hh.get_pandas_df(“show tables”)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 1062, in get_pandas_df
res = self.get_results(hql, schema=schema, hive_conf=hive_conf)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 949, in get_results
header = next(results_iter)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 885, in _get_results
with contextlib.closing(self.get_conn(schema)) as conn, contextlib.closing(conn.cursor()) as cur:
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 870, in get_conn
database=schema or db.schema or ‘default’,
File “/usr/local/airflow/.local/lib/python3.7/site-packages/pyhive/hive.py”, line 104, in connect
return Connection(*args, **kwargs)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/pyhive/hive.py”, line 249, in init
response = self._client.OpenSession(open_session_req)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/TCLIService/TCLIService.py”, line 187, in OpenSession
return self.recv_OpenSession()
File “/usr/local/airflow/.local/lib/python3.7/site-packages/TCLIService/TCLIService.py”, line 199, in recv_OpenSession
(fname, mtype, rseqid) = iprot.readMessageBegin()
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/protocol/TBinaryProtocol.py”, line 148, in readMessageBegin
name = self.trans.readAll(sz)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/transport/TTransport.py”, line 62, in readAll
chunk = self.read(sz - have)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/transport/TTransport.py”, line 164, in read
self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/transport/TSocket.py”, line 167, in read
message=‘TSocket read 0 bytes’)
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes