python2/python3 连接 hive/impala 的问题汇总

最新推荐文章于 2023-09-18 10:14:50 发布

爱知菜

最新推荐文章于 2023-09-18 10:14:50 发布

阅读量6k

点赞数 1

分类专栏： Hadoop/Spark/Hive

本文链接：https://blog.csdn.net/rav009/article/details/87865093

版权

Hadoop/Spark/Hive 专栏收录该内容

72 篇文章 0 订阅

订阅专栏

本文详述了使用Python库impyla和PyHive连接Cloudera Impala和Hive的方法，包括解决常见兼容性和配置问题，如库版本冲突、认证机制选择及错误修复。同时提供了Kerberos和LDAP权限体系下的连接示例。

摘要由CSDN通过智能技术生成

目前Pyhive和impyla不兼容，同一个python不能同时用这两个library

建议连impyla

连Impala

connect函数的源代码: https://github.com/cloudera/impyla/blob/master/impala/dbapi.py

示例:

from impala.dbapi import connect
mcon=connect(host='bd-slave01-pe2.f-pro.cn',port=21050,user='username',password='password',auth_mechanism='PLAIN')

mcon=connect(host='bd-slave01-pe2.f-pro.cn',port=21050,user='username',password='password',auth_mechanism='GSSAPI')

如果VM上有kerberos权限那么可以用 auth_mechanism='GSSAPI' 或 auth_mechanism='PLAIN'

如果美誉kerberos权限, 请用auth_mechanism='PLAIN'.

另附一句命令行的连接impalad的方法:

#kinit first
impala-shell  -u username  -k

https://github.com/cloudera/thrift_sasl/releases

Python3 连接impala正解

# installation for python 3.5.1 , 3.7 
# python 3.9 not support, error happened.
sudo pip3 install impyla
sudo pip3 install thrift_sasl

pip3 install pure-sasl==0.5.1
pip3 install thrift-sasl==0.2.1 --no-deps
pip3 install thrift==0.9.3
pip3 install impyla==0.14.1
pip3 install bitarray==0.8.3
pip3 install thriftpy==0.3.9

~~# TypeError: can't concat str to bytes~~

~~vi /opt/python3.5/lib/python3.5/site-packages/thrift_sasl/__init__.py~~

# 定位到错误的最后一条，在init.py第94行 (注意代码的缩进)
header = struct.pack(">BI", status, len(body))
self._trans.write(header + body)

更改为：
header = struct.pack(">BI", status, len(body))
if(type(body) is str):
body = body.encode()
self._trans.write(header + body)
impyla               0.16.2
thrift                   0.13.0
thrift-sasl           0.4.2
thriftpy               0.3.9
thriftpy2             0.4.11
pure-sasl            0.6.2

问题1.

from thriftpy.transport import TTransportException, TTransportBase, readall
ImportError: cannot import name 'TTransportException'
问题2.

'TSocket' object has no attribute 'isOpen bug: https://github.com/cloudera/impyla/issues/268

'TSaslClientTransport' object has no attribute 'readAll': https://github.com/dropbox/PyHive/issues/151

解决方案:

https://github.com/dropbox/PyHive/commit/5322d8f1420b033ba7446449b5cca2cbf9f6fbc4

pip3 install git+https://github.com/cloudera/thrift_sasl

同时使用impala和pyHive请注意import顺序

连Hive

Python library 版本:

thrift 0.11.0
thrift-sasl 0.3.0 (使用非release版本, 而是用上面的URL来安装)
thriftpy 0.3.9
PyHive 0.6.1

kerberos + LDAP 的权限体系

from pyhive import hive
mcon=hive.connect(host='bd-master01-pe2.f.cn',port=10000,username='someone',password='password',auth='LDAP')
cs = mcon.cursor()
cs.execute('show database')
print(cs.fetchall())
cs.close()
mcon.close()

Kerberos权限体系

from pyhive import hive
import pandas as pd
hcon=hive.connect(host='bd-master01-pe2.f.cn',port=10000,auth ='KERBEROS',kerberos_service_name='hive')
hdata = pd.read_sql('show databases',hcon)
print(hdata)

python2 装 impyla，准备工作:

sudo pip install --upgrade setuptools
sudo yum install -y gcc libffi-devel python-devel openssl-devel gcc-c++
sudo yum install python-devel openldap-devel