python操作hive数据库代码_Python链接Hive读取数据的几种方式

R和Python是数据分析人员常用的两个工具, 笔主在R通过RJDBC链接Hive 介绍了R如何链接Hive, 今天换Python.

以下有几种链接方式, 标题均包含了官方链接, 详情大家可点击查看

注意:

Python链接Hive和Python版本和包版本有很大关系, 建议大家通过Anaconda进行管理, 欢迎阅读Mac下Anaconda的安装和使用

thrift, sasl, thrift_sasl, thriftpy这几个包均会用到, 请提前安装好, 建议用安装对应版本

thrift - 0.11.0

sasl - 0.2.1

thrift_sasl - 0.3.0

thriftpy - 0.3.9

笔主环境

版本

# 系统

macOS Mojave Version 10.14.2

# Python

Python 3.7.1

依赖包准备

包安装

pip install pyhive

链接方式

from pyhive import hive

conn = hive.Connection(host = '', port = 10000, username = '****', database = 'default')

#host主机ip,port：端口号，username:用户名，database:使用的数据库名称

cursor = conn.cursor()

cursor.execute('SHOW DATABASES')

# 打印结果

for result in cursor.fetchall():

print(result)

包安装

pip install impyla

错误处理

impyla pip成功安装后, 但实际执行导入时候报错

In [4]: from impala.dbapi import connect

Traceback (most recent call last):

File "/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code

exec(code_obj, self.user_global_ns, self.user_ns)

File "", line 1, in

from impala.dbapi import connect

File "/anaconda3/lib/python3.7/site-packages/impala/dbapi.py", line 28, in

import impala.hiveserver2 as hs2

File "/anaconda3/lib/python3.7/site-packages/impala/hiveserver2.py", line 340

async=True)

^

SyntaxError: invalid syntax

处理方式

conda install -c anaconda impyla

如果以上处理之后还有问题

/anaconda3/lib/python3.7/site-packages/thriftpy/protocol/binary.py in read_message_begin(inbuf, strict)

176 if strict:

177 raise TProtocolException(type=TProtocolException.BAD_VERSION,

--> 178 message='No protocol version header')

179

180 name = inbuf.read(sz).decode('utf-8')

TProtocolException: TProtocolException(type=4)

建议重新安装相关依赖包, 版本一定要正确.

链接方式

conn = connect(host = '172.18.33.23', port = 10000, auth_mechanism = 'PLAIN')

cursor = conn.cursor()

cursor.execute('SHOW DATABASES')

# 打印结果

for result in cursor.fetchall():

print(result)