python调用hive与java调用区别_[转载]python客户端通过thrift调用hive

Thrift是一个跨语言服务部署框架，最初由Facebook于2007年开发，后于2008年进入Apache孵化器(Apache

Incubator)。

类似于SOAP，COM

和CORBA，Thrift通过定义一个中间定义语言和Thrift代码生成工具，生成指定语言的代码。

目前，Thrift支持C++,Java, Python, PHP, Ruby, Erlang, Perl, Haskell,

C#, Cocoa, Smalltalk和OCaml的代码生成。

1安装thrift

安裝需要使用到的套件：

sudo apt-get install libboost-dev libevent-dev python-dev

automake pkg-config libtool flex bison g++

編譯與安裝thrift：

wget

http://ftp.tcc.edu.tw/pub/Apache//incubator/thrift/0.5.0-incubating/thrift-0.5.0.tar.gz

tar -xvf thrift-0.5.0.tar.gz

mv thrift-0.5.0 thrift

cd thrift

./configure

make

sudo make install

2安装好python thrift模块，但是无法加载

>>> from

thrift.transport import TTransport

Traceback (most recent call last):

from thrift.transport

import TTransport

ImportError: No module named thrift.transport

然而模块文件存在，/usr/lib/python2.6/site-packages/thrift/transport/TTransport.py

$ python

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)

[GCC 4.3.3] on linux2

Type "help", "copyright", "credits" or "license" for more

information.

>>> import

sys

>>>

sys.path

['',

'/usr/lib/python2.6',

'/usr/lib/python2.6/plat-linux2',

'/usr/lib/python2.6/lib-tk',

'/usr/lib/python2.6/lib-old',

'/usr/lib/python2.6/lib-dynload',

'/usr/lib/python2.6/dist-packages',

'/usr/lib/python2.6/dist-packages/Numeric',

'/usr/lib/python2.6/dist-packages/PIL',

'/var/lib/python-support/python2.6',

'/var/lib/python-support/python2.6/gtk-2.0',

'/usr/local/lib/python2.6/dist-packages']

(note that "site-packages" is not in the list). I assume that

I configured my thrift code when the paths were different in my

system, and some upgrade changed python directories. So when I

"make install" thrift, they are still copied into site-packages.

Now, having thrift already installed there, it is a matter of

adding this dir to the search path. This is done with

sys.path.append, according to python's documentation:

>>>

sys.path.append('/usr/lib/python2.6/site-packages')

>>>

sys.path

['',

'/usr/lib/python2.6',

(...)

'/usr/local/lib/python2.6/dist-packages',

'/usr/lib/python2.6/site-packages']

>>> from

thrift.transport import TTransport

(No error message now)

在.bashrc文件中加入:

export PYTHONPATH=/usr/lib/python2.6/site-packages

3启动Hive Thrift Server

(hive --service hiveserver)

4代码

#!/usr/bin/env python

import sys

sys.path.append('/usr/local/hadoop/hive/lib/py')

from hive_service import ThriftHive

from hive_service.ttypes import HiveServerException

from thrift import Thrift

from thrift.transport import TSocket

from thrift.transport import TTransport

from thrift.protocol import TBinaryProtocol

def hiveExe(sql):

try:

transport = TSocket.TSocket('127.0.0.1',

10000)

transport =

TTransport.TBufferedTransport(transport)

protocol =

TBinaryProtocol.TBinaryProtocol(transport)

client = ThriftHive.Client(protocol)

transport.open()

client.execute(sql)

print "The return value is :

"

print client.fetchAll()

print "............"

transport.close()

except

Thrift.TException, tx:

print '%s' % (tx.message)

if __name__ == '__main__':

hiveExe("select * from

t_afan_test")

5执行