1:kafka
1,开启虚拟机zookeeper,kafka进程
2,在pycharm导包
from pykafka import KafkaClient
3,py代码
client = KafkaClient(hosts="datanode1:9092") # print(client.topics) topic = client.topics['car'] #topic名称 consumer = topic.get_simple_consumer() li = list() i = 0 for record in consumer: a=record.value.decode('utf8') if(len(a)>20): print(a) li.append(a.split(',')) i+=1 if(i>12)://循环一直不停,记录到12条之后停止循环 break # print(1) # print(len(consumer)) df = pd.DataFrame(li[1:],columns=li[0])//生成dataframe,方便hive读取 print(df[''])
4,向hive写文件
原理:hive存储基于hdfs,所以操作hdfs上的文件,达到向里面存数据
- 创建hive表:
CREATE TABLE `employee`( `id` string, `name` string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' //分割符
- py 导包&代码:
from pyhdfs import HdfsClient
data=[["qw","qwe"],["asd","asd"]] df=pd.DataFrame(data,columns=["id","name"]) print(df) //测试数据 df.to_csv("D:\\car.csv",sep=',', header=True, index=False) client=HdfsClient(hosts='192.168.153.132:50070',user_name='mmn')#只有hadoop用户拥有写权限 # str='hello world' # client.create('/user/hive/warehouse/employee/py.txt',str)#创建新文件并写入字符串 # client.append('/user/hive/warehouse/employee/py.txt',str) client.copy_from_local("D:\\car.csv",'/user/hive/warehouse/employee/car.csv') //先导入本地再下载到hdfs上
- 遇到问题:向hive写文件 csv文件以,分割 所以hive表要设置程,分割 其次表类型要与数据类型对应。否则会出现null的情况 修改分隔符命令
在hive命令行下 ALTER TABLE table_name SET SERDEPROPERTIES (
'field.delim'
=
',' , 'serialization.format'=','
);
- 尝试用pyhive 连接hive
导包
pip install sasl
pip install thrift
pip install thrift-sasl
pip install PyHive
- 首先开启Hadoop集群
- nohup hive --service metastore > metastore.log 2>&1 &
nohup hive --service hiveserver2 > metastore.log 2>&1 & 挂起hive在后台 - jps 中会有两个runjar进程
- 更改hive-site.xml
python代码
conn = hive.Connection(host='192.168.153.132', port=10000, username='mmn', auth='NOSASL', database='default') curs = conn.cursor() # 获取一个游标 sql = 'select * from employee' # 操作语句 curs.execute(sql) # 执行sql语句 print(curs.fetchall()) # 输出获取结果的所有行
结束!过后补充