使用python连接hdfs 文件系统，实现上传文件

一、在用户的根目录下写配置文件

(mypython) [root@hadoop-yarn-117 ~]# vi .hdfscli.cfg 
[global]
default.alias = dev


[dev.alias]
url = http://hadoop-yarn-117:50070
user = root 

二、使用python 连接

from hdfs import Config

client = Config().get_client()
'''
list_file = client.list('/data01')
print(list_file)
with client.read('/data01/README.txt', chunk_size=1280) as reader:
    for chunk in reader:
        print(chunk.decode().replace(' ', '\n '))
'''
with open('/root/docu/nginx-access.log') as opener, client.write('/data01/nginx.log') as writer:
    for line in opener:
        writer.write(bytes(line, encoding='utf-8'))