背景
- 训练集数据集比较大,目前train 集合已经27G,每次占用的磁盘空间较大
- getmerge 时间太长,15分钟用来获取训练集、测试集、评估集
操作
环境tensorflow1.13 ,python2.7
获取hdfs 的文件目录
- 此处主要参考了:https://blog.csdn.net/liukanglucky/article/details/102952686
import commands
import re
import tensorflow as tf
def get_file_list( root_path,path_pattern=[]):
"""
生成hdfs file list
:param path_pattern:
:param root_path
:return:
"""
cmd = """hadoop fs -ls -R {}""".format(root_path.strip())
if len(path_pattern) == 0:
pattern = "|".join(["(" + str(p.replace('/', '\/')) + ")" for p in path_pattern])
else:
pattern = ""
# 筛选文件
def validate_path_pattern(path