tensorflow读取hdfs上的文件

1. 确保服务器能访问hdfs

2. 在~/.bashrc中加入以下脚本：

export JAVA_HOME=/usr/local/jdk1.8.0_131
export HADOOP_HDFS_HOME=/usr/local/hadoop-2.7.3
export PATH=$PATH:$HADOOP_HDFS_HOME/libexec/hadoop-config.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server
export PATH=$PATH:$HADOOP_HDFS_HOME/bin:$HADOOP_HDFS_HOME/sbin
export CLASSPATH="$(hadoop classpath --glob)"

3. source ~/.bashrc即可

4. 测试代码：

import tensorflow as tf

file_path = "hdfs://ns3-backup/user/dw/*.tfrecord"
files = tf.io.gfile.glob(file_path)
print(files)