hdfs如何查找指定目录是否文件,如何列出Hdfs目录和子目录中文件的路径？

Could not figure out a way to list all files in directory and subdirectories.

Here is the code that I'm using which lists files in a specific directory but files if there is a subdorectory inside:

val conf = new Configuration()

val fs = FileSystem.get(new java.net.URI("hdfs://servername/"), conf)

val status = fs.listStatus(new Path("path/to/folder/"))

status.foreach { x => println(x.getPath.toString()) }

the above code lists all the files inside the directory but I need it to be recursive.

解决方案

You could go for a recursion whenever you discover a new folder:

val hdfs = FileSystem.get(new Configuration())

def listFileNames(hdfsPath: String): List[String] = {

hdfs

.listStatus(new Path(hdfsPath))

.flatMap { status =>

// If it's a file:

if (status.isFile)

List(hdfsPath + "/" + status.getPath.getName)

// If it's a dir and we're in a recursive option:

else

listFileNames(hdfsPath + "/" + status.getPath.getName)

}

.toList

.sorted

}