FileSystem.getFileBlockLocations

 

Details

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: None
• Fix Version/s: 0.22.0
• Component/s: hdfs client,name-node
• Labels:

  None Labels

• Hadoop Flags:

  Incompatible change (1)
  , Reviewed

Description

Currently map-reduce applications (specifically file-based input-formats) use FileSystem.getFileBlockLocations to compute splits. However they are forced to call it once per file.
The downsides are multiple:

1. Even with a few thousand files to process the number of RPCs quickly starts getting noticeable
2. The current implementation of getFileBlockLocations is too slow since each call results in 'search' in the namesystem. Assuming a few thousand input files it results in that many RPCs and 'searches'.

It would be nice to have a FileSystem.getFileBlockLocations which can take in a directory, and return the block-locations for all files in that directory. We could eliminate both the per-file RPC and also the 'search' by a 'scan'.

When I tested this for terasort, a moderate job with 8000 input files the runtime halved from the current 8s to 4s. Clearly this is much more important for latency-sensitive applications...

• Attach one or more files to this issue
• More Options

  • Sort By Name
  • Sort By Date

  • Ascending
  • Descending

  • Manage Attachments

Attachments

1. 

   hdfsListFiles.patch

   23/Jul/10 19:03

   43 kB

   Hairong Kuang

2. 

   hdfsListFiles1.patch

   27/Jul/10 04:42

   40 kB

   Hairong Kuang

3. 

   hdfsListFiles2.patch

   31/Jul/10 00:07

   42 kB

   Hairong Kuang

4. 

   hdfsListFiles3.patch

   02/Aug/10 22:30

   54 kB

   Hairong Kuang

5. 

   hdfsListFiles4.patch

   11/Aug/10 18:20

   47 kB

   Hairong Kuang

6. 

   hdfsListFiles5.patch

   11/Aug/10 19:32

   48 kB

   Hairong Kuang

• Add Link

Issue Links

This issue blocks:
MAPREDUCE-1981	Improve getSplits performance by using listFiles, the new FileSystem API			Delete this link

This issue relates to:
HADOOP-6870	Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations			Delete this link

Activity

Ascending order - Click to sort in descending order

• All
• Comments
• Work Log
• History
• Activity
• Subversion Commits

Hide

Permalink

Doug Cutting added a comment - 08/May/09 17:27

An alternative to passing directories might be to pass a list of files. The request might get larger, but this is more precise, e.g., when only a subset of files in a directory will be used only that subset need be passed. Since globbing is client-side, this requires two round trips, one to list files and one to list their blocks, but that would still be a huge improvement over per-file RPC.

Show

Doug Cutting added a comment - 08/May/09 17:27 An alternative to passing directories might be to pass a list of files. The request might get larger, but this is more precise, e.g., when only a subset of files in a directory will be used only that subset need be passed. Since globbing is client-side, this requires two round trips, one to list files and one to list their blocks, but that would still be a huge improvement over per-file RPC.

Hide

Permalink

Doug Cutting added a comment - 08/May/09 17:48

How about adding something like:
Map<FileStatus, BlockLocation[]> listBlockLocations(Path[]);
This would permit a glob-free job to get everything it needs in a single RPC, and a globbing job to do so with two RPCs.

Show

Doug Cutting added a comment - 08/May/09 17:48 How about adding something like: Map<FileStatus, BlockLocation[]> listBlockLocations(Path[]); This would permit a glob-free job to get everything it needs in a single RPC, and a globbing job to do so with two RPCs.

Hide

Permalink

Arun C Murthy added a comment - 08/May/09 17:49

Map<FileStatus, BlockLocation[]> listBlockLocations(Path[]);

+1

Show

Arun C Murthy added a comment - 08/May/09 17:49

Map<FileStatus, BlockLocation[]> listBlockLocations(Path[]);

+1

Hide

Permalink

Konstantin Shvachko added a comment - 08/May/09 18:14

Currently getBlockLocations(src, offset, length) returns a class calledLocatedBlocks, which contains a list of LocatedBlock belonging to the file.

public class LocatedBlocks implements Writable {
  private long fileLength;
  private List<LocatedBlock> blocks; // array of blocks with prioritized locations
}

The question is whether we should modify LocatedBlocks, which would include the map proposed by Doug and extend the semantics ofgetBlockLocations() to handle directories, or should we introduce a new method (rpc)getBlockLocations(srcDir) returning LocatedBlockMap.
Is there a reason to keep current per file getBlockLocations() if we had a more generic method?

Show

Konstantin Shvachko added a comment - 08/May/09 18:14 Currently getBlockLocations(src, offset, length) returns a class called LocatedBlocks, which contains a list of LocatedBlock belonging to the file.

public class LocatedBlocks implements Writable {
  private long fileLength;
  private List<LocatedBlock> blocks; // array of blocks with prioritized locations
}

The question is whether we should modify LocatedBlocks, which would include the map proposed by Doug and extend the semantics of getBlockLocations() to handle directories, or should we introduce a new method (rpc) getBlockLocations(srcDir) returning LocatedBlockMap. Is there a reason to keep current per file getBlockLocations() if we had a more generic method?

Hide

Permalink

Doug Cutting added a comment - 08/May/09 18:28

> Is there a reason to keep current per file getBlockLocations() if we had a more generic method?

Not that I can think of. +1 for replacing it.

Show

Doug Cutting added a comment - 08/May/09 18:28 > Is there a reason to keep current per file getBlockLocations() if we had a more generic method? Not that I can think of. +1 for replacing it.

Hide

Permalink

dhruba borthakur added a comment - 10/May/09 09:40

If we adopt the approach that Doug has suggested, then the namenode still has to search for each input path in the file system namespace. This approach still has the advantage that the number of RPC calls are reduced. If we adopt Arun's proposal that specifies a directory and the RPC-call returns the splits of all the files in that directory, then it reduces the number of searches in the FS namespace as well as the number of RPC calls. I was kind-of leaning towards Arun's proposal, but Doug's approach is a little more flexible in nature, isn't it?

Show

dhruba borthakur added a comment - 10/May/09 09:40 If we adopt the approach that Doug has suggested, then the namenode still has to search for each input path in the file system namespace. This approach still has the advantage that the number of RPC calls are reduced. If we adopt Arun's proposal that specifies a directory and the RPC-call returns the splits of all the files in that directory, then it reduces the number of searches in the FS namespace as well as the number of RPC calls. I was kind-of leaning towards Arun's proposal, but Doug's approach is a little more flexible in nature, isn't it?

Hide

Permalink

Arun C Murthy added a comment - 11/May/09 17:50

Dhruba, I was thinking it was implict in Doug's proposal that if one of the paths in the Path[] is a directory, then the new api would return block-locations of all its' children (non-recursively?) which would satisfy the original requirement. Doug, can you please confirm?

Show

Arun C Murthy added a comment - 11/May/09 17:50 Dhruba, I was thinking it was implict in Doug's proposal that if one of the paths in the Path[] is a directory, then the new api would return block-locations of all its' children (non-recursively?) which would satisfy the