hadoop 源码笔记

  

aprotected List<FileStatus> listStatus(JobContext job
                                        ) throws IOException {
    List<FileStatus> result = new ArrayList<FileStatus>();
    Path[] dirs = getInputPaths(job);
    if (dirs.length == 0) {
      throw new IOException("No input paths specified in job");
    }

    List<IOException> errors = new ArrayList<IOException>();
   
    // creates a MultiPathFilter with the hiddenFileFilter and the
    // user provided one (if any).
    List<PathFilter> filters = new ArrayList<PathFilter>();
    filters.add(hiddenFileFilter);
    PathFilter jobFilter = getInputPathFilter(job);
    if (jobFilter != null) {
      filters.add(jobFilter);
    }
    PathFilter inputFilter = new MultiPathFilter(filters);
   
    for (int i=0; i < dirs.length; ++i) {
      Path p = dirs[i];
      FileSystem fs = p.getFileSystem(job.getConfiguration());
      FileStatus[] matches = fs.globStatus(p, inputFilter);
      if (matches == null) {
        errors.add(new IOException("Input path does not exist: " + p));
      } else if (matches.length == 0) {
        errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
      } else {
        for (FileStatus globStat: matches) {
          if (globStat.isDir()) {
            for(FileStatus stat: fs.listStatus(globStat.getPath(),
                inputFilter)) {
              result.add(stat);
            }         
          } else {
            result.add(globStat);
          }
        }
      }
    }

    if (!errors.isEmpty()) {
      throw new InvalidInputException(errors);
    }
    LOG.info("Total input paths to process : " + result.size());
    return result;
  }

 

 

rpc 由client server
client有call() Connection() ParallelCal()l ParallelResults() ConnectionID()等几个方法

Connection由ConnectionID来区分，每个ConnectionId包括一个InetSocketAddress（IP地址+端口号或主机名+端口号）对象和一个用户信息对象
private static class ConnectionId {
    InetSocketAddress address;//IP_PortNum or HostName+port
    UserGroupInformation ticket;//用户信息对象
    Class<?> protocol;//在不确定传过去的类型是 可以使用这种代替
....
}

所有的链接都被封装在Client.Connection 中，所有的PRC调用都是通过Connection调用。Connection是一个线程实例。
为了区分在同一个Connection上的不同调用，每个调用都有唯一的id。调用是否结束也需要一个标记，所有的这些都体现在对象Client.Call中