HDFS Details for Multimachine Clusters(2nd)
Checking the NameNodes
Checking the DataNodes
bin/slaves.sh jps | grep Datanode | sort
In fact, I had half of a new cluster fail to start, and it took some time to realize that the newly installed machines had a default firewall that blocked the HDFS port.
bin/hadoop dfsadmin -report 可以查看当前在线的datanode的部分信息
most important factors are network bandwidth and disk throughput. Memory use and CPU overhead for thread handling may also be issues.
The large input-split size reduces the ratio of task setup time to task run time.
Set the maximum number of requests in progress. the more requests in progress, the more contention there is for storage operations and network bandwidth, with a corresponding increase in memory requirements and CPU overhead for handling all of the outstanding requests.
Different factors per cluster.
File Descriptors (http://en.wikipedia.org/wiki/File_descriptor )
Any user that runs processes that access HDFS should have a large limit on file descriptor access, and all applications that open files need careful checking to make sure that the files are explicitly closed.