《Hadoop The Definitive Guide》ch10 Administering Hadoop

1. 集群基准测试程序

c01s02h01ate1:nomad2 # hadoop jar /local/hadoop/hadoop-0.20.203.0/hadoop-test-0.20.203.0.jar 
An example program must be given as the first argument.
Valid program names are:
  DFSCIOTest: Distributed i/o benchmark of libhdfs.
  DistributedFSCheck: Distributed checkup of the file system consistency.
  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  TestDFSIO: Distributed i/o benchmark.
  dfsthroughput: measure hdfs throughput
  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  loadgen: Generic map/reduce load generator
  mapredtest: A map/reduce test check.
  mrbench: A map/reduce benchmark that can create many small jobs
  nnbench: A benchmark that stresses the namenode.
  testarrayfile: A test for flat files of binary key/value pairs.
  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  testfilesystem: A test for FileSystem read/write.
  testipc: A test for ipc.
  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  testrpc: A test for rpc.
  testsequencefile: A test for flat files of binary key value pairs.
  testsequencefileinputformat: A test for sequence file input format.
  testsetfile: A test for flat files of binary key/value pairs.
  testtextinputformat: A test for text input format.
  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

c01s02h01ate1:nomad2 # hadoop jar /local/hadoop/hadoop-0.20.203.0/hadoop-test-0.20.203.0.jar DFSCIOTest
TestDFSIO.0.0.4
Usage: TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileName] [-bufferSize Bytes] 

2.  safe mode

c01s02h01ate1:nomad2 # hadoop dfsadmin -safemode get
Safe mode is OFF

c01s02h01ate1:nomad2 # hadoop dfsadmin -refreshNodes

3.  web界面

http://c01s0201ate2:50075/blockScannerReport

http://c01s0201ate1:50030/logLevel?log=org.apache.hadoop.mapred.JobTracker

http://c01s0201ate1:50030/stacks

4. 负载平衡

c01s02h01ate1:nomad2 # start-balancer.sh 
starting balancer, logging to /local/hadoop/hadoop-0.20.203.0/bin/../logs/hadoop-nomad2-balancer-c01s02h01ate1.out
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
The cluster is balanced. Exiting...
Balancing took 413.0 milliseconds