In this test, use NNThroughputBenchmark to create directories, the dirs parameter is always 10000, and dirsPerDir is 128. Cpu cores is 8.
set handler count to 2
<property>
<name>dfs.namenode.handler.count</name>
<value>2</value>
</property>
test with 2 threads, Elapsed Time: 102022
[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 2 -dirs 10000 -dirsPerDir 128
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:45:10 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark:
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrThreads = 2
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: --- mkdirs stats ---
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Elapsed Time: 102022
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Ops per sec: 98.01807453294387
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Average Time: 20
test with 20 threads, Elapsed Time: 102154
This indicates that increase the client threads will not result in the performance increase.
[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:47:33 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark:
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: --- mkdirs stats ---
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Elapsed Time: 102154
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Ops per sec: 97.89141883822464
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Average Time: 204
set handler count to 20, and restart the namenode.
<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
</property>
test with 20 threads, Elapsed Time: 9799
This indicates that performance improved 10 times.
hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:50:38 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark:
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: --- mkdirs stats ---
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Elapsed Time: 9799
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Ops per sec: 1020.512297173181
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Average Time: 19
test with 30 threads, Elapsed Time: 9695
This indicates that performance is the same as test with 20 threads.
[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 30 -dirs 10000 -dirsPerDir 128
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:52:32 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark:
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrThreads = 30
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: --- mkdirs stats ---
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Elapsed Time: 9695
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Ops per sec: 1031.4595152140278
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Average Time: 28
set handler count to 200, and restart the namenode.
<property>
<name>dfs.namenode.handler.count</name>
<value>200</value>
</property>
test with 20 threads, Elapsed Time: 11288
[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 14:17:39 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark:
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: --- mkdirs stats ---
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Elapsed Time: 11288
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Ops per sec: 885.896527285613
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Average Time: 22
test with 200 threads, Elapsed Time: 1285
Indicates that performance improved 10 times.
[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 200 -dirs 10000 -dirsPerDir 128
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 14:23:27 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark:
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrThreads = 200
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: --- mkdirs stats ---
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Elapsed Time: 1285
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Ops per sec: 7782.101167315175
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Average Time: 23