[转]Increasing the handler count can improve the performance of NameNode significantly

 

In this test, use NNThroughputBenchmark to create directories, the dirs parameter is always 10000, and dirsPerDir is 128. Cpu cores is 8.

 

set handler count to 2

 

  <property>
    <name>dfs.namenode.handler.count</name>
    <value>2</value>
  </property>

 

 

test with 2 threads, Elapsed Time: 102022

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 2 -dirs 10000 -dirsPerDir 128
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:45:10 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrThreads = 2
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Elapsed Time: 102022
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark:  Ops per sec: 98.01807453294387
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Average Time: 20

 

 

test with 20 threads, Elapsed Time: 102154

 

This indicates that increase the client threads will not result in the performance increase.

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:47:33 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Elapsed Time: 102154
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark:  Ops per sec: 97.89141883822464
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Average Time: 204

 

 

set handler count to 20, and restart the namenode.

 

  <property>
    <name>dfs.namenode.handler.count</name>
    <value>20</value>
  </property>

 

 

test with 20 threads, Elapsed Time: 9799

 

This indicates that performance improved 10 times.

 

hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:50:38 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Elapsed Time: 9799
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark:  Ops per sec: 1020.512297173181
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Average Time: 19

 

 

test with 30 threads, Elapsed Time: 9695

 

This indicates that performance is the same as test with 20 threads.

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 30 -dirs 10000 -dirsPerDir 128
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:52:32 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrThreads = 30
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Elapsed Time: 9695
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark:  Ops per sec: 1031.4595152140278
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Average Time: 28

 

 

set handler count to 200, and restart the namenode.

 

  <property>
    <name>dfs.namenode.handler.count</name>
    <value>200</value>
  </property>

 

 

test with 20 threads, Elapsed Time: 11288

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 14:17:39 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: 
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Elapsed Time: 11288
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark:  Ops per sec: 885.896527285613
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Average Time: 22

 

 

test with 200 threads, Elapsed Time: 1285

 

Indicates that performance improved 10 times.

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 200 -dirs 10000 -dirsPerDir 128
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 14:23:27 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: 
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrThreads = 200
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Elapsed Time: 1285
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark:  Ops per sec: 7782.101167315175
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Average Time: 23

 

 

set handler count to 2000, and restart the namenode, but there is no performance improvement. So it is safe to set handler to about 25 times of cores.

---------------------
作者：houzhizhen
来源：CSDN
原文：https://blog.csdn.net/houzhizhen/article/details/78520100
版权声明：本文为作者原创文章，转载请附上博文链接！