Finding the most accessed Table/Region on an HBase Region Server

来源: http://whynosql.com/finding-the-most-accessed-tableregion-on-an-hbase-region-server/

 

要翻墙！

 

Each HBase region server hosts many regions – possibly hundreds or even thousands. How do you find out which one of them is a hotspot? We saw that CPU on one of the region server was shooting up at peak traffic. But the region server had 4 tables (and hundreds of regions) and their access patterns were similar. Each web request was accessing all four of the tables. So how do you find out which table is the bottleneck? Which region(s) is causing hotspotting?

Unfortunately, HBase does not provide any tools for finding out which region is causing hotspotting. You can see HBase committers agreeing to this in this mailing list thread . You will have to simply monitor the traffic between your web servers and the region server and find out which table/region is being accessed the most.

Taking a clue from the mailing list thread, we found out which region is causing Hotspotting using ngrep command. Ngrep helps you sniff traffic. We first dumped a few seconds of ngrep output into a a file by executing the following command on the region server:

ngrep -W byline port 60020 > temp.out


This command created a file temp.out in the current directory. Note that the 60020 is the region server’s default port. -W byline option asks ngrep to organize information in lines. Once you have ngrep output on file, you can simply execute the following command to analyze it:


grep 'TableName,' temp.out | wc -l


The above command printed a count of lines containing “TableName, “. These many requests were made to the region server from client(s).

The reason we tried searching for “TableName, ” was because that’s how region names are named in HBase (as of 0.20.6). A region name is named as

[TableName],[key],[encoded region name]

Anytime a request is made to a region, you will find the region name in the request. This also means that the request will have the table name too. For us, finding out which table got the most traffic was sufficient. However, if you want to find out what the specific region is, you can easily come up with a few simple lines of code that can parse ngrep output (temp.out above) to give you the number of requests sent to each region.