HBase的简单性能测试

测试服务器为Dell  E5410 的Debian linux 2.6，配置为：

1、4核，2.33GHz

2、内存3G

3、SATA硬盘2T

 

web服务器：tomcat5.5

打压工具：Apache Bench

Hadoop：0.20.2

HBase：0.20.6 ， 测试基于Hdfs 

 

测试思路：

1、因为HBase内置了连接池，所以客户端程序相对简单；

2、每个请求做1000次插入；

 

连接管理器：

package lab.winston;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.HBaseAdmin;

public class HbaseManager {
	static private HbaseManager instance; // 唯一实例
	static HBaseConfiguration cfg = null;

	static synchronized public  HbaseManager getInstance(final String ip, int port, int poolSize) throws IOException {
		if (instance == null) {
			instance = new HbaseManager(ip,port,poolSize);
		}		
		return instance;
	}

	private HbaseManager() {
	}
	
	private HbaseManager(final String ip, int port, int poolSize) throws IOException  {
		init(ip,port,poolSize);
	}

	public void init(final String ip, int port, int poolSize)
			throws IOException {
		if (cfg == null) {
	        Configuration HBASE_CONFIG = new Configuration();
	        HBASE_CONFIG.set("hbase.zookeeper.quorum", ip);
	        HBASE_CONFIG.set("hbase.zookeeper.property.clientPort", String.valueOf(port));
	        cfg = new HBaseConfiguration(HBASE_CONFIG);
	        
	        HBaseAdmin admin = new HBaseAdmin(cfg);
	        if (admin.tableExists("test")) {
	            System.out.println("table   Exists!!!");
	        }
	        else{
	            HTableDescriptor tableDesc = new HTableDescriptor("test");
	            tableDesc.addFamily(new HColumnDescriptor("name"));
	            admin.createTable(tableDesc);
	            System.out.println("create table ok .");
	        }
		}
	}
}

测试Servlet：

package lab.winston;

import java.io.IOException;
import java.util.UUID;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;

public class InsertHbaseServlet extends HttpServlet {

	public void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {
		Integer insertNum = Integer.valueOf(req.getParameter("insertNum")
				.toString());
		resp.setContentType("text/html;charset=UTF-8");
		resp.setHeader("Cache-Control", "no-cache");

		HBaseConfiguration cfg = HbaseManager.getInstance("127.0.0.1", 2181, 0).cfg;
		HTable table = new HTable(cfg, "test");
		for(int i=0;i<insertNum;i++){
			String uuid = UUID.randomUUID().toString();
			Put put = new Put(Bytes.toBytes(uuid));
			put.add(Bytes.toBytes("name"), Bytes.toBytes("id"), Bytes
					.toBytes("10000"));		
			put.add(Bytes.toBytes("name"), Bytes.toBytes("value"), Bytes
					.toBytes("this is hbase insert!"));
			table.put(put);			
		}
		System.out.println("add data ok. insertNum:"+insertNum);
		resp.getWriter().write(cfg + ": Insert hbase successed!");
	}

	public void doPost(HttpServletRequest request, HttpServletResponse response)
			throws ServletException, IOException {
	}
}

5个线程并发，插入20万记录：

ab -n 200 -c 5 http://localhost:8080/labWeb/InsertHbase.do?insertNum=1000

结果为：

Concurrency Level:      5
Time taken for tests:   40.251845 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      66000 bytes
HTML transferred:       28600 bytes
Requests per second:    4.97 [#/sec] (mean)
Time per request:       1006.296 [ms] (mean)
Time per request:       201.259 [ms] (mean, across all concurrent requests)
Transfer rate:          1.59 [Kbytes/sec] received

cpu负载：90%以上

耗时40秒，平均每秒插入（QPS）为5000。

 

10个线程并发，插入20万记录：

ab -n 200 -c 10 http://localhost:8080/labWeb/InsertHbase.do?insertNum=1000

结果为：

Concurrency Level:      10
Time taken for tests:   56.608753 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      66000 bytes
HTML transferred:       28600 bytes
Requests per second:    3.53 [#/sec] (mean)
Time per request:       2830.438 [ms] (mean)
Time per request:       283.044 [ms] (mean, across all concurrent requests)
Transfer rate:          1.13 [Kbytes/sec] received

cpu负载：90%以上

耗时56秒，平均每秒插入（QPS）为3600。