轻松把玩HttpClient之封装HttpClient工具类(八),优化启用Http连接池策略

       写了HttpClient工具类后,有人一直在问我怎么启用http连接池,其实我没考虑过这个问题难过。不过闲暇的时候,突然间想起了这个问题,就想把这个问题搞一搞。

       之前没用过,但是理解起来应该不算难。作为一个Coder,就算没用过http连接池,但是肯定用过数据库连接池。二者的功能是类似的,就是把建立链接和断开链接的时间节省下来。众所周知,http建立链接是需要3次握手,了解的深入一些的,还知道断开链接需要4次握手。这些操作都是自动完成的,如果我们把这些建立和断开链接的时间节省掉,对于大批量的http请求(如爬虫)是很有用的。

       关于HttpClient如何启用连接池,可以参考这篇文章:http://www.cnblogs.com/likaitai/p/5431246.html。介绍了如何通过连接池获取链接,以及在不用连接时,如果处理不会导致链接直接关闭。

       说了这么多,下面切入正题:HttpClient工具类如何启用http连接池?其实只需要修改创建链接方法即可:

  之前在工具类的核心方法execute方法里获取httpclient对象,调用的是create(String url)方法。返回的是默认的一个HttpClient对象。现在要启用连接池,必须修改此方法。配置连接池的类是HCB,而execute方法接受的参数是HttpConfig参数,所以,首先要在HttpConfig里添加一个HCB对象。然后修改create方法。具体如下:
	/**
	 * HCB对象,用于创建HttpClient对象
	 */
	private HCB hcb;

	public HCB hcb() {
		return hcb;
	}

	/**
	 * HCB对象,用于自动从连接池中获得HttpClient对象<br>
	 * <font color="red"><b>请调用hcb.pool方法设置连接池</b></font>
	 * @throws HttpProcessException 
	 */
	public HttpConfig hcb(HCB hcb) throws HttpProcessException {
		this.hcb = hcb;
		return this;
	}
	/**
	 * 判定是否开启连接池、及url是http还是https <br>
	 * 		如果已开启连接池,则自动调用build方法,从连接池中获取client对象<br>
	 * 		否则,直接返回相应的默认client对象<br>
	 * 
	 * @throws HttpProcessException 
	 */
	private static void create(HttpConfig config) throws HttpProcessException  {
		if(config.hcb()!=null && config.hcb().isSetPool){ //如果设置了hcb对象,且配置了连接池,则直接从连接池取
			if(config.url().toLowerCase().startsWith("https://")){
				config.client(config.hcb().ssl().build());
			}else{
				config.client(config.hcb().build());
			}
		}else{
			if(config.client()==null){//如果为空,设为默认client对象
				if(config.url().toLowerCase().startsWith("https://")){
					config.client(client4HTTPS);
				}else{
					config.client(client4HTTP);
				}
			}
		}
	}
       至于关闭方面,fmt2String以及fmt2Stream方法中,在EntityUtils.toString和EntityUtils.consume方法中已经close了instream,释放了资源。最后调用close(HttpClient)即执行client.close()方法。这样就不会直接关闭链接了,会被连接池自动回收再次使用。

       最后分享一个测试类,分组测试Get请求、Down操作,在开启和关闭Http线程池完成请求的耗时情况。代码如下:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import org.apache.http.Header;

import com.tgb.ccl.http.common.HttpConfig;
import com.tgb.ccl.http.common.HttpHeader;
import com.tgb.ccl.http.exception.HttpProcessException;
import com.tgb.ccl.http.httpclient.HttpClientUtil;
import com.tgb.ccl.http.httpclient.builder.HCB;

/**
 * 测试启用http连接池
 * 
 * @author arron
 * @date 2016年11月7日 下午1:08:07 
 * @version 1.0
 */
public class TestHttpPool {
	
	// 设置header信息
	private static final Header[] headers = HttpHeader.custom().userAgent("Mozilla/5.0").from("http://blog.csdn.net/newest.html").build();
	
	// URL列表数组,GET请求
	private static final String[] urls = {
			"http://blog.csdn.net/xiaoxian8023/article/details/49883113",
			"http://blog.csdn.net/xiaoxian8023/article/details/49909359",
			"http://blog.csdn.net/xiaoxian8023/article/details/49910127",
			"http://blog.csdn.net/xiaoxian8023/article/details/49910885",
			"http://blog.csdn.net/xiaoxian8023/article/details/51606865",
	};
	
	// 图片URL列表数组,Down操作
	private static final String[] imgurls ={
			"http://ss.bdimg.com/static/superman/img/logo/logo_white_fe6da1ec.png",
			"https://scontent-hkg3-1.xx.fbcdn.net/hphotos-xaf1/t39.2365-6/11057093_824152007634067_1766252919_n.png"
	};
	
	private static StringBuffer buf1=new StringBuffer();
	private static StringBuffer buf2=new StringBuffer();
	
	//多线程get请求
	public static void testMultiGet(HttpConfig cfg, int count) throws HttpProcessException{
	        try {
				int pagecount = urls.length;
				ExecutorService executors = Executors.newFixedThreadPool(pagecount);
				CountDownLatch countDownLatch = new CountDownLatch(count);   
				//启动线程抓取
				for(int i = 0; i< count;i++){
				    executors.execute(new GetRunnable(countDownLatch,cfg.headers(headers).url(urls[i%pagecount])));
				}
				countDownLatch.await();
				executors.shutdown();
			} catch (InterruptedException e) {
				e.printStackTrace();
	        }
	}
	
	//多线程下载
	public static void testMultiDown(HttpConfig cfg, int count) throws HttpProcessException{
		try {
			int pagecount = imgurls.length;
			ExecutorService executors = Executors.newFixedThreadPool(pagecount);
			CountDownLatch countDownLatch = new CountDownLatch(count);   
			//启动线程抓取
			for(int i = 0; i< count;i++){
			    executors.execute(new GetRunnable(countDownLatch, cfg.url(imgurls[i%2]), new FileOutputStream(new File("d://aaa//"+(i+1)+".png"))));
			}
			countDownLatch.await();
			executors.shutdown();
		} catch (FileNotFoundException e) {
			e.printStackTrace();
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
	}
	
	 static class GetRunnable implements Runnable {
	        private CountDownLatch countDownLatch;
	        private HttpConfig config = null;
	        private FileOutputStream out = null;

	        public GetRunnable(CountDownLatch countDownLatch,HttpConfig config){
	           this(countDownLatch, config, null);
	        }
	        public GetRunnable(CountDownLatch countDownLatch,HttpConfig config,FileOutputStream out){
	        	this.countDownLatch = countDownLatch;
	        	this.config = config;
	        	this.out = out;
	        }
	        
	        @Override
	        public void run() {
	            try {
	            	config.out(out);
	            	if(config.out()==null){
	            		String response = null;
	            		response =  HttpClientUtil.get(config);
	            		System.out.println(Thread.currentThread().getName()+"--获取内容长度:"+response.length());
	            		response = null;

	            	}else{
	            		HttpClientUtil.down(config);
	            		try {
							config.out().flush();
							config.out().close();
						} catch (IOException e) {
							e.printStackTrace();
						}
	            		System.out.println(Thread.currentThread().getName()+"--下载完毕");
	            	}
	            } catch (HttpProcessException e) {
					e.printStackTrace();
				} finally {
	                countDownLatch.countDown();
	            }
	        }
	    }  


	/**
	 * 测试不启用http连接池,get100次,down20次的执行时间
	 * @throws HttpProcessException
	 */
	private static void testNoPool(int getCount, int downCount) throws HttpProcessException {
		long start = System.currentTimeMillis();

		if(getCount>0){
			HttpConfig cfg1 = HttpConfig.custom().client(HCB.custom().build()).headers(headers);
			testMultiGet(cfg1, getCount);
		}
		if(downCount>0){
			HttpConfig cfg2 = HttpConfig.custom().client(HCB.custom().build());
			testMultiDown(cfg2, downCount);
		}
		
		System.out.println("-----所有线程已完成!------");
        long end = System.currentTimeMillis();
        System.out.println("总耗时(毫秒): -> " + (end - start));
        buf1.append("\t").append((end-start));
	}

	
	/**
	 * 测试启用http连接池,get100次,down20次的执行时间
	 * @throws HttpProcessException
	 */
	private static void testByPool(int getCount, int downCount) throws HttpProcessException {
		long start = System.currentTimeMillis();
		
		HCB hcb= HCB.custom().timeout(10000).pool(10, 10).ssl();
		
		if(getCount>0){
			HttpConfig cfg3 = HttpConfig.custom().hcb(hcb);
			testMultiGet(cfg3, getCount);
		}
		if(downCount>0){
			HttpConfig cfg4 = HttpConfig.custom().hcb(hcb);
			testMultiDown(cfg4, downCount);
		}

		System.out.println("-----所有线程已完成!------");
        long end = System.currentTimeMillis();
        System.out.println("总耗时(毫秒): -> " + (end - start));
        buf2.append("\t").append((end-start));
	}
	
	public static void main(String[] args) throws Exception {
		File file = new File("d://aaa");
		if(!file.exists() && file.isDirectory()){
			file.mkdir();
		}
		
		//-------------------------------------------
		//  以下2个方法
		//  分别测试 (get次数,down次数) 
		//  {100,0},{200,0},{500,0},{1000,0}
		//  {0,10},{0,20},{0,50},{0,100}
		//  {100,10},{200,20},{500,50},{1000,100}
		//-------------------------------------------
		
		int[][] times1 = {
   {100,0} ,{ 200,0 },{ 500,0 },{ 1000,0}};
		int[][] times2 = {
   {0,10},{0,20},{0,50},{0,100}};
		int[][] times3 = {
   {100,10},{200,20},{500,50},{1000,100}};
		List<int[][]> list = Arrays.asList(times1,times2,times3);
		int n=5;
		
		int t=0;
		//测试未启用http连接池,
		for (int[][] time : list) {
			buf1.append("\n");
			for (int i = 0; i < time.length; i++) {
				for (int j = 0; j < n; j++) {
					testNoPool(time[i][0],time[i][1]);
					Thread.sleep(100);
					System.gc();
					Thread.sleep(100);
				}
				buf1.append("\n");
			}
		}

		t=0;
		//测试启用http连接池
		for (int[][] time : list) {
			buf2.append("\n");
			for (int i = 0; i < time.length; i++) {
				for (int j = 0; j < n; j++) {
					testByPool(time[i][0],time[i][1]);
					Thread.sleep(100);
					System.gc();
					Thread.sleep(100);
				}
				buf2.append("\n");
			}
			t++;
		}
		
		//把结果打印到Console中
		String[] results1 = buf1.toString().split("\n");
		String[] results2 = buf2.toString().split("\n");
		
		for (int i = 0; i < results1.length; i++) {
			System.out.println(results1[i]);
			System.out.println(results2[i]);
		}
		
	}
}
       测试结果如下:
操作 请求次数 是否启用Pool 第1次
  • 2
    点赞
  • 14
    收藏
    觉得还不错? 一键收藏
  • 6
    评论
你可以使用 Apache HttpComponents 中的 HttpClient 来进行 https 请求,并使用连接池来提高性能。下面是一个示例代码: ```java import org.apache.http.HttpEntity; import org.apache.http.HttpResponse; import org.apache.http.client.HttpClient; import org.apache.http.client.config.RequestConfig; import org.apache.http.client.entity.UrlEncodedFormEntity; import org.apache.http.client.methods.HttpPost; import org.apache.http.config.Registry; import org.apache.http.config.RegistryBuilder; import org.apache.http.conn.socket.ConnectionSocketFactory; import org.apache.http.conn.ssl.SSLConnectionSocketFactory; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClientBuilder; import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; import org.apache.http.message.BasicNameValuePair; import org.apache.http.ssl.SSLContexts; import org.apache.http.util.EntityUtils; import javax.net.ssl.SSLContext; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.security.KeyManagementException; import java.security.KeyStoreException; import java.security.NoSuchAlgorithmException; import java.util.ArrayList; import java.util.List; public class HttpsClientUtil { private static final int MAX_TOTAL_CONNECTIONS = 200; private static final int MAX_PER_ROUTE_CONNECTIONS = 20; private static final int CONNECTION_TIMEOUT_MS = 5000; private static final int SOCKET_TIMEOUT_MS = 10000; private static CloseableHttpClient httpClient; static { try { SSLContext sslContext = SSLContexts.custom().build(); SSLConnectionSocketFactory socketFactory = new SSLConnectionSocketFactory(sslContext, new String[]{"TLSv1.2"}, null, SSLConnectionSocketFactory.getDefaultHostnameVerifier()); Registry<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create() .register("https", socketFactory) .build(); PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(registry); connectionManager.setMaxTotal(MAX_TOTAL_CONNECTIONS); connectionManager.setDefaultMaxPerRoute(MAX_PER_ROUTE_CONNECTIONS); RequestConfig requestConfig = RequestConfig.custom() .setConnectTimeout(CONNECTION_TIMEOUT_MS) .setSocketTimeout(SOCKET_TIMEOUT_MS) .build(); httpClient = HttpClientBuilder.create() .setConnectionManager(connectionManager) .setDefaultRequestConfig(requestConfig) .build(); } catch (NoSuchAlgorithmException | KeyManagementException | KeyStoreException e) { throw new RuntimeException(e); } } public static String post(String url, List<BasicNameValuePair> parameters) throws IOException { HttpPost httpPost = new HttpPost(url); httpPost.setEntity(new UrlEncodedFormEntity(parameters, StandardCharsets.UTF_8)); HttpResponse response = httpClient.execute(httpPost); HttpEntity entity = response.getEntity(); return EntityUtils.toString(entity, StandardCharsets.UTF_8); } } ``` 使用方法: ```java List<BasicNameValuePair> parameters = new ArrayList<>(); parameters.add(new BasicNameValuePair("key1", "value1")); parameters.add(new BasicNameValuePair("key2", "value2")); String response = HttpsClientUtil.post("https://example.com/api", parameters); ``` 此处仅提供一个简单的示例,实际使用时需要根据具体情况进行调整和扩展。
评论 6
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值