apache httpclient早已在我当成毕业设计（12306抢票软件）时就已经用上了，去模拟用户请求，然后实现自动订票功能。包括现在的工作里面也还一直在用httpclient，但用的功能都非常浅，也没有深入去了解。最近决定优化一下jnavi项目，所以就潜下心来好好的读一读httpclient代码吧。
其实现在httpclient已经end of life了，取而代之的是现在的Apache HttpComponents ，而HttpComponents又分为 HttpClient 和 HttpCore 模块了。按照版本来说也就是httpclient 3.X就是之前的httpclient，已经不再维护了。现在的HttpComponents也就是httpclient4.X开始的。
http://hc.apache.org/httpclient-3.x/
The Commons HttpClient project is now end of life, and is no longer being developed. It has been replaced by the Apache HttpComponents project in its HttpClient and HttpCore modules, which offer better performance and more flexibility.
现在的HttpComponents 主页：
http://hc.apache.org/
HttpComponents主要分为3个模块，
httpcore：最新GA版本4.4.4 ，底层的核心包，提供了一些核心的http功能，可以供http client端和server端使用。其提供BIO和NIO两种传输模式。
httpclient：最新GA版本4.5.2，实现了http 1.1协议，基于httpcore，用于客户端开发。
AsyncClient：最新GA版本4.1.1，基于httpcore和httpclient，提供了支持异步处理的功能。
异步客户端例子：
http://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
同步并发请求
由于我们项目中用到了30多个底层接口，我们需要用httpclient分别去请求每个接口，并合并，处理结果最后给我们服务的调用方返回，目前我们用的还是线程池加同步的httpclient来实现的。其中主要用了httpclient的PoolingClientConnectionManager来实现并发请求。
defaultMaxPerRoute ： 默认每个路由的最大连接数，可以分别设置，每个路由就是 ip：port
maxTotal ： 即此PoolingClientConnectionManager的最大连接个数。
timeToLive ： 连接的最大生存时间，0表示不限
connectTimeout ： 连接超时时间（表示多长时间能连接上，建立socket的时间，client 发出 syn 包，server端在你指定的时间内没有回复ack）
socketTimeout ： 读取超时时间
其中defaultMaxPerRoute和maxTotal的关系是，首先根据当前route判断当前是否到达MaxPerRoute，没有的话，在和maxTotal做比较，都没有的话则创建连接。
此块最核心的代码在AbstractConnPool.getPoolEntryBlocking方法里面
private E getPoolEntryBlocking(
final T route, final Object state,
final long timeout, final TimeUnit tunit,
final PoolEntryFuture<E> future)
throws IOException, InterruptedException, TimeoutException {
Date deadline = null;
if (timeout > 0) {
deadline = new Date
(System.currentTimeMillis() + tunit.toMillis(timeout));
}
this.lock.lock();
try {
RouteSpecificPool<T, C, E> pool = getPool(route);
E entry = null;
while (entry == null) {
if (this.isShutDown) {
throw new IllegalStateException("Connection pool shut down");
}
for (;;) {//先从空闲的连接里面取
entry = pool.getFree(state);
if (entry == null) {
break;
}
if (entry.isClosed() || entry.isExpired(System.currentTimeMillis())) {
entry.close();
this.available.remove(entry);
pool.free(entry, false);
} else {
break;
}
}
if (entry != null) {
this.available.remove(entry);
this.leased.add(entry);
return entry;
}
// New connection is needed
int maxPerRoute = getMax(route);
// Shrink the pool prior to allocating a new connection
int excess = Math.max(0, pool.getAllocatedCount() + 1 - maxPerRoute);
if (excess > 0) {
for (int i = 0; i < excess; i++) {
E lastUsed = pool.getLastUsed();
if (lastUsed == null) {
break;
}
lastUsed.close();
this.available.remove(lastUsed);
pool.remove(lastUsed);
}
}
if (pool.getAllocatedCount() < maxPerRoute) { //先和maxPerRoute作比较
int totalUsed = this.leased.size();
int freeCapacity = Math.max(this.maxTotal - totalUsed, 0);
if (freeCapacity > 0) {//再和maxTotal作比较
int totalAvailable = this.available.size();
if (totalAvailable > freeCapacity - 1) {
if (!this.available.isEmpty()) {
E lastUsed = this.available.removeLast();
lastUsed.close();
RouteSpecificPool<T, C, E> otherpool = getPool(lastUsed.getRoute());
otherpool.remove(lastUsed);
}
}
C conn = this.connFactory.create(route);//创建新的连接
entry = pool.add(conn);
this.leased.add(entry);
return entry;
}
}
boolean success = false;
try {
pool.queue(future);
this.pending.add(future);
success = future.await(deadline);//等待空闲的连接
} finally {
// In case of 'success', we were woken up by the
// connection pool and should now have a connection
// waiting for us, or else we're shutting down.
// Just continue in the loop, both cases are checked.
pool.unqueue(future);
this.pending.remove(future);
}
// check for spurious wakeup vs. timeout
if (!success && (deadline != null) &&
(deadline.getTime() <= System.currentTimeMillis())) {
break;
}
}
throw new TimeoutException("Timeout waiting for connection");//等待超时,超过connectTimeout
} finally {
this.lock.unlock();
}
}
异步并发请求
由于我们最近在做httpclient方面的优化，发现如果把现有的同步请求改成异步请求，那么业务里面则不需要再使用线程池，而且不必为每个请求再分配一个线程来执行，将大大减少业务线程的数量，节省不少的线程上下文切换开销，给机器性能提升不少。
还是来先看下apache AsyncClient的例子吧，发现非常有用，适合我们现在的使用方法。
第一个例子，展示了异步请求url，并有响应时，可以定义一个parser去解析响应，并通过future的方式来返回结果。更重要的是提供了一个callback主动提示完成的消息，需要的时候，可以避免通过调用future的get方法造成的阻塞情况。
package cn.myroute.httpcompont;
import java.io.IOException;
import java.nio.CharBuffer;
import java.util.concurrent.Future;
import org.apache.http.HttpResponse;
import org.apache.http.concurrent.FutureCallback;
import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.nio.IOControl;
import org.apache.http.nio.client.methods.AsyncCharConsumer;
import org.apache.http.nio.client.methods.HttpAsyncMethods;
import org.apache.http.protocol.HttpContext;
/**
* This example demonstrates an asynchronous HTTP request / response exchange with
* a full content streaming.
*/
public class AsyncClientHttpExchangeStreaming {
public static void main(final String[] args) throws Exception {
CloseableHttpAsyncClient httpclient = HttpAsyncClients.createDefault();
try {
FutureCallback<Boolean> callback = new FutureCallback<Boolean>() {
@Override
public void cancelled() {
// TODO Auto-generated method stub
System.out.println("cancelled");
}
@Override
public void completed(Boolean arg0) {
// TODO Auto-generated method stub
System.out.println("completed");
}
@Override
public void failed(Exception arg0) {
// TODO Auto-generated method stub
System.out.println("failed");
}
};
httpclient.start();
Future<Boolean> future = httpclient.execute(
HttpAsyncMethods.createGet("http://10.103.29.188/pay/get.json"),
new MyResponseConsumer(), callback);
Boolean result = future.get();
if (result != null && result.booleanValue()) {
System.out.println("Request successfully executed");
} else {
System.out.println("Request failed");
}
System.out.println("Shutting down");
} finally {
httpclient.close();
}
System.out.println("Done");
}
static class MyResponseConsumer extends AsyncCharConsumer<Boolean> {
@Override
protected void onResponseReceived(final HttpResponse response) {
System.out.println("onResponseReceived");
}
@Override
protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) throws IOException {
while (buf.hasRemaining()) {
System.out.print(buf.get());
}
}
@Override
protected void releaseResources() {
System.out.println("releaseResources");
}
@Override
protected Boolean buildResult(final HttpContext context) {
System.out.println("buildResult");
return Boolean.TRUE;
}
}
}第二个例子，展示了使用PoolingNHttpClientConnectionManager达到并发异步请求的效果，模仿的同步的PoolingClientConnectionManager。
package cn.myroute.httpcompont;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.concurrent.FutureCallback;
import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.nio.conn.NHttpClientConnectionManager;
import org.apache.http.nio.reactor.ConnectingIOReactor;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
/**
* Example demonstrating how to evict expired and idle connections
* from the connection pool.
*/
public class AsyncClientEvictExpiredConnections {
public static void main(String[] args) throws Exception {
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor);
cm.setMaxTotal(100);
CloseableHttpAsyncClient httpclient = HttpAsyncClients.custom()
.setConnectionManager(cm)
.build();
try {
httpclient.start();
// create an array of URIs to perform GETs on
String[] urisToGet = {
"http://hc.apache.org/",
"http://hc.apache.org/httpcomponents-core-ga/",
"http://hc.apache.org/httpcomponents-client-ga/",
};
IdleConnectionEvictor connEvictor = new IdleConnectionEvictor(cm);
connEvictor.start();
final CountDownLatch latch = new CountDownLatch(urisToGet.length);
for (final String uri: urisToGet) {
final HttpGet httpget = new HttpGet(uri);
httpclient.execute(httpget, new FutureCallback<HttpResponse>() {
@Override
public void completed(final HttpResponse response) {
latch.countDown();
System.out.println(httpget.getRequestLine() + "->" + response.getStatusLine());
}
@Override
public void failed(final Exception ex) {
latch.countDown();
System.out.println(httpget.getRequestLine() + "->" + ex);
}
@Override
public void cancelled() {
latch.countDown();
System.out.println(httpget.getRequestLine() + " cancelled");
}
});
}
latch.await();
// Sleep 10 sec and let the connection evictor do its job
Thread.sleep(20000);
// Shut down the evictor thread
connEvictor.shutdown();
connEvictor.join();
} finally {
httpclient.close();
}
}
public static class IdleConnectionEvictor extends Thread {
private final NHttpClientConnectionManager connMgr;
private volatile boolean shutdown;
public IdleConnectionEvictor(NHttpClientConnectionManager connMgr) {
super();
this.connMgr = connMgr;
}
@Override
public void run() {
try {
while (!shutdown) {
synchronized (this) {
wait(5000);
// Close expired connections
connMgr.closeExpiredConnections();
// Optionally, close connections
// that have been idle longer than 5 sec
connMgr.closeIdleConnections(5, TimeUnit.SECONDS);
}
}
} catch (InterruptedException ex) {
// terminate
}
}
public void shutdown() {
shutdown = true;
synchronized (this) {
notifyAll();
}
}
}
}此时觉得异步ni在这种情况下是不二之选，N多并发请求，而且请求时间都耗在了服务端的执行。
异步httpclient，相信以后不会在使用同步client！
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
