org.apache.http.client.HttpClient; HttpClient 4.3超时设置

最新推荐文章于 2023-11-15 19:22:32 发布

weixin_34212762

最新推荐文章于 2023-11-15 19:22:32 发布

阅读量177

点赞数

文章标签：爬虫

可用的code

import org.apache.commons.lang.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;




public static String doPost(String url, String params, String contentType) /*throws IOException */{
        CloseableHttpClient client = HttpClients.createDefault();
        RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(httpSoTimeout).setConnectTimeout(connectionTimeout).build();//设置请求和传输超时时间


        String strResult = null;
        try {
            HttpPost post = new HttpPost(url);
            post.setConfig(requestConfig);
            if (StringUtils.isNotBlank(contentType)) {
                post.setHeader("Content-Type", contentType);
            }
            StringEntity se = new StringEntity(params);
            post.setEntity(se);

            HttpResponse response = client.execute(post);

            if(response.getStatusLine().getStatusCode() == HttpStatus.SC_OK && response.getEntity() != null){
                HttpEntity entity = response.getEntity();
                strResult = EntityUtils.toString(entity, "utf-8");
                EntityUtils.consume(entity);
            }
        } catch (Exception e) {
            log.error("http post error:{} ", e.getMessage());
            return null;
        }
        return strResult;
    }
}

最近用到了HttpClient写爬虫，可能我有新版本强迫症，老是喜欢用新版本的东西(虽说新版本不一定好用),然后就用了HttpClient 4.3。HttpClient这货和Lucene一样，每个版本的API都变化很大，这有点让人头疼。就好比创建一个HttpClient对象吧，每一个版本的都不一样，

3.X是这样的

HttpClient httpClient=new DefaultHttpClient();

4.3是这样的

CloseableHttpClient httpClient = HttpClients.createDefault();

当然，上面这些变化只不过是一些小变化，大家看看API大家就都会了。

我要讲的是超时设置,HttpClient有三种超时设置，最近比较忙，没时间具体归纳总结，以后再补上，我这里就讲一些最简单最易用的超时设置方法。

这是个3.X的超时设置方法

HttpClient client = new HttpClient();
client.setConnectionTimeout(30000);  
client.setTimeout(30000);

HttpClient httpClient= new HttpClient();  
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(5000);

4.X版本的超时设置(4.3后已过时)

HttpClient httpClient=new DefaultHttpClient();
httpClient.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,2000);//连接时间
httpClient.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT,2000);//数据传输时间

4.3版本超时设置

CloseableHttpClient httpClient = HttpClients.createDefault();
HttpGet httpGet=new HttpGet("http://www.baidu.com");//HTTP Get请求(POST雷同)
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000).setConnectTimeout(2000).build();//设置请求和传输超时时间 httpGet.setConfig(requestConfig); httpClient.execute(httpGet);//执行请求

BTW,4.3版本不设置超时的话，一旦服务器没有响应，等待时间N久(>24小时)。