HttpClient使用代理

1、为什么使用代理

当爬取网页的时候,有的目标站点有反爬虫机制,对于频繁访问站点以及规则性访问站点的行为,会采用屏蔽IP的措施,这时候代理IP就派上用场了。

2、代理的分类

(一)透明代理(Transparent Proxy)-透明代理虽然可以直接隐藏你的IP地址,但是还是从HTTP_X_FORWARD_FOR 来查到你是谁

REMOTE_ADDR= Proxy IP
HTTP_VIA = Proxy IP
HTTP_X_FORWORAD_FOR= YOUR IP

(二)匿名代理(Anonymous Proxy)-匿名代理比透明代理进步一点,别人只能知道你用了代理,无法知道你是谁

REMOTE_ADDR= proxy Ip
HTTP_VIA = proxy IP
HTTP_X_FORWARD_FOR = proxy_IP

(三)混淆代理(Distorting Proxies)-与匿名代理相同,如果使用了混淆代理,别人还是能知道你在用代理,但是会得到一个假的IP地址,伪装的更逼真

REMOTE_ADDR=PROXY_IP
HTTP_VIA =PROXY IP
HTTP_X_FOREARD_FOR=Random IP ADDRESS

(四)高匿代理(Elite Proxy 或High Anonymity Proxy)-可以看出,高匿代理让别人无法发现你是在用代理,是爬虫最好的选择

REMOTE_ADDR=PROXY_IP
HTTP_VIA = not determined
HTTP_X_FORWARD_FOR= not determined

3、 代理IP的获取

@Test
public void testHttpProxy() throws  Exception{
    HttpClient httpClient = HttpClients.createDefault();
    HttpGet httpGet = new HttpGet("http://www.baidu.com");
    //使用代理服务器
    HttpHost httpHost = new HttpHost("220.194.55.160",3128);
    RequestConfig config = RequestConfig.custom().setProxy(httpHost).build();
    httpGet.setConfig(config);
    CloseableHttpResponse response = (CloseableHttpResponse) httpClient.execute(httpGet);
    HttpEntity entity = response.getEntity();
    //输出网页内容
    System.out.println("网页内容:");
    System.out.println(EntityUtils.toString(entity,"utf-8"));
    response.close();
}

4、HttpClient代理配置
HttpClient支持复杂的路由方案和代理链,同样也支持直接或者只通过一跳的连接。

(一)使用代理服务器最简单的方式,执行一个默认的默认的代理。

HttpHost proxy = new HttpHost("someproxy", 8080);  
DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);  
CloseableHttpClient httpclient = HttpClients.custom()  
        .setRoutePlanner(routePlanner)  
        .build();

(二) HttpClient使用jre代理服务器

SystemDefaultRoutePlanner routePlanner = new SystemDefaultRoutePlanner(  
        ProxySelector.getDefault());  
CloseableHttpClient httpclient = HttpClients.custom()  
        .setRoutePlanner(routePlanner)  
        .build();   

手动配置RoutePlanner,这样就可以完全控制Http路由的过程

HttpRoutePlanner routePlanner = new HttpRoutePlanner() {    
    public HttpRoute determineRoute(  
            HttpHost target,  
            HttpRequest request,  
            HttpContext context) throws HttpException {  
        return new HttpRoute(target, null,  new HttpHost("someproxy", 8080),  
                "https".equalsIgnoreCase(target.getSchemeName()));  
    }   
};  
CloseableHttpClient httpclient = HttpClients.custom()  
        .setRoutePlanner(routePlanner)  
        .build();  
    }  
}      

例子

package com.book.common.utils;

import com.alibaba.fastjson.JSONObject;
import com.book.common.constant.StaticConstant;
import com.book.common.result.Failed;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.HttpHost;
import org.apache.http.NameValuePair;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.*;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.entity.ByteArrayEntity;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.ssl.SSLContexts;
import org.apache.http.util.EntityUtils;
import org.springframework.util.CollectionUtils;

import javax.net.ssl.SSLContext;
import java.io.*;
import java.net.*;
import java.nio.charset.StandardCharsets;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Random;

/**
 * Description:
 * HTTP调用;https路径会转化为http,所以需要能使用http调用访问的路径才可以使用
 * 用法:HttpUtils.doGet(HttpUtils.create().addHostPort("http://suyun.nevtsp.com/").addProxy("116.30.240.246:80").build());
 *
 * @Author: leo.xiong
 * @CreateDate: 2021/11/30 9:43
 * @Email: leo.xiong@suyun360.com
 * @Since:
 */
@Slf4j
public final class HttpUtils {

    private static final int DOWN_LOAD_READ_BYTE_BUFFER = 1024;

    private static final int SO_TIMEOUT_MS = 8000;

    private static final int CONNECTION_TIMEOUT_MS = 3000;

    private static final int SUCCESS = 200;

    private static final int HTTP_PORT = 80;

    private static final int HTTPS_PORT = 443;

    private static final Random RANDOM = new Random();

    /**
     * 创建参数对象信息
     *
     * @return
     */
    public static HttpClientParam create() {
        return new HttpClientParam();
    }

    /**
     * get
     *
     * @param httpClientParam
     * @return
     */
    public static String doGet(final HttpClientParam httpClientParam) {
        HttpGet request = new HttpGet(httpClientParam.getUrl());
        buildHeaderProxy(request, httpClientParam);
        return execute(httpClientParam.getCloseableHttpClient(), request);
    }

    /**
     * Post From
     *
     * @param httpClientParam
     * @return
     */
    public static String doPost(final HttpClientParam httpClientParam) {
        HttpPost request = new HttpPost(httpClientParam.getUrl());
        buildHeaderProxy(request, httpClientParam);
        if (!CollectionUtils.isEmpty(httpClientParam.getBodyMap())) {
            List<NameValuePair> nameValuePairList = Lists.newArrayList();
            for (String key : httpClientParam.getBodyMap().keySet()) {
                nameValuePairList.add(new BasicNameValuePair(key, httpClientParam.getBodyMap().get(key)));
            }
            UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(nameValuePairList, StandardCharsets.UTF_8);
            formEntity.setContentType("application/x-www-form-urlencoded; charset=UTF-8");
            request.setEntity(formEntity);
        } else if (!StringUtils.isEmpty(httpClientParam.getBodyStr())) {
            request.setEntity(new StringEntity(httpClientParam.getBodyStr(), StandardCharsets.UTF_8));
        } else if (httpClientParam.getBody() != null) {
            request.setEntity(new ByteArrayEntity(httpClientParam.getBody()));
        }
        return execute(httpClientParam.getCloseableHttpClient(), request);
    }

    /**
     * Put Stream
     *
     * @param httpClientParam
     * @return
     */
    public static String doPut(final HttpClientParam httpClientParam) {
        HttpPut request = new HttpPut(httpClientParam.getUrl());
        buildHeaderProxy(request, httpClientParam);
        if (StringUtils.isNotEmpty(httpClientParam.getBodyStr())) {
            request.setEntity(new StringEntity(httpClientParam.getBodyStr(), StandardCharsets.UTF_8));
        } else if (httpClientParam.getBody() != null) {
            request.setEntity(new ByteArrayEntity(httpClientParam.getBody()));
        }
        return execute(httpClientParam.getCloseableHttpClient(), request);
    }

    /**
     * Delete Stream
     *
     * @param httpClientParam
     * @return
     */
    public static String doDelete(final HttpClientParam httpClientParam) {
        HttpDelete request = new HttpDelete(httpClientParam.getUrl());
        buildHeaderProxy(request, httpClientParam);
        return execute(httpClientParam.getCloseableHttpClient(), request);
    }

    /**
     * 执行信息
     *
     * @param httpClient
     * @param request
     * @return
     */
    private static String execute(CloseableHttpClient httpClient, HttpUriRequest request) {
        CloseableHttpResponse response = null;
        try {
            response = httpClient.execute(request);
            if (response == null) {
                return JSONObject.toJSONString(new Failed("100000", "返回信息为空"));
            } else {
                if (response.getStatusLine().getStatusCode() == SUCCESS) {
                    return EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
                } else {
                    return JSONObject.toJSONString(new Failed(response.getStatusLine().getStatusCode() + "", "返回信息失败"));
                }
            }
        } catch (IOException e) {
            log.warn("执行失败", e);
            return JSONObject.toJSONString(new Failed("100001", e.getMessage()));
        } finally {
            try {
                if (response != null) {
                    response.close();
                }
                if (httpClient != null) {
                    httpClient.close();
                }
            } catch (IOException e) {
            }
        }
    }

    /**
     * 根据URL下载文件信息,保存在指定的目录下面
     *
     * @param urlStr
     * @param fileName
     * @param savePath
     */
    public static void downLoadFromUrl(String urlStr, String fileName, String savePath) {
        downLoadFromUrl(urlStr, fileName, savePath, null, null);
    }

    /**
     * 从网络Url中下载文件
     * http://suyun-nevtsp-foreverfile.oss-cn-shenzhen.aliyuncs.com/vehicle_admin/dbc_excel/2020/11/16/L16_V1.1_20201116095257890.xlsx
     *
     * @param urlStr   下载URL
     * @param fileName 下载文件名
     * @param savePath 保存的路径
     * @param tokeName Token 名称
     * @param token    需要的TOKEN新
     * @throws IOException
     */
    public static void downLoadFromUrl(String urlStr, String fileName, String savePath, String tokeName, String token) {
        FileOutputStream fos = null;
        InputStream is = null;
        try {
            URL url = new URL(urlStr);
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            //设置超时间为3秒
            conn.setConnectTimeout(CONNECTION_TIMEOUT_MS);
            //防止屏蔽程序抓取而返回403错误
            conn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");
            if (StringUtils.isNotEmpty(token)) {
                conn.setRequestProperty(tokeName, token);
            }
            //得到输入流
            is = conn.getInputStream();
            //获取自己数组
            byte[] getData = readInputStream(is);
            //文件保存位置
            File saveDir = new File(savePath);
            if (!saveDir.exists()) {
                saveDir.mkdir();
            }
            File file = new File(saveDir + File.separator + fileName);
            fos = new FileOutputStream(file);
            fos.write(getData);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fos != null) {
                    fos.close();
                }
                if (is != null) {
                    is.close();
                }
            } catch (IOException e) {

            }
        }
    }


    /**
     * 添加头部信息,判断是否使用代理
     *
     * @param httpRequestBase
     * @param httpClientParam
     */
    private static void buildHeaderProxy(HttpRequestBase httpRequestBase, final HttpClientParam httpClientParam) {
        addHeader(httpRequestBase, httpClientParam.getHeaderMap());
        if (StringUtils.isEmpty(httpClientParam.getProxy()) || !httpClientParam.getProxy().contains(StaticConstant.Common.COLON)) {
            httpRequestBase.setConfig(RequestConfig.custom()
                    .setSocketTimeout(SO_TIMEOUT_MS)
                    .setConnectTimeout(CONNECTION_TIMEOUT_MS).build());
            return;
        }
        Proxy proxy = null;
        if (httpClientParam.getProxy().startsWith(StaticConstant.Common.HTTP)) {
            proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(httpClientParam.getProxy().replace(StaticConstant.Common.HTTP, StaticConstant.Common.EMPTY_STR), HTTP_PORT));
        } else if (httpClientParam.getProxy().startsWith(StaticConstant.Common.HTTPS)) {
            proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(httpClientParam.getProxy().replace(StaticConstant.Common.HTTPS, StaticConstant.Common.EMPTY_STR), HTTPS_PORT));
        } else {
            String[] ipPort = httpClientParam.getProxy().split(StaticConstant.Common.COLON);
            proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ipPort[0], Integer.parseInt(ipPort[1])));
        }
        InetSocketAddress socketAddress = (InetSocketAddress) proxy.address();
        String ip = socketAddress.getAddress().getHostAddress();
        Integer port = socketAddress.getPort();
        if (StringUtils.isNotEmpty(ip) && port != null) {
            HttpHost httpHost = new HttpHost(ip, port);
            httpRequestBase.setConfig(RequestConfig.custom().setProxy(httpHost)
                    .setSocketTimeout(SO_TIMEOUT_MS)
                    .setConnectTimeout(CONNECTION_TIMEOUT_MS).build());
        } else {
            httpRequestBase.setConfig(RequestConfig.custom()
                    .setSocketTimeout(SO_TIMEOUT_MS)
                    .setConnectTimeout(CONNECTION_TIMEOUT_MS).build());
        }
        if (httpRequestBase.getHeaders("X-Forwarded-For") == null || httpRequestBase.getHeaders("X-Forwarded-For").length == 0) {
            httpRequestBase.addHeader("X-Forwarded-For", httpClientParam.getForwardIp());
        }
    }

    private static void addHeader(HttpRequestBase httpRequestBase, Map<String, String> headerMap) {
        if (!CollectionUtils.isEmpty(headerMap)) {
            for (Map.Entry<String, String> header : headerMap.entrySet()) {
                httpRequestBase.addHeader(header.getKey(), header.getValue());
            }
        }
        if (headerMap.containsKey("User-Agent")) {
            return;
        }
        httpRequestBase.addHeader("User-Agent", USER_AGENT_ALL[RANDOM.nextInt(USER_AGENT_ALL.length - 1)]);
    }

    /**
     * 从输入流中获取字节数组
     *
     * @param inputStream
     * @return
     * @throws IOException
     */
    private static byte[] readInputStream(InputStream inputStream) throws IOException {
        byte[] buffer = new byte[DOWN_LOAD_READ_BYTE_BUFFER];
        int len = 0;
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        while ((len = inputStream.read(buffer)) != -1) {
            bos.write(buffer, 0, len);
        }
        bos.close();
        return bos.toByteArray();
    }

    /**
     * 访问参数
     */
    @Getter
    public static class HttpClientParam {
        /**
         * 拼装之后的URL信息
         */
        private String url;
        /**
         * 访问客户端
         */
        private CloseableHttpClient closeableHttpClient;
        /**
         * 访问协议IP和端口例如:http://10.0.0.5:8080 不能为空
         */
        private String hostPort;
        /**
         * 访问的方法 例如: /a/query 可以为空
         */
        private String path;
        /**
         * URL带的参数信息 key 为name value为值 可以为空 ,host+path+url会拼接成真正的URL
         */
        private Map<String, String> urlParamMap = Maps.newHashMap();
        /**
         * 头部参数 key 为name value为值 可以为空
         */
        private Map<String, String> headerMap = Maps.newHashMap();
        /**
         * FormData 参数信息
         */
        private Map<String, String> bodyMap = Maps.newHashMap();
        /**
         * body字符串
         */
        private String bodyStr;
        /**
         * body byte数组 bodyMap,bodyStr,body三个只能使用一个,优先级 bodyMap>bodyStr>body
         */
        private byte[] body;
        /**
         * 是否使用代理,如果为空,表示不代理 10.0.0.1:8080
         */
        private String proxy;

        private String forwardIp;

        private HttpClientParam() {
        }

        /**
         * 访问协议IP和端口例如:http://10.0.0.5:8080 不能为空
         */
        public HttpClientParam addHostPort(String hostPort) {
            this.hostPort = hostPort;
            return this;
        }

        /**
         * 访问的方法映射 例如: /a/query 可以为空
         */
        public HttpClientParam addPath(String path) {
            this.path = path;
            return this;
        }

        /**
         * URL带的参数信息 key 为name value为值 可以为空 ,host+path+url会拼接成真正的URL
         *
         * @param key
         * @param value
         * @return
         */
        public HttpClientParam addQuery(String key, String value) {
            this.urlParamMap.put(key, value);
            return this;
        }

        /**
         * URL带的参数信息 key 为name value为值 可以为空 ,host+path+url会拼接成真正的URL
         *
         * @param queryMap
         * @return
         */
        public HttpClientParam addQueryAll(Map<String, String> queryMap) {
            this.urlParamMap.putAll(queryMap);
            return this;
        }

        /**
         * 头部参数 key 为name value为值 可以为空
         *
         * @param key
         * @param value
         * @return
         */
        public HttpClientParam addHeader(String key, String value) {
            this.headerMap.put(key, value);
            return this;
        }

        /**
         * 头部参数 key 为name value为值 可以为空
         *
         * @param headerMap
         * @return
         */
        public HttpClientParam addHeaderAll(Map<String, String> headerMap) {
            this.headerMap.putAll(headerMap);
            return this;
        }

        /**
         * FormData 参数信息 bodyMap,bodyStr,body三个只能使用一个,优先级 bodyMap>bodyStr>body
         *
         * @param key
         * @param value
         */
        public void addBodyMap(String key, String value) {
            this.bodyMap.put(key, value);
        }

        /**
         * FormData 参数信息 bodyMap,bodyStr,body三个只能使用一个,优先级 bodyMap>bodyStr>body
         *
         * @param bodyMap
         */
        public void addBodyMapAll(Map<String, String> bodyMap) {
            this.bodyMap.putAll(bodyMap);
        }

        /**
         * body字符串 bodyMap,bodyStr,body三个只能使用一个,优先级 bodyMap>bodyStr>body
         *
         * @param bodyStr
         * @return
         */
        public HttpClientParam addBodyStr(String bodyStr) {
            this.bodyStr = bodyStr;
            return this;
        }

        /**
         * byte数组 bodyMap,bodyStr,body三个只能使用一个,优先级 bodyMap>bodyStr>body
         *
         * @param body
         * @return
         */
        public HttpClientParam addBody(byte[] body) {
            this.body = body;
            return this;
        }

        /**
         * 是否使用代理,如果为空,表示不代理 10.0.0.1:8080
         *
         * @param proxy
         * @return
         */
        public HttpClientParam addProxy(String proxy) {
            this.proxy = proxy;
            return this;
        }

        /**
         * 创建对象信息
         */
        public HttpClientParam build() {
            boolean isNoValid = StringUtils.isEmpty(this.hostPort) || (!this.hostPort.startsWith(StaticConstant.Common.HTTP) && !this.hostPort.startsWith(StaticConstant.Common.HTTPS));
            if (isNoValid) {
                log.warn("路径信息错误 hostPort:{}", this.hostPort);
                return null;
            }
            if (this.hostPort.endsWith(StaticConstant.Common.SLASH)) {
                this.hostPort = this.hostPort.substring(0, this.hostPort.lastIndexOf(StaticConstant.Common.COLON));
            }
            if (StringUtils.isNotEmpty(this.path) && !this.path.startsWith(StaticConstant.Common.SLASH)) {
                this.path = StaticConstant.Common.SLASH + this.path;
            }
            try {
                if (this.proxy != null) {
                    String tempProxy = null;
                    this.forwardIp = this.proxy.substring(0, this.proxy.lastIndexOf(StaticConstant.Common.COLON));
                    if (this.hostPort.startsWith(StaticConstant.Common.HTTP)) {
                        tempProxy = this.hostPort.replace(StaticConstant.Common.HTTP, StaticConstant.Common.EMPTY_STR);
                        this.hostPort = StaticConstant.Common.HTTP + this.proxy;
                    } else {
                        tempProxy = this.hostPort.replace(StaticConstant.Common.HTTPS, StaticConstant.Common.EMPTY_STR);
                        this.hostPort = StaticConstant.Common.HTTPS + this.proxy;
                    }
                    this.proxy = tempProxy;
                }
                this.url = buildUrl(this.hostPort, this.path, this.urlParamMap);
                this.closeableHttpClient = wrapClient(this.hostPort);
            } catch (Exception e) {
                log.warn("创建访问对象错误", e);
                return null;
            }
            return this;
        }

        /**
         * 拼接URL
         *
         * @param host
         * @param path
         * @param queryMap
         * @return
         * @throws UnsupportedEncodingException
         */
        private String buildUrl(String host, String path, Map<String, String> queryMap) {
            StringBuilder sbUrl = new StringBuilder();
            sbUrl.append(host);
            if (!StringUtils.isBlank(path)) {
                sbUrl.append(path);
            }
            if (CollectionUtils.isEmpty(queryMap)) {
                return sbUrl.toString();
            }
            StringBuilder sbQuery = new StringBuilder();
            for (Map.Entry<String, String> query : queryMap.entrySet()) {
                if (sbQuery.length() > 0) {
                    sbQuery.append(StaticConstant.Common.AND);
                }
                if (StringUtils.isBlank(query.getKey()) && !StringUtils.isBlank(query.getValue())) {
                    sbQuery.append(query.getValue());
                }
                if (!StringUtils.isBlank(query.getKey())) {
                    sbQuery.append(query.getKey());
                    if (!StringUtils.isBlank(query.getValue())) {
                        sbQuery.append("=");
                        try {
                            sbQuery.append(URLEncoder.encode(query.getValue(), StandardCharsets.UTF_8.name()));
                        } catch (UnsupportedEncodingException e) {
                            log.warn("编码失败 value: {}", query.getValue());
                        }
                    }
                }
            }
            if (sbQuery.length() > 0) {
                sbUrl.append(StaticConstant.Common.UN_KNOW).append(sbQuery);
            }
            return sbUrl.toString();
        }

        private CloseableHttpClient wrapClient(String host) {
            CloseableHttpClient httpClient = HttpClients.createDefault();
            if (host.startsWith(StaticConstant.Common.HTTPS)) {
                SSLContext sslContext = null;
                try {
                    sslContext = SSLContexts.custom().loadTrustMaterial(null, (x509Certificates, s) -> {
                        return true;
                    }).build();
                } catch (NoSuchAlgorithmException | KeyManagementException | KeyStoreException e) {
                    e.printStackTrace();
                }
                //创建httpClient
                return HttpClients.custom().setSSLContext(sslContext).
                        setSSLHostnameVerifier(new NoopHostnameVerifier()).build();
            }
            return httpClient;
        }
    }

    private static final String[] USER_AGENT_ALL = {
            "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36",
            "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36",
            "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36",
            "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:21.0.0) Gecko/20121011 Firefox/21.0.0",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
            "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
            "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
            "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
            "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
            "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
            "Mozilla/5.0 (Macintosh; U; Mac OS X Mach-O; en-US; rv:2.0a) Gecko/20040614 Firefox/3.0.0 ",
            "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Firefox/3.0.3",
            "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5",
            "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.14) Gecko/20110218 AlexaToolbar/alxf-2.0 Firefox/3.6.14",
            "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10",
            "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/533.17.8 (KHTML, like Gecko) Version/5.0.1 Safari/533.17.8",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5",
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0",
            "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
            "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
            "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
    };
}

private static void proxy() {
        String result = HttpUtils.doGet(HttpUtils.create()
                //访问的服务器IP
                .addHostPort("http://116.30.240.246:8080")
                //访问的方法
                .addPath("/a/login")
                //代理IP和端口

效果:
在这里插入图片描述
在这里插入图片描述
使用代理
在这里插入图片描述
不用代理
在这里插入图片描述
StaticConstant.Common.COLON=“:”;
StaticConstant.Common.HTTP=“http”;
StaticConstant.Common.HTTPS=“https”;
StaticConstant.Common.EMPTY_STR=“”;
StaticConstant.Common.SLASH=“/”;
StaticConstant.Common.AND=“&”;
StaticConstant.Common.UN_KNOW=“?”;
Filed类里面有code errorMsg2属性;

Java使用HttpClient进行代理访问百度,你可以按照以下步骤进行: 1. 创建一个HttpClient实例。 2. 配置HttpClient使用代理服务器。 3. 创建HttpGet或者HttpPost请求实例,并指定要访问的百度URL。 4. 执行请求,并获取HttpResponse。 5. 从HttpResponse中获取响应内容。 以下是一个使用Java HttpClient通过代理访问百度的示例代码: ```java import org.apache.http.HttpEntity; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.util.EntityUtils; public class HttpProxyExample { public static void main(String[] args) throws Exception { // 创建HttpClient实例 CloseableHttpClient httpClient = HttpClients.custom().build(); // 创建HttpGet请求实例,并指定百度的URL HttpGet httpGet = new HttpGet("http://www.baidu.com"); // 配置代理服务器,这里假设代理服务器地址为127.0.0.1,端口为8888 httpClient.getParams().setParameter("http.proxyHost", "127.0.0.1"); httpClient.getParams().setParameter("http.proxyPort", 8888); // 可以选择设置代理的用户名和密码(如果需要的话) // httpClient.getParams().setParameter("http.proxyUser", "username"); // httpClient.getParams().setParameter("http.proxyPassword", "password"); // 执行请求 CloseableHttpResponse response = httpClient.execute(httpGet); // 获取响应实体内容 HttpEntity entity = response.getEntity(); // 打印响应内容到控制台 System.out.println(EntityUtils.toString(entity)); // 关闭连接 EntityUtils.consume(entity); httpClient.close(); } } ``` 注意:在实际应用中,代理服务器地址、端口以及可能的用户名和密码应该根据你所使用代理服务器来设置。
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值