代理地址存放在quene队列或redis的list中,格式为ip:port
建立一个更换代理的类
public
class
Proxy {
public
static
String
ip
=
"218.189.26.20"
;
public
static
int
port
= 8080;
public
void
change() {
RedisUtil redisUtil = new RedisUtil();
String poll = redisUtil.poll(" proxy");
String[] split = poll.split(":");
ip = split[0];
port=Integer.parseInt(split[1]);
System.
out
.println(
"代理更换成功"
+
ip
+
"-------"
+
port
);
}
}
在httpclient中,调用上述类的静态变量
HttpClientBuilder
builder
= HttpClients. custom();
String
hostname
= Proxy.
ip
;
int
port
= Proxy.
port
;
HttpHost
proxy
=
new
HttpHost(
hostname
,
port
);
CloseableHttpClient
client
=
builder
.setProxy(
proxy
).build();
当页面下载失败时,调用change方法,重新从队列中获取代理,此时还应该将下载失败的url重新放回下载队列
if
(
content
.contains(
"存在异常流量"
)){
logger
.info(
"代理已失效,开始换代理"
);
Proxy
proxy
=
new
Proxy();
proxy
.change();
}