为爬虫设置User-Agent

 

java里面设置User-Agent代码如下

HttpURLConnection uc = (HttpURLConnection) baseURL.openConnection();
uc.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 3.0.04506)");

 

这个User-Agent是我用ieHTTPheaders的在浏览器里面看到的结果，挑了几个改了下

写道

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; CIBA)