今天的目标是从学习OJ爬取后台数据,拿到了一个链接的接口,修改题目编号和操作码就能下载数据,但是需要登录账号的Cookie
所以记录一下用HttpClient 发送网络请求,并下载文件
public static void doPostWithParam(String postUrl, Map<String,String> params,Map<String,String>headers, String saveDir, String fileName) throws Exception{
CloseableHttpClient httpClient = HttpClients.createDefault();
HttpPost post = new HttpPost(postUrl);
List<NameValuePair> list = new ArrayList<>();
/**
* 添加 params
*/
params.forEach((key,value) -> {
list.add(new BasicNameValuePair(key,value));
});
StringEntity entity = new UrlEncodedFormEntity(list,"utf-8");
post.setEntity(entity);
/**
* 添加 Headers
*/
headers.forEach((key,value) -> {
post.addHeader(key,value);
});
CloseableHttpResponse response =httpClient.execute(post);
String string = EntityUtils.toString(response.getEntity());
try {
File file = new File(saveDir);
if(!file.exists()){
file.mkdirs();
}
file = new File(saveDir + fileName);
FileWriter fw = new FileWriter(file);
if(!file.exists()){
file.createNewFile();
}
BufferedWriter bw = new BufferedWriter(fw);
bw.write(string);
bw.close();
fw.close();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
response.close();
httpClient.close();
}
上面的工具传入 url,params 的map,headers的map,保存文件路径,保存文件名
Map<String,String> params = new HashMap<>();
Map<String,String>headers = new HashMap<>();
headers.put("Cookie","...");
try {
doPostWithParam("https://oj.bnuz.edu.cn:8081/problem/" + sources[i][2],params,headers,"E:/JAVA/Java_Work_Idea/Spider/src/data/" + id + "/",id + ".html");
} catch (Exception e) {
e.printStackTrace();
}