webmagic+chromedriver 获取登录后的cookies后爬取其他网页内容

登录获取cookies:

public class GetCookieByLogin {
	private static Logger logger = LogManager.getLogger(GetCookieByLogin.class);
	//用来存储cookie信息
    private Set<Cookie> cookies = null;
    
 	//使用selenium来模拟用户登录获取Cookie
    public void login(){
        WebDriver driver = new ChromeDriver();
        String loginurl = "登录页面url";
		driver.get(loginurl); //打开登录页面
		//模拟登录开始操作，注意这里By用的是标签的name属性
		 //在用户名输入框中填你的用户名
         driver.findElement(By.name("name")).clear();
         driver.findElement(By.name("name")).sendKeys("用户名");
		//在密码输入框中填你的密码
         driver.findElement(By.name("mima")).clear();
         driver.findElement(By.name("mima")).sendKeys("密码");

        //等待路径跳转，如果不跳转则未登录失败
        //这里的重复4*3000=12秒，页面跳转后才算登录成功，不然获取的cookie大概率是不对的
          String curUrl = driver.getCurrentUrl();//获取当前url
          int tryTimes = 1;
          do{
              logger.info("当前driver的url为："+curUrl);
              logger.info("当前login的url为："+logUrl);
              curUrl = driver.getCurrentUrl();
              tryTimes++;
              try {
                  Thread.sleep(3000);
              } catch (InterruptedException e) {
                  e.printStackTrace();
              }
          }while(loginurl.equals(curUrl) && tryTimes<5);
          if(!loginurl.equals(curUrl)){
              //获取cookie信息
              cookies = driver.manage().getCookies();
          }

          driver.close();
}

process返回的site装载上获取的cookies:

@Override
	public Site getSite() {
		if(cookies!=null && !cookies.isEmpty()){
			//将获取到的cookie信息添加到webmagic中
			for (Cookie cookie : cookies) {
				site.addCookie(cookie.getName().toString(),cookie.getValue().toString());
			}
			return site.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1");
		}
		return site;
	}

downloaderder的download函数中，我从网上下载的downloader代码，打开网页后才去填充cookies，而此时的页面已经被跳转到了登录页面

	@Override
	public Page download(Request request, Task task) {
	...
	//第一次打开目标url
	webDriver.get(request.getUrl());
	...
	//等待打开
	try {
			Thread.sleep(sleepTime);
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
	//获取从process返回的site携带的cookies，填充后第二次打开url
	//这里可以增加判断：获取当前浏览器的url判断是否被跳转登录页面
	Site site = task.getSite();
	if (site.getCookies() != null) {
		for (Map.Entry<String, String> cookieEntry : site.getCookies()
				.entrySet()) {
			Cookie cookie = new Cookie(cookieEntry.getKey(),
					cookieEntry.getValue());
			manage.addCookie(cookie);
		}
		try{
			webDriver.get(request.getUrl());
		}catch(Exception e){
			logger.info("再次打开失败：",e);
		}
	}
	...
	}