Scrapy博客园selenium保持登录状态(需要手动登录获取cookie,验证码太难破解了呜呜呜~)
创建好scrapy和sipder后
重写start_requests方法:
def start_requests(self):
# 这是你下载chromedriver.exe的绝对路径
path = "E:\zhuomian\project\python\ArticleSpider\ArticleSpider\spiders\chromedriver.exe"
liu = webdriver.Chrome(path)
liu.get("https://account.cnblogs.com/signin?returnUrl=https:%2F%2Fnews.cnblogs.com%2F")
# 打开登陆界面后让程序睡30秒 这个时间用自己的账号登录上!
time.sleep(30)
# 获取cookie
cookiese = liu.get_cookies()
# 将cookie保存 方便后续使用
pickle.dump(cookiese, open("E:/zhuomian/project/python/ArticleSpider/cookies/boke.cookie", "wb"))
# cookie保存到字典中 用于返回到response
cookie_dict = {}
# 取name和value的值(取重要的就可以)
for cookie in cookiese:
cookie_dict[cookie["name"]] = cookie["value"]
# 返回到response
return [scrapy.Request(url=self.start_urls[0], dont_filter=True, cookies=cookie_dict)]
因为返回到了response所以用parse(self, response)方法来调用一下(自己看就行,不演示了)
登录一次后就已经把cookie获取到了,后续直接读取cookie文件就行,不需要再登录获取了
修改代码:
def start_requests(self):
cookiese = pickle.load(open("E:/zhuomian/project/python/ArticleSpider/cookies/boke.cookie", "rb"))
cookie_dict = {}
for cookie in cookiese:
cookie_dict[cookie["name"]] = cookie["value"]
return [scrapy.Request(url=self.start_urls[0], dont_filter=True, cookies=cookie_dict)]
# 这是你下载chromedriver.exe的绝对路径
# path = "E:\zhuomian\project\python\ArticleSpider\ArticleSpider\spiders\chromedriver.exe"
# liu = webdriver.Chrome(path)
# liu.get("https://account.cnblogs.com/signin?returnUrl=https:%2F%2Fnews.cnblogs.com%2F")
# 打开登陆界面后让程序睡30秒 这个时间用自己的账号登录上!
# time.sleep(30)
# 获取cookie
# cookiese = liu.get_cookies()
# 将cookie保存 方便后续使用
# pickle.dump(cookiese, open("E:/zhuomian/project/python/ArticleSpider/cookies/boke.cookie", "wb"))
# cookie保存到字典中 用于返回到response
# cookie_dict = {}
# 取name和value的值(取重要的就可以)
# for cookie in cookiese:
# cookie_dict[cookie["name"]] = cookie["value"]
# 返回到response
# return [scrapy.Request(url=self.start_urls[0], dont_filter=True, cookies=cookie_dict)]