MarkAdc的博客

自己笔记本电脑在公司跑爬虫，然后下班了，我把爬虫先暂停，然后把电脑带回家。回家后我再接着跑爬虫，它不香吗

项目目录在middlewares.py中import randomclass RandomUserAgent: def get_ua(self): first_num = random.randint(55, 62) third_num = random.randint(0, 3200) fourth_num = random.randint(0, 150) os_type = [ '(Window.

在setting文件中COOKIES_ENABLED = FalseFalse– 使用的是setting文件里，DEFAULT_REQUEST_HEADERS中的cookie值COOKIES_ENABLED = TrueTrue– 就可以在爬虫文件中使用自定义的cookie，传入的cookie必须是字典类型# COOKIES_ENABLED = TrueCOOKIES_ENABLED被注释– 则是不使用cookie...

代码from twisted.internet.error import TCPTimedOutError, TimeoutErrorclass ProcessException: def process_exception(self, request, exception, spider): if isinstance(exception, TCPTimedOutError): print(f"出异常了(tcp超时) --> {exception

代码在爬虫文件中@classmethod def from_crawler(cls, crawler, *args, **kwargs): spider = super(T1Spider, cls).from_crawler(crawler, *args, **kwargs) spider.id = f"t1-{datetime.datetime.now()}" crawler.signals.connect(spider.item_scraped

代码yield Request(url, meta={ 'dont_redirect': True, 'handle_httpstatus_list': [302] }, callback=self.parse)