scrapy 爬虫问题记录

一、301 重定向

原因：网站的反扒机制

解决方法：补充header，模拟正常请求

 custom_settings = {
        "ITEM_PIPELINES": {
            'spider.spiders.medlive.medlive_guide_cn.MysqlPipeline': 1
        },
        "DEFAULT_REQUEST_HEADERS": {
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
            "Accept-Encoding": "gzip, deflate",
            "Accept-Language": "zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7",
            "Cache-Control": "max-age=0",
            "Connection": "keep-alive",
            "Cookie": "。。。",
            "Host": "guide.medlive.cn",
            "Referer": "http://www.medlive.cn/",
            "Upgrade-Insecure-Requests": "1",
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36"}
    }