scrapy爬知乎返回500

最新推荐文章于 2021-05-12 15:26:30 发布

高翔Sean

最新推荐文章于 2021-05-12 15:26:30 发布

阅读量2.4k

点赞数

分类专栏： python

本文链接：https://blog.csdn.net/seanblog/article/details/78883874

版权

python 专栏收录该内容

25 篇文章

订阅专栏

本文介绍使用Scrapy爬虫框架抓取知乎日报时遇到返回500错误的问题及解决方案。作者通过调整User-Agent设置成功解决了该问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

用scrapy爬知乎日报时，总是返回500

# -*- coding: utf-8 -*-
import scrapy
#import pdfkit
from zhihudaily.items import ZhihudailyItem
class ZhihuSpider(scrapy.Spider):
    name = 'zhihu'
    allowed_domains = ['daily.zhihu.com']
    start_urls = ['http://daily.zhihu.com/']

    def parse(self, response):
        item = ZhihudailyItem()
        sub_url = response.xpath("//div[@class='col-lg-4'][1]//div[@class='wrap'][1]//@href").extract_first()
        fix_url = "http://daily.zhihu.com"
        item["url"] = fix_url+sub_url

        title  = response.xpath("//div[@class='col-lg-4'][1]//div[@class='wrap'][1]//span/text()").extract_first()
        item["filename"] =  "C:/Users/高翔/Desktop/"+title+".pdf"
        print(item["filename"], item['url'])
        yield item
        #pdfkit.from_url(mix_url, filename)

在settings文件中添加USER_AGENT也不好使：

DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
  'Accept-Language': 'zh-CN,zh;q=0.9',
  'Cache-Control':'max-age=0',
  'Connection':'keep-alive',
  'Host':'daily.zhihu.com',
  'USER_AGENT':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3269.3 Safari/537.36'
}

原因在于， settings文件有自己的USER_ANGET，并不是在headers中添加的。

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3269.3 Safari/537.36'

问题解决
基础不扎实...