scrapy爬虫获取Ajax请求

本次练习网站为智联招聘网，要先注册登录。 传送门
看到Ajax：登录网站–>F12–>network–>XHR–>F5–>Preview

展开result，可以看到很多的招聘数据，这就是我们需要的。

现在开始写代码
一、创建scrapy项目。
scrapy startproject CrawPosition
二、使用pycharm来编辑代码
open项目之后可以看到scrapy框架已经帮我们创建好一些py文件

CrawPosition/
    scrapy.cfg
    CrawPosition/
        __init__.py
        items.py
        pipelines.py
        settings.py
        spiders/
            __init__.py
            ...

三、在spiders目录下创建一个 crawl.py爬虫文件 ,

• 编写爬虫文件

  crawl.py
  -----------------------------------------------------------
  from scrapy import Spider
  import time
  import scrapy
  from crawposition.items import CrawpositionItem
  import json
  class Craw(Spider):
      name='Craw'
  
      def start_requests(self):
              headers = {
                  'User-Agent': ' Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36',
                  'Connection': 'keep-alive',
                  'Cookie':'你的cookies'
              }
              for i in range(1, 2):
              	#这里注意，不能使用中文，要改变的变量只有&kw。
                  url = 'https://fe-api.zhaopin.com/c/i/sou?start={}&pageSize=90&cityId=489&salary=0,0&workExperience=-1&education=-1&companyType=-1&kw=%E5%A4%A7%E6%95%B0%E6%8D%AE&kt=3&=0'.format(i)
                  #只是单纯的学习，不想给其服务器造成负担。嘿嘿嘿~~
                  time.sleep(5)
                  print(url)
                  yield scrapy.Request(url, self.parse, headers=headers)
  
      def parse(self, response):
          html=response.text
          item=CrawpositionItem()
          if json.loads(html).get('data'):
              data = json.loads(html).get('data')
              results = data.get('results')
              for result in results:
                  item['companyName'] = result['company']['name']
                  item['city'] = result['city']['display']
                  item['extractSkillTag']=result['extractSkillTag']
                  item['recruitCount']=result['recruitCount']
                  item['salary']=result['salary']
                  item['jobName']=result['jobName']
  
                  yield item
  

• 编写items.py文件

  items.py
  ------------------------------------------------------
  import scrapy
  
  class CrawpositionItem(scrapy.Item):
      # define the fields for your item here like:
      # name = scrapy.Field()
      companyName=scrapy.Field()
      city=scrapy.Field()
      extractSkillTag=scrapy.Field()
      recruitCount=scrapy.Field()
      salary=scrapy.Field()
      jobName=scrapy.Field()
  

• 编写pipelines.py文件

class CrawpositionPipeline(object):

    def __init__(self):
        self.file=open('ans.json','a',encoding='utf-8')

    def process_item(self, item, spider):
        lines=dict(item)
        line={"name":lines['jobName'],"detail":{lines['companyName'],lines['city'],lines['recruitCount'],lines['salary']}}
        print(line)
        self.file.write(str(line))
        self.file.write('\n')

        return item

    def spider_closed(self,spider):
        self.file.close()

• 编写settings.py文件

BOT_NAME = 'CrawPosition'

SPIDER_MODULES = ['CrawPosition.spiders']
NEWSPIDER_MODULE = 'CrawPosition.spiders'

#这里的CrawpositionPipeline为pipelines.py文件的class类名。300是优先级
ITEM_PIPELINES={
    'CrawPosition.pipelines.CrawpositionPipeline':300
}
LOG_LEVEL='INFO'
ROBOTSTXT_OBEY = True

运行项目
scrapy crawl Crawl