scrapy异步爬取猫眼实时票房

目前为止，比较厉害的网页都在采用ajax异步加载的形式在网页中加载数据。也就是说，我们爬取html页面后，可能根本找不到在浏览器中看到的内容。所以我们要通过使用ajax异步加载取拿到自己想要的数据。

我这里以猫眼实时票房

最为事例

先F12打开调试台，在network中选取xhr，我们只查看xhr请求，双击左侧加载的ajax数据，这个就是我们要找的。
直接上教程

创建爬虫项目

scrapy startproject maoyan

创建爬虫文件

cd maoyan
scrapy genspider mao www.maoyan

items.py

  class MaoyanItem(scrapy.Item):
  
  movie_id = scrapy.Field()
    #电影名    
    movie_name = scrapy.Field()
    #票房占比   
     movie_account = scrapy.Field()
    #上映天数    
    movie_day = scrapy.Field()
    #排片场次   
     movie_num = scrapy.Field()

mao.py

 import scrapy
from maoyan.items import MaoyanItem
import json
class MaoSpider(scrapy.Spider):
    name = 'mao'    
    #allowed_domains = ['www.baodu']    
    start_urls = ['https://piaofang.maoyan.com/dashboard-ajax/movie?movieId=1302235&orderType=0&uuid=54ab1980-c361-44d6-bd01-2842a8067c59']
    def parse(self, response):
        id = 1        
        html = response.text
        item = MaoyanItem()
        #判断json数据中是否存在movieList
        if json.loads(html).get("movieList"):
            movieList = json.loads(html).get("movieList")
            list = movieList.get("list")
            #判断list是否为空
            if list is None or list == []:
                print("我是空的!")
                for li in list:
                    #id值
                    item["movie_id"] = id
                    id += 1     
                    #电影名称           
                    item["movie_name"] = li["movieInfo"]["movieName"]
                    #票房占比
                    item["movie_account"] = li["boxRate"]
                    #上映天数
                    item["movie_day"] = li["movieInfo"]["releaseInfo"]
                    #票房人数
                    item["movie_num"] = li["showCount"]
                    yield item

pipelines.py

 - class MaoyanPipeline:
def process_item(self, item, spider):
    print("%s. 电影名称:%s ,票房占比:%s ,上映天数:%s ,排片场次:%s \n" % (item["movie_id"],item["movie_name"],item["movie_account"],item["movie_day"],item["movie_num"]))
    return item

运行项目

scrapy crawl mao