目前为止,比较厉害的网页都在采用ajax异步加载的形式在网页中加载数据。也就是说,我们爬取html页面后,可能根本找不到在浏览器中看到的内容。所以我们要通过使用ajax异步加载取拿到自己想要的数据。
我这里以猫眼实时票房
最为事例
先F12打开调试台,在network中选取xhr,我们只查看xhr请求,双击左侧加载的ajax数据,这个就是我们要找的。
直接上教程
创建爬虫项目
scrapy startproject maoyan
创建爬虫文件
cd maoyan
scrapy genspider mao www.maoyan
items.py
class MaoyanItem(scrapy.Item):
movie_id = scrapy.Field()
#电影名
movie_name = scrapy.Field()
#票房占比
movie_account = scrapy.Field()
#上映天数
movie_day = scrapy.Field()
#排片场次
movie_num = scrapy.Field()
mao.py
import scrapy
from maoyan.items import MaoyanItem
import json
class MaoSpider(scrapy.Spider):
name = 'mao'
#allowed_domains = ['www.baodu']
start_urls = ['https://piaofang.maoyan.com/dashboard-ajax/movie?movieId=1302235&orderType=0&uuid=54ab1980-c361-44d6-bd01-2842a8067c59']
def parse(self, response):
id = 1
html = response.text
item = MaoyanItem()
#判断json数据中是否存在movieList
if json.loads(html).get("movieList"):
movieList = json.loads(html).get("movieList")
list = movieList.get("list")
#判断list是否为空
if list is None or list == []:
print("我是空的!")
for li in list:
#id值
item["movie_id"] = id
id += 1
#电影名称
item["movie_name"] = li["movieInfo"]["movieName"]
#票房占比
item["movie_account"] = li["boxRate"]
#上映天数
item["movie_day"] = li["movieInfo"]["releaseInfo"]
#票房人数
item["movie_num"] = li["showCount"]
yield item
pipelines.py
- class MaoyanPipeline:
def process_item(self, item, spider):
print("%s. 电影名称:%s ,票房占比:%s ,上映天数:%s ,排片场次:%s \n" % (item["movie_id"],item["movie_name"],item["movie_account"],item["movie_day"],item["movie_num"]))
return item
运行项目
scrapy crawl mao