scrapy 去除html,用Scrapy Splash 来抓取渲染后的html页面

很多复杂的网页都是用javascript来对网页进行填充，这样用request的body和在浏览器中看到的不一样啊。这个时候splash就可以使用了，它是提供一个轻量级的api，传给它网址，它返回网页内容。这样就OK了

1.安装splash

要先安装dockerdocker pull registry.docker-cn.com/scrapinghub/splash #从docker镜像中拉取splash实例

docker run -p 8050:8050 scrapinghub/splash #启动splash实例

现在docker 可以用国内的源了，所以要加上registry.docker-cn.com来加快速度

还需要安装个scrapy-splashsudo pip3 install scrapy-splash

2. setting 设置

SPLASH_URL = 'http://localhost:8050'

DOWNLOADER_MIDDLEWARES = {

'scrapy_splash.SplashCookiesMiddleware': 723,

'scrapy_splash.SplashMiddleware': 725,

'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,

}

SPIDER_MIDDLEWARES = {

'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,

}

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

3. spider中使用

import scrapy

import logging

import re

import datetime

from tutorial import settings

from scrapy_splash import SplashRequest

class JanDanSpider(scrapy.Spider):

name = "jandan"

def start_requests(self):

#start_url = 'http://jandan.net/ooxx'

start_url = 'http://www.baidu.com'

headers = {

'Connection' : 'keep - alive',

'User-Agent' : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36'

}

yield SplashRequest(url=start_url, callback = self.parse, args={'wait': 1.0})

def parse(self, response):

# --------------------------------body image url--------------------------------------------

image_urls = response.css('img::attr("src")').extract()

new_image_urls = []

for i in range(len(image_urls)):

new_image_urls.append('http://' + image_urls[i][2:])

pass

主要这里用了SplashRequest，加了一个等待时间，callback回来就跟默认的一样了。进行爬虫的其他处理。。。

Post Views: 23