Python爬虫:splash+requests简单示例

版权声明:本文为博主原创文章,欢迎转载,请注明出处 https://blog.csdn.net/mouday/article/details/82843401

说明:
render是get方式
execute是post方式

render

import requests

def splash_render(url):
    splash_url = "http://localhost:8050/render.html"
    
    args = {
        "url": url,
        "timeout": 5,
        "image": 0
    }

    response = requests.get(splash_url, params=args)
    return response.text


if __name__ == '__main__':

    url = "http://quotes.toscrape.com/js/"

    html = splash_render(url)

args参数说明:
url: 需要渲染的页面地址
timeout: 超时时间
proxy:代理
wait:等待渲染时间
images: 是否下载,默认1(下载)
js_source: 渲染页面前执行的js代码

execute


import json
import requests

def splash_execute(url):
    splash_url = "http://localhost:8050/execute"

    script = """
    function main(splash)
        local url="{url}"
        splash:set_user_agent("Mozilla/5.0  Chrome/69.0.3497.100 Safari/537.36")
        splash:go(url)
        splash:wait(2)
        splash:go(url)
        return {
            html = splash:html()
        }
    end
    """
    script = script.replace("{url}", url)
    data = json.dumps({
        "timeout": 5,
        "lua_source": script
    })

    headers = {
        "User-Agent": "Mozilla/5.0 Chrome/69.0.3497.100 Safari/537.36",
        "content-type": "application/json"
    }

    response = requests.post(splash_url, headers=headers, data=data)

    return response.json().get("html")
    

if __name__ == '__main__':
    url = "http://quotes.toscrape.com/js/"

    html = splash_execute(url)

参数说明:
timeout 超时
lua_source lua脚本
proxy 代理

模拟登录

以下是lua脚本
splash提供的select选择器,使用方法和jQuery类似

function main(splash, args)
	-- jquery加载比较慢
	splash:autoload("https://code.jquery.com/jquery-3.3.1.min.js")
	splash:set_viewport_size(1366, 768)
	splash:set_user_agent("Mozilla/5.0  Chrome/69.0.3497.100 Safari/537.36")
	
	-- 从首页点击登录按钮
	splash:go(splash.args.url)
	splash:wait(3)
	splash:runjs("$('#login').click()")
	splash:wait(2)
	
	-- 登录页输入账号密码,并提交
	splash:select("#username"):send_text("username")
	splash:select("#password"):send_text("password")
	splash:wait(5)

	-- 可以使用splash自带的鼠标点击,并指定点击位置
	local button = splash:select("#button")
    local bounds = button:bounds()
    button:mouse_click{x=bounds.width/3, y=bounds.height/3}
	splash:wait(2)

	-- 返回
	return {
		html=splash:html(),
		png = splash:png(),
		cookie=splash:get_cookies()
	}
end

参考:
splash文档:https://splash.readthedocs.io/en/stable/scripting-ref.html

展开阅读全文

没有更多推荐了,返回首页