爬虫之mitmproxy

mitmproxy是一个代理工具，我对它主要用在：逆向时替换请求中的参数。

一、下载安装

直接使用：pip install mitmproxy

可能会报错，好像是需要安装Visual C++ Build Tools for Visual Studio 2015 ，但是我没报错。

Visual C++安装地址：https://my.visualstudio.com/Downloads?q=Visual%20Studio%202015%20update%203

安装成功后，还需要安装https证书，这个比较简单就不记录了。

二、启动

在命令行中：>>>mitmdump -q  -p 8888 -s v1.py

-q是不显示一些乱七八糟的日志信息

-p是端口号

-s是代码文件

启动之后还要在电脑中的设置里找到代理服务器，设置好端口号打开，然后就可以了。

需注意的是只在电脑中吧代理服务器打开，而不开mitmproxy会没有网络的

三、请求

1.获得请求中常见的参数

from mitmproxy import http
from mitmproxy.http import Request


def request(flow: http.HTTPFlow):
    
    print("请求-->", flow.request.url)
    print("请求-->", flow.request.host)
    print("请求-->", flow.request.path)
    print("请求-->", flow.request.query)
    print("请求-->", flow.request.cookies)
    print("请求-->", flow.request.headers)
    print("请求-->", flow.request.method)
    print("请求-->", flow.request.content)


def response(flow: http.HTTPFlow):
    pass

2. 修改请求

from mitmproxy import http
from mitmproxy.http import Request
from mitmproxy.http import HTTPFlow


def request(flow: HTTPFlow):
    flow.request.url = "https://movie.douban.com/j/search_subjects?type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=20&page_start=20"


def response(flow: http.HTTPFlow):
    pass

3. 拦截请求并返回指定内容

from mitmproxy import http
from mitmproxy.http import Request
from mitmproxy.http import HTTPFlow
from mitmproxy.http import Response


def request(flow: HTTPFlow):
    if flow.request.url.startswith("网址"):
        flow.response = Response.make(
            200,  # (optional) status code
            b"Hello World",  # (optional) content
            {"Content-Type": "text/html"}  # (optional) headers
        )


def response(flow: http.HTTPFlow):
    pass

可以拦截，也可以直接杀死请求

from mitmproxy import http
from mitmproxy.http import Request
from mitmproxy.http import HTTPFlow


def request(flow: HTTPFlow):
    if flow.request.url.startswith("网址"):
        flow.kill()


def response(flow: http.HTTPFlow):

四、响应 

1.读取响应

from mitmproxy import http
from mitmproxy.http import Request
from mitmproxy.http import HTTPFlow
from mitmproxy.http import Response


def response(flow: http.HTTPFlow):
    print(flow.request.url)
    
    print(flow.response.status_code)
    print(flow.response.cookies)
    print(flow.response.headers)
    print(flow.response.content)

2.修改响应

from mitmproxy import http
from mitmproxy.http import Request
from mitmproxy.http import HTTPFlow
from mitmproxy.http import Response


def response(flow: http.HTTPFlow):
    print(flow.request.url)
    print(flow.response.status_code)
    print(flow.response.cookies)
    print(flow.response.headers)
    # print(flow.response.content)
    
    flow.response = Response.make(
        200,  # (optional) status code
        b"Hello World",  # (optional) content
        {"Content-Type": "text/html"}  # (optional) headers
    )