mitmproxy工具

目录

1、安装

2、Windows本机抓包初体验

（1）mitmdump抓包

（2）mitmproxy抓包 

（3）mitmweb抓包 

3、安装证书

（1）Windows安装证书抓取https包

（2）Android安装证书抓包

4、抓取get、post请求实战

（1）遇到问题 

​

​

（2）实时打印请求入参和响应出参

（3）抓包情况保存在cvs文件中

5、启动

（1）mitmproxy自带启动API

（2）subprocess启动

（3）多进程方式

6、实际案例

（1）pytest + mitmproxy

（2）pytest + mitmproxy+selenium

（3）测试中试试mock请求

appium+mitmproxy

为什么要用这个工具

（1）比Charles、wireshark这些界面抓包工具，可以集成到代码中，python就有三方包

（2） 比Charles、wireshark这些界面抓包工具，可以实时的分析过滤加工数据

（3）支持的平台多

官网的使用例子：Examples

1、安装

pip install mitmproxy

检查是否安装正确（三个工具的版本都能查看）

mitmproxy --version

mitmdump --version

mitmweb --version

  

2、Windows本机抓包初体验

（1）mitmdump抓包

启动：mitmdump   

# 默认启动的是8080端口

访问抓包：curl --proxy http://127.0.0.1:8080 "测试url"

 本机设置代理

（1）整个网络设置代理

 出现了！！！！

 （2）只对一个浏览器做代理

谷歌：

（1）命令行方式

第一步：找到浏览器的安装位置：桌面浏览器右键打开文件所在位置

第二步：在CMD中启动浏览器，并设置代理，要代理生效，必须没有打开的浏览器才行（那就把所有的火狐浏览页面都关闭）

"C:\Users\Administrator\AppData\Local\Google\Chrome\Application\chrome.exe" --proxy-server=127.0.0.1:8080 --ignore-certificate-errors

（2）浏览器中设置：忽略，自行百度

火狐：

（1）命令行方式：

只百度到了，火狐的代理文件在哪里放着，但是我本地却没有这个代理文件，所以后面在说吧

（2）浏览器中设置：忽略，自行百度 

（2）mitmproxy抓包 

启动：mitmproxy

进入交互式界面，如上，对自己的Windows机处理好代理，使用比较少

参考文章：mitmproxy的使用详解 - 会跑的熊 - 博客园

（3）mitmweb抓包 

下面都是基于mitmweb做的，和Charles等界面抓包工具类似

启动：mitmweb

3、安装证书

（1）Windows安装证书抓取https包

用户的目录下（启动mitmdump就生成了）

官网也有解释Certificates

命令行请求如何带证书：

curl --proxy 127.0.0.1:8080 --cacert C:/Users\hanfang9/.mitmproxy/mitmproxy-ca-cert.p12 https://example.com/

curl -x 172.23.68.1:80 -X POST "http://smart-robot-pre.jd.com/callResultForCallId?callId=626a876a-d36f-123a-cf82-f000ac1976e6"

wget -e https_proxy=127.0.0.1:8080 --ca-certificate ~/.mitmproxy/mitmproxy-ca-cert.pem https://example.com/

 再次刷新上面B站网页就好了，也抓到了https的包

（2）Android安装证书抓包

 参考文章：手机app数据的爬取之mitmproxy安装教程 - BruceLong - 博客园

 mitmproxy抓包工具！！！ 从安装到简单使用_哔哩哔哩_bilibili

第一步：给手机配置代理

第二步：手机浏览器下载证书（我的手机是华为mate20 pro）

浏览器输入：mitm.it

 手机里面查看证书，也可以删除

4、抓取get、post请求实战

目的：抓取所有的入参和出参等信息

方法：mitmdump\mitmweb都可以 -s 指定脚本，实时处理请求，拦截下来请求入参和出参等信息

（1）遇到问题 

这个接口没有拦截到post请求的入参

 curl -X POST "http://测试url/callResultForCallId?callId=626a876a-d36f-123a-cf82-f000ac1976e6" -x 127.0.0.1:8080 -H "content-type: application/json; charset=UTF-8" -d '{"d":1}'

 原因：mitmproxy代码分析了一波，我本身代码获取post的请求入参用的是urlencode_form的属性值，就有问题了，如图这样修改就在终端获取到了，虽然我不知道mitmweb展示的数据是怎么获取的，哈哈~~~（这里也可以照着urlencode_form创建一个属性值就不改变原有mitmproxy的逻辑了）

post请求入参有中文，编码格式（可以先看现象和解决，然后根据下面实时打印的代码试验）

 原因：mitmproxy根据header中content-type的编码格式来解码请求参数，cmd中的编码格式和header中传的不一致就会有问题

 试验：第一次不设置content-type

curl -X POST "http://smart-robot-pre.jd.com/callResultForCallId?callId=626a876a-d36f-123a-cf82-f000ac1976e6" -x 127.0.0.1:8080  -d '{"d":1,"f":"人民"}'

试验：第二次设置为utf-8

 curl -X POST "http://smart-robot-pre.jd.com/callResultForCallId?callId=626a876a-d36f-123a-cf82-f000ac1976e6" -x 127.0.0.1:8080 -H "content-type: application/json; charset=utf-8" -d '{"d":1,"f":"人民"}'

试验：第三次设置为gbk

下图：说明这次设置的匹配上了

curl -X POST "http://测试域名/callResultForCallId?callId=626a876a-d36f-123a-cf82-f000ac1976e6" -x 127.0.0.1:8080 -H "content-type: application/json; charset=gbk" -d '{"d":1,"f":"人民"}'

************************************************************************************************************

引发问题：修改cmd的编码格式，发现不太好使呢，我本来想变更cmd的是utf-8和header对应上，下面是cmd变更编码格式的文章

cmd 查看当前窗口的编码_lyw603267094的专栏-CSDN博客_cmd查看编码

（2）实时打印请求入参和响应出参

代码robot.py：

import json

from mitmproxy import ctx, http


class Robot:

    def request(self, flow: http.HTTPFlow):
        if flow.request.host : 
            ctx.log.warn("*" * 100)
            ctx.log.info("请求url： " + flow.request.url)
            ctx.log.info("请求方法： " + flow.request.method)
            ctx.log.info("请求headers： " + str(flow.request.headers))
            # query抓取的是get的数据
            get_p = flow.request.query
            # ctx.log.info("请求get入参： " + str(get_p))
            temp_json = {}
            for k, v in get_p.items():
                temp_json[k] = v
                # ctx.log.info(k+" "+v) unicode的value单独打印是中文，下面整体打印就是Unicode字符串，如果校验姓名这样的带星星的在说
            ctx.log.info("请求get入参： " + json.dumps(temp_json))
            ctx.log.info("请求get入参： " + str(temp_json)) #和上一句话在遇到中文的时候不一样，conlse展示最好选择下面的
            # post入参
            post_p = flow.request.urlencoded_form
            post_temp = ""
            for k in post_p.keys():
                post_temp = k
                # with open("D://dd.txt", "a", encoding="utf-8") as f:
                #     f.write(k)
            ctx.log.info("请求post入参： " + post_temp)
            ctx.log.warn("*" * 100)

    def response(self, flow: http.HTTPFlow):
        if flow.request.host: 
            ctx.log.warn("-" * 100)
            response = flow.response.content
            response = response.decode("utf-8")
            ctx.log.info("请求response： " + response)
            ctx.log.warn("-" * 100)

代码addons.py：

import robot

addons = [
    robot.Robot(),
]

（3）抓包情况保存在cvs文件中

代码robot.py：

import csv
import json
import os
import sys
import time
import urllib
from functools import reduce

from mitmproxy import http

headline = ["ID", "url", "request_method", "request_head", "get_params", "post_params", "reponse_status",
            "response_text"]


class CsvUtils:

    @staticmethod
    def generate_name_current_time():
        name = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
        return name

    @staticmethod
    def write_csv_by_add(csv_path, text):
        """
        :param csv_path: 文件路径
        :param text: 要写入的信息，必须是字典
        :return:
        """
        try:
            if text not in [None, ""]:
                if isinstance(text, dict):
                    with open(csv_path, "a+", encoding="utf-8", newline="") as w:
                        csv.field_size_limit(2000000000)
                        csv_writer = csv.DictWriter(w, fieldnames=headline)
                        csv_writer.writerow(text)
                    with open(csv_path, "r", encoding="utf-8") as r:
                        csv_reader = csv.DictReader(r)
                        temp = [row for row in csv_reader]
                        if temp:
                            print()
                            print("*" * 100)
                            for k, v in temp[-1].items():
                                print(k, "：", v.replace('\0', ''))
                            print("*" * 100)
                            print()
                else:
                    print("写入的text，不是列表格式")
            else:
                print("写入的text，不能为空")
        except Exception as e:
            print("%s写csv文件时发生未知异常" % e)


class MitmBasic:
    def __init__(self):
        # 刚才写的获取当前时间的工具，我们要习惯如果这个代码块常用我们就要把他写成一个工具类，方便日后调用
        self.result = {}
        self.count = 1
        self.result_name = CsvUtils.generate_name_current_time()
        self.result_path = os.path.dirname(os.path.realpath(__file__)) + "/../record_%s.csv" % self.result_name
        CsvUtils.write_csv_by_add(self.result_path, {k: k for k in headline})

    def request(self, flow: http.HTTPFlow):
        if flow.request.host:  # == "smart-robot-pre.jd.com"
            headers = {k: v for k, v in flow.request.headers.items()}
            query = {k: v for k, v in flow.request.query.items()}
            for k, v in query.items():
                if k == "productType":
                    print(v == "京东金条")
                print(k, v)
            post_param = None
            if flow.request.urlencoded_form:
                try:
                    post_p = eval(reduce(lambda x, y: x + y, flow.request.urlencoded_form.keys()))
                    post_param = {k: v for k, v in map(lambda x: x.split(":"), post_p[1:-1].split(","))}
                except Exception:
                    pass
            case_id = '{:0>5d}'.format(self.count)  # 生成用例id
            values = [case_id, urllib.parse.unquote(flow.request.url), flow.request.method, json.dumps(headers),
                      json.dumps(query), json.dumps(post_param)]
            self.result = {k: v for k, v in zip(headline, values)}
            print(self.result)

    def response(self, flow: http.HTTPFlow):
        if flow.request.host:  # == "smart-robot-pre.jd.com":
            response = flow.response
            return_code = response.status_code
            try:
                content = response.content.decode("utf-8")
            except UnicodeError:
                content = response.content
            self.result[headline[-2]] = return_code
            self.result[headline[-1]] = content
            CsvUtils.write_csv_by_add(self.result_path, self.result)
            self.count += 1

addons.py 

import robot

addons = [
    robot.MitmBasic(),
]

 效果：

5、启动

前言：我们命令行mitmdump、mitmweb是配置了C:\Python\Python39\Scripts这个path环境变量，所以找到了这个目录下的执行文件

在CMD下用tasklist命令可以查到mitmproxy的进程：

taskkill /F /PID  2148      （杀死进程）

tasklist|findstr "mitm"    （查找）

cmd = r"C:\Python\Python39\Scripts\mitmweb.exe -s D:\mycode\gitpull\playwright_study\mitmproxy_hf\addons.py"
subprocess.Popen(cmd, shell=True)

（1）mitmproxy自带启动API

这个版本支持的不是最新版本，需要降级到，我还没有找到最近的版本支持的config怎么配置

import asyncio
from mitmproxy import proxy, options, ctx
from mitmproxy.tools.dump import DumpMaster
from mitmproxy import http


class AdjustBody:
    def response(self, flow: http.HTTPFlow) -> None:
        if "google" in flow.request.url:
            print("Before intercept: %s" % flow.response.text)
            flow.response.content = bytes("This is replacement response", "UTF-8")
            print("After intercept: %s" % flow.response.text)


def start():
    add_on = AdjustBody()

    opts = options.Options(listen_host='127.0.0.1', listen_port=8888)
    proxy_conf = proxy.config.ProxyConfig(opts)

    dump_master = DumpMaster(opts)
    dump_master.server = proxy.server.ProxyServer(proxy_conf)
    dump_master.addons.add(add_on)

    try:
        asyncio.ensure_future(stop())
        dump_master.run()
    except KeyboardInterrupt:
        dump_master.shutdown()


async def stop():
    # Sleep 10s to do intercept
    await asyncio.sleep(10)
    ctx.master.shutdown()

if __name__ == '__main__':
    start()

（2）subprocess启动

cmd = r"C:\Python\Python39\Scripts\mitmdump.exe -s D:\mycode\gitpull\playwright_study\mitmproxy_hf\addons.py"
process = subprocess.Popen(cmd, creationflags=subprocess.CREATE_NEW_CONSOLE)
print(process.pid)
time.sleep(60)
process.terminate()

（3）多进程方式

pass

6、实际案例

（1）pytest + mitmproxy

conftest.py文件内容

def pytest_addoption(parser):
    parser.addoption("--mitm_path", action="store",
                     default="",
                     type=str,
                     help="--mitm_path：mitmproxy生成的cvs文件路径")
    parser.addoption("--mitm_proxy", action="store",
                     default="127.0.0.1:8080",
                     type=str,
                     help="--mitm_proxy：mitmproxy设置代理")


@pytest.fixture(scope="session", autouse=True)
def set_env_mitm_path(request):
    mitm_value = request.config.getoption("--mitm_path")
    os.environ['mitm_path'] = mitm_value
    print('\n --mitm_path参数值：', mitm_value)
    return mitm_value


@pytest.fixture(scope="session", autouse=True)
def set_env_mitm_proxy(request):
    mitm_proxy = request.config.getoption("--mitm_proxy")
    os.environ['mitm_proxy'] = mitm_proxy
    print('\n --mitm_proxy参数值：', mitm_proxy)
    return mitm_proxy


@pytest.fixture(scope="session")
def setup_mitmdump():
    # 获取 mitm保存cvs的文件名，没有设置就用用例做文件名
    if not os.environ.get("mitm_path"):
        caller = os.environ.get('PYTEST_CURRENT_TEST').split(':')[-1].split(' ')[0]
        mitm_path = "./testdata/" + caller + ".csv"
        os.environ["mitm_path"] = mitm_path
    # 启动mitmproxy子进程，这里也可以通过命令行控制，这里就不处理了，参照mitm_cvs_name可以设置
    cmd = r"mitmdump -s D:\mycode\gitpull\playwright_study\mitmproxy_hf\addons.py"
    print("start mitm")
    process = subprocess.Popen(cmd, creationflags=subprocess.CREATE_NEW_CONSOLE)
    time.sleep(1)
    yield
    time.sleep(5)
    print("stop mitm")
    process.kill()

 robot.py文件

import csv
import json
import os
import time
import urllib
from functools import reduce

from mitmproxy import http

headline = ["ID", "url", "request_method", "request_head", "get_params", "post_params", "reponse_status",
            "response_text"]


class CsvUtils:

    @staticmethod
    def generate_name_current_time():
        name = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
        return name

    @staticmethod
    def write_csv_by_add(csv_path, text):
        """
        :param csv_path: 文件路径
        :param text: 要写入的信息，必须是字典
        :return:
        """
        if not os.path.isdir(os.path.split(csv_path)[0]):
            os.makedirs(os.path.split(csv_path)[0])
        try:
            if text not in [None, ""]:
                if isinstance(text, dict):
                    with open(csv_path, "a+", encoding="utf-8", newline="") as w:
                        csv.field_size_limit(2000000000)
                        csv_writer = csv.DictWriter(w, fieldnames=headline)
                        csv_writer.writerow(text)
                    with open(csv_path, "r", encoding="utf-8") as r:
                        csv_reader = csv.DictReader(r)
                        temp = [row for row in csv_reader]
                        if temp:
                            print()
                            print("*" * 100)
                            for k, v in temp[-1].items():
                                print(k, "：", v.replace('\0', ''))
                            print("*" * 100)
                            print()
                else:
                    print("写入的text，不是列表格式")
            else:
                print("写入的text，不能为空")
        except Exception as e:
            print("%s写csv文件时发生未知异常" % e)


class MitmBasic:
    def __init__(self):
        # 刚才写的获取当前时间的工具，我们要习惯如果这个代码块常用我们就要把他写成一个工具类，方便日后调用
        self.result = {}
        self.count = 1
        if os.environ.get("mitm_path"):
            self.result_path = os.environ.get("mitm_path")
        else:
            self.result_name = "sys_" + CsvUtils.generate_name_current_time()
            self.result_path = os.path.dirname(os.path.realpath(__file__)) + "/../record/%s.csv" % self.result_name
        # subprocess 子进程设置的os.environ在主进程不能使用
        # os.environ["mitm_cvs_path"] = self.result_path
        # print(os.environ.get("mitm_cvs_path"))
        CsvUtils.write_csv_by_add(self.result_path, {k: k for k in headline})

    def request(self, flow: http.HTTPFlow):
        if flow.request.host:  # == "smart-robot-pre.jd.com"
            headers = {k: v for k, v in flow.request.headers.items()}
            query = {k: v for k, v in flow.request.query.items()}
            post_param = None
            if flow.request.urlencoded_form:
                try:
                    post_p = eval(reduce(lambda x, y: x + y, flow.request.urlencoded_form.keys()))
                    post_param = {k: v for k, v in map(lambda x: x.split(":"), post_p[1:-1].split(","))}
                except Exception:
                    pass
            case_id = '{:0>5d}'.format(self.count)  # 生成用例id
            values = [case_id, urllib.parse.unquote(flow.request.url), flow.request.method, json.dumps(headers),
                      json.dumps(query), json.dumps(post_param)]
            self.result = {k: v for k, v in zip(headline, values)}
            print(self.result)

    def response(self, flow: http.HTTPFlow):
        if flow.request.host:  # == "smart-robot-pre.jd.com":
            response = flow.response
            return_code = response.status_code
            try:
                content = response.content.decode("utf-8")
            except UnicodeError:
                content = response.content
            self.result[headline[-2]] = return_code
            self.result[headline[-1]] = content
            CsvUtils.write_csv_by_add(self.result_path, self.result)
            self.count += 1

 addons.py文件

import robot

addons = [
    robot.MitmBasic(),
]

 测试文件

import csv
import os

import allure
import pytest
import requests


class TestDemo:

    @pytest.mark.parametrize(
        "name,assert_word",
        [
            pytest.param("1", "smart", id="第一个", marks=pytest.mark.skip("不想跑")),
            pytest.param("2", "smart", id="第二个")
        ]
    )
    def test_002(self, setup_mitmdump, name, assert_word):
        print("我是用例%s" % name)
        url = "**********************" #这里是访问的url
        params = {"d": 1}
        headers = {"content-type": "application/json; charset=UTF-8"}
        proxies = {'http': 'http://127.0.0.1:8080'}  # 代理mitmproxy地址
        option = requests.post(url=url, headers=headers, params=params, proxies=proxies)
        print(option.status_code)

    @pytest.mark.usefixtures("setup_mitmdump")
    @pytest.mark.parametrize(
        "name,assert_word",
        [
            pytest.param("1", "smart", id="第一个"),
            pytest.param("2", "smart", id="第二个")
        ]
    )
    def test_003(self, name, assert_word):
        print("我是用例test_%s" % name)
        url = "**********************" #这里是访问的url
        params = {"d": 1}
        headers = {"content-type": "application/json; charset=UTF-8"}
        proxies = {'http': 'http://%s' % os.environ.get("mitm_proxy")}  # 代理mitmproxy地址
        option = requests.post(url=url, headers=headers, params=params, proxies=proxies)

        # 这里可以对抓包的结果处理：找到响应200的请求集合（断言部分）
        def _get_uri():
            if os.environ.get("mitm_path"):
                mitm_cvs_name = os.environ["mitm_path"]
                if os.path.isfile(mitm_cvs_name):
                    with open(mitm_cvs_name, "r", encoding="utf-8") as r:
                        csv_reader = csv.DictReader(r)
                        temp = [row for row in csv_reader]
                        for i in temp:
                            # 找到响应200的数据，可以写自己的数据了
                            if i["reponse_status"] == "200":
                                print()
                                print("%" * 100)
                                for k, v in i.items():
                                    print(k, "：", v.replace('\0', ''))
                                print("%" * 100)
                                print()
                    return True
                else:
                    return AssertionError("抓包结果路径不对：%s" % mitm_cvs_name)
            else:
                return AssertionError("没有找到抓包结果文件")

        _get_uri()

 执行效果：

pytest -s mitmproxy_hf\test1.py::TestDemo::test_003 --mitm_path="./tesss/newname.csv"

 视频：mitmproxy结合pytest_哔哩哔哩_bilibili

（2）pytest + mitmproxy+selenium

提供下代码和效果（可以自己对已有selenium的用例加上抓包，检测数据加密传输，检查调用接口失败率什么的）

这里写入csv开销不小，在读取也有开销，讲从csv读取改为存入前print到conlse

robot.py

import csv
import json
import os
import time
import urllib
from functools import reduce

from mitmproxy import http, ctx

headline = ["ID", "url", "request_method", "request_head", "get_params", "post_params", "reponse_status",
            "response_text"]


class CsvUtils:

    @staticmethod
    def generate_name_current_time():
        name = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
        return name

    @staticmethod
    def write_csv_by_add(csv_path, text):
        """
        :param csv_path: 文件路径
        :param text: 要写入的信息，必须是字典
        :return:
        """
        if not os.path.isdir(os.path.split(csv_path)[0]):
            os.makedirs(os.path.split(csv_path)[0])
        try:
            if text not in [None, ""]:
                if isinstance(text, dict):
                    with open(csv_path, "a+", encoding="utf-8", newline="") as w:
                        csv.field_size_limit(2000000000)
                        csv_writer = csv.DictWriter(w, fieldnames=headline)
                        csv_writer.writerow(text)
                else:
                    print("写入的text，不是列表格式")
            else:
                print("写入的text，不能为空")
        except Exception as e:
            print("%s写csv文件时发生未知异常" % e)


class MitmBasic:
    def __init__(self):
        # 刚才写的获取当前时间的工具，我们要习惯如果这个代码块常用我们就要把他写成一个工具类，方便日后调用
        self.result = {}
        self.count = 1
        if os.environ.get("mitm_path"):
            self.result_path = os.environ.get("mitm_path")
        else:
            self.result_name = "sys_" + CsvUtils.generate_name_current_time()
            self.result_path = os.path.dirname(os.path.realpath(__file__)) + "/../record/%s.csv" % self.result_name
        CsvUtils.write_csv_by_add(self.result_path, {k: k for k in headline})

    def request(self, flow: http.HTTPFlow):
        if flow.request.host:  
            headers = {k: v for k, v in flow.request.headers.items()}
            query = {k: v for k, v in flow.request.query.items()}
            post_param = None
            if flow.request.urlencoded_form:
                try:
                    post_p = eval(reduce(lambda x, y: x + y, flow.request.urlencoded_form.keys()))
                    post_param = {k: v for k, v in map(lambda x: x.split(":"), post_p[1:-1].split(","))}
                except Exception:
                    pass
            case_id = '{:0>5d}'.format(self.count)  # 生成用例id
            values = [case_id, urllib.parse.unquote(flow.request.url), flow.request.method, json.dumps(headers),
                      str(query), str(post_param)]
            self.result = {k: v for k, v in zip(headline, values)}

    def response(self, flow: http.HTTPFlow):
        if flow.request.host:  
            response = flow.response
            return_code = response.status_code
            try:
                content = response.content.decode("utf-8")
            except UnicodeError:
                content = ""
            self.result[headline[-2]] = return_code
            self.result[headline[-1]] = content[:10]
            ctx.log.info("*"*100)
            for k, v in self.result.items():
                ctx.log.info(k + " : " + str(v))
            ctx.log.info("*" * 100)
            CsvUtils.write_csv_by_add(self.result_path, self.result)
            self.count += 1

conftest.py

import os
import subprocess
import time

import pytest

from mitmproxy_hf.pytest_mitm_selenium.rap_kanban import *


def pytest_collection_modifyitems(items):
    """
    测试用例收集完成时，将收集到的item的name和nodeid的中文显示在控制台上
    """
    for item in items:
        item.name = item.name.encode("utf-8").decode("unicode_escape")
        item._nodeid = item.nodeid.encode("utf-8").decode("unicode_escape")
        if item.obj.__doc__:
            item._nodeid += ":" + item.obj.__doc__.strip()


def pytest_addoption(parser):
    parser.addoption("--mitm_path", action="store",
                     default="",
                     type=str,
                     help="--mitm_path：mitmproxy生成的cvs文件名称")
    parser.addoption("--mitm_proxy", action="store",
                     default="127.0.0.1:8080",
                     type=str,
                     help="--mitm_proxy：mitmproxy设置代理")


@pytest.fixture(scope="session", autouse=True)
def set_env_mitm_path(request):
    mitm_value = request.config.getoption("--mitm_path")
    os.environ['mitm_path'] = mitm_value
    print('\n --mitm_path参数值：', mitm_value)
    return mitm_value


@pytest.fixture(scope="session", autouse=True)
def set_env_mitm_proxy(request):
    mitm_proxy = request.config.getoption("--mitm_proxy")
    os.environ['mitm_proxy'] = mitm_proxy
    print('\n --mitm_proxy参数值：', mitm_proxy)
    return mitm_proxy


@pytest.fixture(scope="session")
def setup_mitmdump():
    if not os.environ.get("mitm_path"):
        caller = os.environ.get('PYTEST_CURRENT_TEST').split(':')[-1].split(' ')[0]
        mitm_path = "./testdata/" + caller + ".csv"
        print(mitm_path)
        os.environ["mitm_path"] = mitm_path
    cmd = r"mitmdump -s D:\mycode\gitpull\playwright_study\mitmproxy_hf\pytest_mitm_selenium\addons.py"
    #cmd = "mitmdump"
    process = subprocess.Popen(cmd, creationflags=subprocess.CREATE_NEW_CONSOLE)
    time.sleep(1)
    yield
    time.sleep(1)
    print("stop mitm")
    process.kill()


@pytest.fixture(scope="session")
def setup_driver():
    my_driver = RAPDriver()
    my_driver.login()
    yield my_driver
    my_driver.login_out()

selenium的demo文件rap_kanban.py

import os
import time
from datetime import datetime
from functools import reduce

import allure
from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.support.ui import WebDriverWait


URL = "*******************" #被测试的url
USERNAME = "用户名"
PASSWORD = "密码"


def detection_time(func):
    def wrap(*args, **kwargs):
        start_time = time.time()
        res = func(*args, **kwargs)
        end_time = time.time()
        print("耗时：%s" % (end_time - start_time - res))
        return res

    return wrap


class RAPDriver:
    def __init__(self, url=URL, username=USERNAME, password=PASSWORD):
        self.url = url
        self.username = username
        self.password = password
        options = webdriver.ChromeOptions()
        options.add_argument(r"user-data-dir=C:\Users\hanfang9\AppData\Local\Google\Chrome\User Data1")
        options.add_argument("--proxy-server=http://%s" % os.environ.get("mitm_proxy"))
        options.add_experimental_option("excludeSwitches", ['enable-automation', 'enable-logging'])
        self.driver = webdriver.Chrome(options=options,
                                       executable_path=r"C:\Program Files\Google\Chrome\Application\chromedriver.exe")
        # driver.implicitly_wait(10)  executable_path='<path-to-chrome>'
        self.title = None
        self.wait = WebDriverWait(self.driver, 10000)
        self.result = []
        self.result_png = ''
        # 时间戳控制结果
        self.my_time = datetime.now().strftime("%Y-%m-%d-%H-%M-%S")

    @allure.step("登录")
    def login(self):
        with allure.step("用户名：{}---密码：{}".format(self.username, self.password)):
            self.driver.get(self.url)
            self.driver.maximize_window()
            username = self.wait.until(lambda x: x.find_element(by="id", value="username"))
            username.clear()
            username.send_keys(self.username)
            passworld = self.driver.find_element(by="id", value="password")
            passworld.clear()
            passworld.send_keys(self.password)
            login_button = self.driver.find_element(by="id", value="formsubmitButton")
            login_button.click()
            print("-------------------------------------登录成功-------------------------------------")

    @allure.step("登出")
    def login_out(self):
        self.driver.quit()
        print("-------------------------------------登出成功-------------------------------------")

    def check_page(self):
        for w in range(10):
            print("\t\t\t第%s次尝试点击下一页" % w)
            try:
                button_next = self.wait.until(lambda x: x.find_element(by="xpath",
                                                                       value='//*/li[@title="下一页"]'))
                button_next.click()
                time.sleep(3)
                # 滚动到浏览器顶部
                js_top = "var q=document.documentElement.scrollTop=0"
                self.driver.execute_script(js_top)
                return
            except Exception:
                continue

    def click_tu(self):
        for w in range(10):
            print("\t\t\t第%s次尝试获取查询的看板" % w)
            tus = self.wait.until(lambda x: x.find_elements(by="class name", value="card-with-img___2zBnS"))
            try:
                for ru in tus:
                    ru.click()
                    time.sleep(1)
                return
            except Exception:
                continue

    def clear_headle(self):
        time.sleep(10)
        handles = self.driver.window_handles
        handle_to = None
        for handle in handles:
            self.driver.switch_to.window(handle)
            if self.driver.title != self.title:
                try:
                    # 截图搞出来
                    temp = self.driver.current_url
                    self.screenshot(temp.split("/")[-1])
                    if "暂无数据" in self.driver.page_source:
                        time.sleep(1)
                        pass_url = []
                        if temp not in pass_url:
                            self.result.append(temp)
                        # temp = temp.split("/")[-1]
                        # self.screenshot(temp)
                    self.driver.close()
                except Exception as e:
                    print("关闭失败")
                    continue
            else:
                handle_to = handle
        self.driver.switch_to.window(handle_to)

    def screenshot(self, png_name):
        # 这个是二进制文件流
        png_data = self.driver.get_screenshot_as_png()
        allure.attach(png_data, png_name, allure.attachment_type.PNG)

    @detection_time
    def test_tu(self, test_data):
        self.login()
        # 看板查询按钮和输入看板名称输入框
        input1 = self.wait.until(lambda x: x.find_element(by="xpath", value='//*[@id="advanced_search_name"]'))
        search_button = self.wait.until(
            lambda x: x.find_element(by="xpath", value='//*[@id="advanced_search"]/div/div[5]/button[1]/span'))
        index = 0
        self.title = self.driver.title
        print(self.title)
        for i in test_data:
            try:
                index += 1
                print("查询第%s个数据: %s" % (index, i))
                time.sleep(1)
                # 输入查询的看板名称
                input1.send_keys(Keys.CONTROL, 'a')
                time.sleep(1)
                input1.send_keys(i)
                search_button.click()
                time.sleep(3)
                total_num = self.driver.find_element(by="xpath",
                                                     value='//*[@id="root"]/div/section/div/main/div/div[2]/div/div/div[1]/ul/li[1]')
                total_num = total_num.text.split()[-2]
                page_num = int(int(total_num) / 10) + 1
                print("\t一共多少页：%s--%s" % (total_num, page_num))
                for y in range(page_num):
                    print("\t\t第%s页" % (y + 1))
                    self.click_tu()
                    if y != page_num - 1:
                        self.check_page()
                    self.clear_headle()
            except Exception as e:
                print(e)
                print("失败：%s" % i)
                continue

        self.login_out()
        for i in self.result:
            print(i)

    @detection_time
    @allure.step("测试方法入口")
    def test_case(self, test_data):
        res = []
        res_index = 0 if len(self.result) == 0 else len(self.result)
        # 看板查询按钮和输入看板名称输入框
        input1 = self.wait.until(lambda x: x.find_element(by="xpath", value='//*[@id="advanced_search_name"]'))
        search_button = self.wait.until(
            lambda x: x.find_element(by="xpath", value='//*[@id="advanced_search"]/div/div[5]/button[1]/span'))
        self.title = self.driver.title
        try:
            print("查询数据: %s" % test_data)
            time.sleep(1)
            # 输入查询的看板名称
            input1.send_keys(Keys.CONTROL, 'a')
            time.sleep(1)
            input1.send_keys(test_data)
            search_button.click()
            time.sleep(3)
            total_num = self.driver.find_element(by="xpath",
                                                 value='//*[@id="root"]/div/section/div/main/div/div[2]/div/div/div[1]/ul/li[1]')
            total_num = total_num.text.split()[-2]
            page_num = int(int(total_num) / 10) + 1
            print("\t一共多少页：%s--%s" % (total_num, page_num))
            for y in range(page_num):
                print("\t\t第%s页" % (y + 1))
                self.click_tu()
                if y != page_num - 1:
                    self.check_page()
                self.clear_headle()
        except Exception as e:
            print(e)
            print("失败：%s" % test_data)
        try:
            res = self.result[res_index:]
            result_url = reduce(lambda a, b: a + b,
                                map(lambda x: '<a href="{}" target="_blank">{}</a></li><li>'.format(x, x), res))
        except Exception:
            result_url = ""
        if result_url:
            html_content = """
                    <!DOCTYPE html>
            <html lang="en">
                <body>
                    <style type="text/css">.ffone{ list-style-type:decimal-leading-zero; }</style>
                        <ul class="ffone">
                            <li>""" + result_url[:-4] + """
                        </ul>
                </body>
            </html>
                    """
            print(result_url)
            allure.attach(html_content, "查询失败的url", allure.attachment_type.HTML)
            result_url_total = reduce(lambda a, b: a + b,
                                      map(lambda x: '<a href="{}" target="_blank">{}</a></li><li>'.format(x, x),
                                          self.result))
            html_content_total = """
                                <!DOCTYPE html>
                        <html lang="en">
                            <body>
                                <style type="text/css">.ffone{ list-style-type:decimal-leading-zero; }</style>
                                    <ul class="ffone">
                                        <li>""" + result_url_total[:-4] + """
                                    </ul>
                            </body>
                        </html>
                                """
            allure.attach(html_content_total, "查询失败的url_total", allure.attachment_type.HTML)

        if result_url:
            return False
        else:
            return True


if __name__ == '__main__':
    driver = RAPDriver(URL, USERNAME, PASSWORD)
    driver.test_tu(["测试"])

 用例文件：

import allure
import pytest

testdata = ['智能坐席助手', '通过率', '互联网金融协会当日发起实时查询量']


@allure.feature("看板检查")
@pytest.mark.usefixtures("setup_mitmdump")
@pytest.mark.parametrize("test_data", testdata)
def test_kanban(test_data, setup_driver):
    result = setup_driver.test_case(test_data)
    assert result, "有查询失败数据"

 效果：mitmproxy+selenium+pytest+allure_哔哩哔哩_bilibili

pytest -v 路径\test_rap.py --alluredir=./report

allure generate ./report -o ./html

（3）测试中试试mock请求

工作中遇到问题：A. 外部依赖接口没有数据，不想等着找其他部门去找有数的情况

                             B.特定的入参才有数据的接口，需要实时修改入参

A问题

B问题

 这里的请求header中Content-Type:application/json

 上面获取不到json的入参的时候我修改了urlencoded_form方法的代码，这里似乎不太好用了

我考虑欠佳

在上图代码下面添加

    def _get_application_json(self):
        is_valid_content_type = "application/json" in self.headers.get("content-type", "").lower()
        if is_valid_content_type:
            try:
                return jsonpart.decode(self.headers, self.content)
            except ValueError:
                pass
        return ()

    def _set_application_json(self, value):
        self.content = mitmproxy.net.http.jsonpart.encode(self.headers, value)
        self.headers["content-type"] = "application/json"

    @property
    def application_json(self):
        return multidict.MultiDictView(
            self._get_application_json,
            self._set_application_json
        )

    @application_json.setter
    def application_json(self, value):
        self._set_application_json(value)

 其实是仿照multipart_form写的

import re
import mimetypes
from urllib.parse import quote
from mitmproxy.net.http import headers
import json


def encode(head, l):
    k = head.get("content-type")
    if k:
        k = headers.parse_content_type(k)
        if k is not None:
            return bytes(l[0],encoding="utf-8")


def decode(hdrs, content):
    """
        Takes a multipart boundary encoded string and returns list of (key, value) tuples.
    """
    v = hdrs.get("content-type")
    if v:
        v = headers.parse_content_type(v)
        r = []
        if content is not None:
            r.append(content.decode("utf-8"))
        return r
    return []

 然后自己的业务代码：

使用方法：代码写好后，我启动了mitmdump后，启动一个浏览器设置代理，这样这个浏览器有代理，其他浏览器也不耽误使用，不必要吧所有访问都修改拦截 

 

 

 

appium+mitmproxy

（未更新完）

第一步：python能使用appium

（1）安装nodejs：Node.js

                        配置环境变量：D:\Program Files\nodejs（你的安装路径）

（2）安装appium：Appium: Mobile App Automation Made Awesome.

（3）安装安卓软件开发工具包:AndroidDevTools - Android开发工具 Android SDK下载 Android Studio下载 Gradle下载 SDK Tools下载

     也可以通过Android Studio下载SDK：https://developer.android.com/studio/index.html?hl=zh-cn           

        配置环境变量：ANDROID_HOME=C:\Users\hanfang9\AppData\Local\Android\Sdk

                                 PATH添加  %ANDROID_HOME%\tools     

                                                   %ANDROID_HOME%\platform-tools

（4）JDK（省略）

（5）python和驱动

         pip install Appium-Python-Client

（6）模拟器：MuMu模拟器官网_安卓模拟器_网易MuMu手游模拟器

*****************************************************************************************************

我的手机是华为的鸿蒙系统：还没有搞定

鸿蒙系统链接appium：appium学习笔记03-鸿蒙系统连接appium - Rookie_C - 博客园

adb命令链接模拟器：

adb connect 127.0.0.1:7555

adb devices

adb shell

dumpsys activity|grep mFocusedActivity

 

selenium+mitmproxy