playwright-har 试用

DAOSquare

于 2024-07-01 02:21:43 发布

阅读量7

点赞数

playwright-har是puppeteer-har 的playwright 移植，使用方法上类似，以下是一个简单试用

环境准备

docker-compose
使用了browserless

version: "3"
services:
  browser:
    image: ghcr.io/browserless/chromium:latest
    environment:
      - CONCURRENT=40
      - QUEUED=20
      - CORS=true
      - CORS_MAX_AGE=300
      - DATA_DIR=/tmp/my-profile
      - TOKEN=6R0W53R135510
    volumes:
      - ./my-profile:/tmp/my-profile
    ports:
      - "3000:3000"

使用

安装依赖yarn add playwright playwright-har

dalongv3.js

const { chromium } = require("playwright");
const {PlaywrightHar} = require('playwright-har');
 
(async () => {
  let browser = await chromium.connectOverCDP(
    "ws://localhost:3000?token=6R0W53R135510"
  );
  let bContext = await browser.newContext({
    userAgent:
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
  });
 
  let page = await bContext.newPage();
 
  const playwrightHar = new PlaywrightHar(page);
  await playwrightHar.start();
 
  await page.goto("https://news.baidu.com/", {
    waitUntil: "networkidle",
  });
  let json_result = []
  let resutlv2 = await page.locator(".column").all();
  for (const row of await resutlv2) {
    json_result.push({address:"https://news.baidu.com/",content:await row.innerHTML(),title:"dalong.js"})
  }
  console.log(JSON.stringify(json_result))
  await playwrightHar.stop('./baidu.har');
  await browser.close();
})();

har 文件效果

playwright-har 试用_Network

内部实现

内部实现上使用了chrome-har 这个npm 包

参考实现

import { harFromMessages } from 'chrome-har';
import { writeFileSync } from 'fs';
import { CDPSession, Page } from 'playwright-chromium';
import { PlaywrightHarConfig } from './playwright-har-config';
 
export class PlaywrightHar {
 
    private page: Page;
    private client: CDPSession;
    private addResponseBodyPromises = [];
    private events = [];
    private config: PlaywrightHarConfig;
 
    constructor(page: Page, config: PlaywrightHarConfig = null) {
        this.page = page;
 
        if (config == null) {
          this.config = new PlaywrightHarConfig();
        }
    }
 
    async start() {
        //@ts-ignore
        // newCDPSession is only available for ChromiumBrowserContext
       // 通过page 的创建新的cdp 回话，之后就是对于page 不同事件的监听，内容包含了事件以及事件的content body 
        this.client = await this.page.context().newCDPSession(this.page);
        await this.client.send('Page.enable');
        await this.client.send('Network.enable');
        const observe = [
            'Page.loadEventFired',
            'Page.domContentEventFired',
            'Page.frameStartedLoading',
            'Page.frameAttached',
            'Page.frameScheduledNavigation',
            'Network.requestWillBeSent',
            'Network.requestServedFromCache',
            'Network.dataReceived',
            'Network.responseReceived',
            'Network.resourceChangedPriority',
            'Network.loadingFinished',
            'Network.loadingFailed',
            'Network.getResponseBody'
        ];
        observe.forEach(method => {
            //@ts-ignore
            // Doesn't work when array contains symbols instead of strings
            this.client.on(method, params => {
                const harEvent = { method, params };
                this.events.push(harEvent);
                if (method === 'Network.responseReceived') {
                    if (this.config.recordResponses === false) {
                        return;
                    }
 
                    const response = harEvent.params.response;
                    const requestId = harEvent.params.requestId;
                    // Response body is unavailable for redirects, no-content, image, audio and video responses
                    if (
                        response.status !== 204 &&
                        response.headers.location == null &&
                        !response.mimeType.includes('image') &&
                        !response.mimeType.includes('audio') &&
                        !response.mimeType.includes('video')
                    ) {
                        const addResponseBodyPromise = this.client.send('Network.getResponseBody', { requestId }).then(
                            responseBody => {
                                // Set the response so chrome-har can add it to the HAR file
                                harEvent.params.response = {
                                    ...response,
                                    body: Buffer.from(responseBody.body, responseBody.base64Encoded ? 'base64' : undefined).toString()
                                };
                            },
                            reason => { }
                        );
                        this.addResponseBodyPromises.push(addResponseBodyPromise);
                    }
                }
            });
        });
    }
  // 对于保存的事件以及事件内容，基于harFromMessages 包装har 内容，之后就是内容的写入
    async stop(path?: string) {
        await Promise.all(this.addResponseBodyPromises);
        const harObject = harFromMessages(this.events, { includeTextFromResponseBody: this.config.recordResponses !== false });
        this.events = [];
        this.addResponseBodyPromises = [];
        if (path) {
            writeFileSync(path, JSON.stringify(harObject));
        }
        else {
            return harObject
        }
 
    }
}

说明

har 好多时候是分析问题一个很不错的选择，playwright-har 值得试用下

参考资料

https://github.com/janzaremski/playwright-har
https://github.com/Everettss/puppeteer-har

原创作者: rongfengliang 转载于: https://blog.51cto.com/rongfengliang/11305972

DAOSquare

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
playwright-har 试用

playwright-har是puppeteer-har 的playwright 移植，使用方法上类似，以下是一个简单试用环境准备docker-compose 使用了browserless登录后复制 version: "3"services: browser: image: ghcr.io/browser...
复制链接

扫一扫