浏览器中实现绘制人体骨架
最近在为后续项目做技术预研遇到了一个问题,需要再视频上绘制人体骨架,并在相应的关节点上显示相关数据。
开始方案是python通过模型识别人体关键点位置并将位置推送到mqtt服务上并将视频在推送到视频服务器上,然后前端订阅mqtt的数据并拉取python推送的流数据
这样实现后发现视频流会存在很大的延迟,并且没法和响应时间的数据对应上。后面再不停地找新的实现方案的时候,发现现在浏览器已经能使用一些模型了,
后就预研了一下这个方案
方案说明
前端通过chrome浏览器提供的运行模型的能力来进行人体骨架的一个识别,识别完成后,在video元素上叠加一个canvas元素,
在canvas通过识别出来的人体骨架关键点进行人体骨架的绘制
安装插件
pnpm i @mediapipe/tasks-vision
官网文档地址在文章末尾
引用插件
import {DrawingUtils, PoseLandmarker,} from "@mediapipe/tasks-vision";
使用
先创建模型
const createPoseLandmarker = async () => {
poseLandmarker = await PoseLandmarker.createFromOptions({
wasmBinaryPath: 'http://my.zsyou.top/2024/vision_wasm_internal.wasm', //运行模型需要相关的文件,原始官网是通过调用一个网址获取这两个文件地址,原地址需要科学上网,
wasmLoaderPath: 'http://my.zsyou.top/2024/vision_wasm_internal.js'
}, {
baseOptions: {
modelAssetPath: `http://my.zsyou.top/2024/pose_landmarker_lite.task`, //模型文件
delegate: "GPU"
},
runningMode: 'VIDEO',
numPoses: 2
});
console.log('poseLandmarker=====>', poseLandmarker)
};
获取视频流
const start = async () => {
await createPoseLandmarker() //因为下载文件需要时间所以提前下载
const constraints = {
video: true
};
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
videoElement1.value.srcObject = stream;
videoElement1.value.addEventListener("loadeddata", predictWebcam);
});
} else {
message.error('调用摄像头失败')
}
}
绘制人体骨架
async function predictWebcam() {
const render = () => {
const canvasCtx = canvasElement1.value.getContext("2d");
const drawingUtils = new DrawingUtils(canvasCtx);
let startTimeMs = performance.now();
if (lastVideoTime !== videoElement1.value.currentTime) {
lastVideoTime = videoElement1.value.currentTime;
poseLandmarker.detectForVideo(videoElement1.value, startTimeMs, (result) => {
canvasCtx.save();
canvasCtx.clearRect(0, 0, canvasElement1.value.width, canvasElement1.value.height);
for (const landmark of result.landmarks) {
drawingUtils.drawLandmarks(landmark, {
color: '#FF0000',
fillColor: '#FF0000',
radius: 2
})
drawingUtils.drawConnectors(landmark, PoseLandmarker.POSE_CONNECTIONS, {
lineWidth: 2,
color: '#00FF00',
fillColor: '#00FF00',
});
}
canvasCtx.restore();
})
}
requestAnimationFrame(render);
}
render();
}
完整代码
<!--------------------------------------------
* @Author: shiyzhang zsyou0911@126.com
* @Date: 2024/8/23 下午2:31
* @Description:
*
------------------------------------------------>
<template>
<div style="width: 100%">
<ASpace>
<AButton @click="start">开 始</AButton>
<AButton @click="stop"> 关闭摄像头</AButton>
</ASpace>
<div class="pose_land_marker">
<video ref="videoElement1" autoplay class="local_video" style="height: 480px;width: 640px"></video>
<canvas ref="canvasElement1" class="canvas_element" height="480" style="height: 480px;width: 640px;right: 0"
width="640"></canvas>
</div>
</div>
</template>
<script lang="ts" setup>
import {onMounted, ref} from 'vue'
import {message} from 'ant-design-vue'
import dayjs from 'dayjs'
import {DrawingUtils, PoseLandmarker,} from "@mediapipe/tasks-vision";
import {appendWriteFile, getUserBaseDirHandle} from './untis/index'
let poseLandmarker: PoseLandmarker = undefined;
let lastVideoTime = -1;
const videoElement1 = ref(null)
const canvasElement1 = ref(null)
const height = 480
const width = 640
const createPoseLandmarker = async () => {
poseLandmarker = await PoseLandmarker.createFromOptions({
wasmBinaryPath: 'http://my.zsyou.top/2024/vision_wasm_internal.wasm',
wasmLoaderPath: 'http://my.zsyou.top/2024/vision_wasm_internal.js'
}, {
baseOptions: {
modelAssetPath: `http://my.zsyou.top/2024/pose_landmarker_lite.task`,
delegate: "GPU"
},
runningMode: 'VIDEO',
numPoses: 2
});
console.log('poseLandmarker=====>', poseLandmarker)
};
onMounted(() => {
})
//停止
const stop = () => {
if (videoElement1.value.srcObject) {
videoElement1.value.srcObject?.getTracks().forEach((track) => track.stop());
}
}
//开始
const start = async () => {
await createPoseLandmarker()
const constraints = {
video: true
};
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
videoElement1.value.srcObject = stream;
videoElement1.value.addEventListener("loadeddata", predictWebcam);
});
} else {
message.error('调用摄像头失败')
}
}
async function predictWebcam() {
const render = () => {
const canvasCtx = canvasElement1.value.getContext("2d");
const drawingUtils = new DrawingUtils(canvasCtx);
let startTimeMs = performance.now();
if (lastVideoTime !== videoElement1.value.currentTime) {
lastVideoTime = videoElement1.value.currentTime;
poseLandmarker.detectForVideo(videoElement1.value, startTimeMs, (result) => {
canvasCtx.save();
canvasCtx.clearRect(0, 0, canvasElement1.value.width, canvasElement1.value.height);
for (const landmark of result.landmarks) {
drawingUtils.drawLandmarks(landmark, {
color: '#FF0000',
fillColor: '#FF0000',
radius: 2
})
drawingUtils.drawConnectors(landmark, PoseLandmarker.POSE_CONNECTIONS, {
lineWidth: 2,
color: '#00FF00',
fillColor: '#00FF00',
});
}
canvasCtx.restore();
})
}
requestAnimationFrame(render);
}
render();
}
</script>
<style scoped>
.pose_land_marker {
position: relative;
margin-top: 10px;
height: 480px;
.canvas_element, .local_video {
position: absolute;
left: 0;
top: 0;
}
}
</style>
官网api
https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker/web_js?hl=zh-cn
需要科学上网,里面有多种模型,以及示例
注意部分的@mediapipe/tasks-vision
包不一致