【深度学习实战—12】：基于MediaPipe的手势识别

✨博客主页：王乐予🎈
✨年轻人要：Living for the moment（活在当下）！💪
🏆推荐专栏：【图像处理】【千锤百炼Python】【深度学习】【排序算法】

GitHub 源码地址：https://github.com/WangLeYuu/Gesture-Recognition

😺一、MediaPipe概述

MediaPipe 是一款由 Google Research 开发并开源的多媒体机器学习模型应用框架。

MediaPipe目前支持的解决方案(Solution)及支持的平台如下图所示：

😺二、MediaPipe手部特征点检测

官网描述：

https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker
The MediaPipe Hand Landmarker task lets you detect the landmarks of the hands in an image. You can use this task to locate key points of hands and render visual effects on them. This task operates on image data with a machine learning (ML) model as static data or a continuous stream and outputs hand landmarks in image coordinates, hand landmarks in world coordinates and handedness(left/right hand) of multiple detected hands.

该模型会跟踪手部 21 个关键点位置：

😺三、任务描述

本文将通过 Mediapipe 检测出手部关键点，并通过对各种关键点的位置判别，以达到手势识别的目的。本文将对如下 6 种手势进行判定识别：

1. OK
2. Return
3. Left
4. Right
5. Like
6. Pause

😺四、代码实现

🐶4.1 度量函数

🦄4.1.1 距离度量

计算两点之间的距离如下：

"""
计算两个点之间的距离：L2距离（欧式距离）
"""
def points_distance(x0, y0, x1, y1):
    return math.sqrt((x0 - x1) ** 2 + (y0 - y1) ** 2)

🦄4.1.2 角度度量

计算两向量之间的角度如下：

"""
计算两条线段之间的夹角，以弧度表示
"""
def compute_angle(x0, y0, x1, y1, x2, y2, x3, y3):
    AB = [x1 - x0, y1 - y0]
    CD = [x3 - x2, y3 - y2]

    dot_product = AB[0] * CD[0] + AB[1] * CD[1]

    AB_distance = points_distance(x0, y0, x1, y1) + 0.001   # 防止分母出现0
    CD_distance = points_distance(x2, y2, x3, y3) + 0.001

    cos_theta = dot_product / (AB_distance * CD_distance)

    theta = math.acos(cos_theta)

    return theta

🐶4.2 工作流程

1. 通过 Mediapipe 获取所有手部关键点；
2. 检测每根手指的状态（弯曲 or 伸直）；
3. 判断当前手势（定义多组手势判断函数）；
4. 如果连续30帧均为同一种手势，则可视化手势内容。

🐶4.3 代码实现

🦄4.3.1 定义手指状态函数（弯曲 or 伸直）

def detect_all_finger_state(all_points):

    finger_first_angle_bend_threshold = math.pi * 0.25  # 大拇指弯曲阈值
    finger_other_angle_bend_threshold = math.pi * 0.5  # 其他手指弯曲阈值
    finger_other_angle_straighten_threshold = math.pi * 0.2  # 其他手指伸直阈值

    first_is_bend = False
    first_is_straighten = False
    second_is_bend = False
    second_is_straighten = False
    third_is_bend = False
    third_is_straighten = False
    fourth_is_bend = False
    fourth_is_straighten = False
    fifth_is_bend = False
    fifth_is_straighten = False

    finger_first_angle = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point1'][0], all_points['point1'][1],
                                        all_points['point2'][0], all_points['point2'][1], all_points['point4'][0], all_points['point4'][1])
    finger_sencond_angle = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point5'][0], all_points['point5'][1],
                                        all_points['point6'][0], all_points['point6'][1], all_points['point8'][0], all_points['point8'][1])
    finger_third_angle = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point9'][0], all_points['point9'][1],
                                        all_points['point10'][0], all_points['point10'][1], all_points['point12'][0], all_points['point12'][1])
    finger_fourth_angle = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point13'][0], all_points['point13'][1],
                                        all_points['point14'][0], all_points['point14'][1], all_points['point16'][0], all_points['point16'][1])
    finger_fifth_angle = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point17'][0], all_points['point17'][1],
                                        all_points['point18'][0], all_points['point18'][1], all_points['point20'][0], all_points['point20'][1])

    if finger_first_angle > finger_first_angle_bend_threshold:              # 判断大拇指是否弯曲
        first_is_bend = True
        first_is_straighten = False
    else:
        first_is_bend = False
        first_is_straighten = True

    if finger_sencond_angle > finger_other_angle_bend_threshold:            # 判断食指是否弯曲
        second_is_bend = True
    elif finger_sencond_angle < finger_other_angle_straighten_threshold:
        second_is_straighten = True
    else:
        second_is_bend = False
        second_is_straighten = False

    if finger_third_angle > finger_other_angle_bend_threshold:              # 判断中指是否弯曲
        third_is_bend = True
    elif finger_third_angle < finger_other_angle_straighten_threshold:
        third_is_straighten = True
    else:
        third_is_bend = False
        third_is_straighten = False

    if finger_fourth_angle > finger_other_angle_bend_threshold:             # 判断无名指是否弯曲
        fourth_is_bend = True
    elif finger_fourth_angle < finger_other_angle_straighten_threshold:
        fourth_is_straighten = True
    else:
        fourth_is_bend = False
        fourth_is_straighten = False

    if finger_fifth_angle > finger_other_angle_bend_threshold:              # 判断小拇指是否弯曲
        fifth_is_bend = True
    elif finger_fifth_angle < finger_other_angle_straighten_threshold:
        fifth_is_straighten = True
    else:
        fifth_is_bend = False
        fifth_is_straighten = False

    # 将手指的弯曲或伸直状态存在字典中，简化后续函数的参数
    bend_states = {
   'first': first_is_bend, 'second': second_is_bend, 'third': third_is_bend, 'fourth': fourth_is_bend, 'fifth': fifth_is_bend}
    straighten_states = {
   'first': first_is_straighten, 'second': second_is_straighten, 'third': third_is_straighten, 'fourth': fourth_is_straighten, 'fifth': fifth_is_straighten}

    return bend_states, straighten_states

🦄4.3.2 定义 OK 手势判断函数

OK 手势判断流程如图：

def judge_OK(all_points, bend_states, straighten_states):

    angle5_6_and_6_8 = compute_angle(all_points['point5'][0], all_points['point5'][1], all_points['point6'][0], all_points['point6'][1],
                                    all_points['point6'][0], all_points['point6'][1], all_points['point8'][0], all_points['point8'][1])

    if angle5_6_and_6_8 > 0.1 * math.pi and straighten_states['third'] and straighten_states['fourth'] and straighten_states['fifth']:

        distance4_and_8 = points_distance(all_points['point4'][0], all_points['point4'][1], all_points['point8'][0], all_points['point8'][1])
        distance2_and_6 = points_distance(all_points['point2'][0], all_points['point2'][1], all_points['point6'][0], all_points['point6'][1])
        distance4_and_6 = points_distance(all_points['point4'][0], all_points['point4'][1], all_points['point6'][0], all_points['point6'][1])

        if distance4_and_8 < distance2_and_6 and distance4_and_6 > distance4_and_8 and all_points['point11'][1] < all_points['point10'][1]:
            return 'OK'
        else:
            return False
    else:
        return False

🦄4.3.3 定义 Return 手势判断函数

Return 手势判断流程如图：

def judge_Return(all_points, bend_states, straighten_states):

    angle18_6_and_18_18_ = compute_angle(all_points['point18'][0], all_points['point18'][1], all_points['point6'][0], all_points['point6'][1],
                                        all_points['point18'][0], all_points['point18'][1], all_points['point18'][0] + 10, all_points['point18'][1])
    angle_6_18_and_6_6_ = compute_angle(all_points['point6'][0], all_points['point6'][1], all_points['point18'][0], all_points['point18'][1],
                                        all_points['point6'][0], all_points['point6'][1], all_points['point6'][0] + 10, all_points['point6'][1])
    angle_0_2_and_0_17 = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point2'][0], all_points['point2'][1],
                                        all_points['point0'][0], all_points['point0'][1], all_points['point17'][0], all_points['point17'][1])

    if (bend_states['first'] and bend_states['second'] and bend_states['third'] and bend_states['fourth'] and bend_states['fifth'] and
            angle_0_2_and_0_17 > 0.15 * math.pi and
            all_points['point7'][1] > all_points['point6'][1] and all_points['point11'][1] > all_points['point10'][1] and
            all_points['point15'][1] > all_points['point14'][1] and all_points['point19'][1] > all_points['point18'][1]):

        if angle18_6_and_18_18_ < 0.1 * math.pi or angle_6_18_and_6_6_ < 0.1 * math.pi:
            return 'Return'
        else:
            return False
    else:
        return False

🦄4.3.4 定义 Left 手势判断函数

Left 手势判断流程如图：

def judge_Left(all_points, bend_states, straighten_states):

    angle5_6_and_6_8 = compute_angle(all_points['point5'][0], all_points['point5'][1], all_points['point6'][0], all_points['point6'][1],
                                    all_points['point6'][0], all_points['point6'][1], all_points['point8'][0], all_points['point8'][1])
    angle9_10_and_10_12 = compute_angle(all_points['point9'][0], all_points['point9'][1], all_points['point10'][0], all_points['point10'][1],
                                        all_points['point10'][0], all_points['point10'][1], all_points['point12'][0], all_points['point12'][1])
    angle13_14_and_14_16 = compute_angle(all_points['point13'][0], all_points['point13'][1], all_points['point14'][0], all_points['point14'][1],
                                        all_points['point14'][0], all_points['point14'][1], all_points['point16'][0], all_points['point16'][1])
    angle17_18_and_18_20 = compute_angle(all_points['point17'][0], all_points['point17'][1], all_points['point18'][0], all_points['point18'][1],
                                        all_points['point18'][0], all_points['point18'][1], all_points['point20'][0], all_points['point20'][1])
    angle0_6_and_0_4 = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point6'][0], all_points['point6'][1],
                                    all_points['point0'][0], all_points['point0'][1], all_points['point4'][0], all_points['point4'][1])
    angle0_5_and_0_17 = compute_angle(all_points['point0'][0], all_points['point0'][1], all_points['point5'][0], all_points['point5'][1],
                                    all_points['point0'][0], all_points['point0'][1], all_points['point17'][0], all_points['point17'][1])

    if ((straighten_states['first'] and bend_states['second'] and bend_states['third'] and bend_states['fourth'] and bend_states['fifth']) or
        (straighten_states['first'] and angle5_6_and_6_8 > 0.2 * math.pi