旭日x3派,手势识别之Momo Quanghuang学习记录

实现内容:
USB摄像头通过旭日x3派的手势识别模块采集人体姿态,并按照协议在PC机上代替拳皇游戏中按键的输入,以实现在游戏中简单锻炼身体,并放松心情的目的。

准备工作

注: 需要提前准备好python环境

  • 1 下载 Mame模拟器
    在这里插入图片描述
    在这里插入图片描述
    点击 立即下载后,任选一下载地址即可,下载完成后需要解压,即可看到启动程序.
    在这里插入图片描述

  • 2 安装拳皇及配置相关设置
    下载部分:
    搞定了模拟器,还需要有游戏的roms才能运行,大家可以自行在网上找各种资源,这里提供的是拳皇
    在这里插入图片描述

下载之后,将下载的zip文件直接放到刚才解压出来文件夹下面的roms下面。
在这里插入图片描述

接下来,在你刚才解压出来的文件夹目录下,在地址栏输入cmd,回车。
在这里插入图片描述
在弹出的命令行里 输入:

mame -keyboardprovider win32 sfiii3nr1

即可启动游戏
配置设置:按Tab键后 选择 Input(this machine) 进行部分按键更改 .更改时,注意鼠标双击需要更改的按键或者按Enter键选择需要更改的按键 ,然后在键盘上输入更改的按键, 注意有一秒左右的延时 不要重复键入
![在这里插入图片描述](https://img-blog.csdnimg.cn/7d32ae21c0ec47de9b01ecc942f3734b.png
更改按键可参照以下 按键说明(将左边的对应更改为右边的即可):

‘COIN_P1’: ‘5’,
‘COIN_P2’: ‘6’,
‘P1_START’: ‘1’,
‘P2_START’: ‘2’,
‘P1_JPUNCH’: ‘L-CTRL’,
‘P2_JPUNCH’: ‘A’,
‘P1_RIGHT’: ‘RIGHT’,
‘P2_RIGHT’: ‘G’,
‘P1_LEFT’: ‘L’,
‘P2_LEFT’: ‘D’,
‘P1_DOWN’: ‘DOWN’,
‘P2_DOWN’: ‘F’,
‘P1_UP’: ‘UP’,
‘P2_UP’: ‘R’,
‘P1_SKICK’: ‘L-SHIFT’,
‘P2_SKICK’: ‘W’,

更改完成后按Esc退出即可
补充: 将游戏页面最小化 Ctrl+Esc 即可

  • 3 windows 电脑端启动信号接收程序
import os 
import re 
import socket 
import argparse 
import time 
from threading import Thread 
 
try: 
    import win32api 
    import win32con 
except ModuleNotFoundError: 
    os.system('pip3 install pywin32') 
    import win32api 
    import win32con 
 
key2code = {'0': 48, '1': 49, '2': 50, '3': 51, '4': 52, '5': 53, '6': 54, '7': 55, '8': 56, '9': 57, 'a': 65, 
            'b': 66, 'c': 67, 'd': 68, 'e': 69, 'f': 70, 'g': 71, 'h': 72, 'i': 73, 'j': 74, 'k': 75, 'l': 76, 
            'm': 77, 'n': 78, 'r': 82, 'o': 79, 'p': 80, 'q': 81, 's': 83, 't': 84, 'u': 85, 'v': 86, 'w': 87, 'x': 88, 
            'y': 89, 'z': 90, '/0': 96, 
            '/1': 97, '/2': 98, '/3': 99, '/4': 100, '/5': 101, '/6': 102, '/7': 103, '/8': 104, '/9': 105, '*': 106, 
            '+': 107, 'enter': 108, '-': 109, '.': 110, '/': 111, 'f1': 112, 'f2': 113, 'f3': 114, 'f4': 115, 'f5': 116, 
            'f6': 117, 'f7': 118, 'f8': 119, 'f9': 120, 'f10': 121, 'f11': 122, 'f12': 123, 'backspace': 8, 'tab': 9, 
            'clear': 12, 'l-shift': 16, 'r-shift': 16, 'l-ctrl': 17, 'r-ctrl': 17, 'alt': 18, 'capslock': 20, 'esc': 27, 
            'space': 32, 
            'pageup': 33, 'pagedown': 34, 'end': 35, 'home': 36, 'left': 37, 'up': 38, 'right': 39, 'down': 40, 
            'insert': 45, 'delete': 46, 'help': 47, 'numlock': 144} 
key2scan = {'numlock': [0x45, 0xc5], '/': [0xe035, 0xe0b5], '*': [0x09, 0x89], '-': [0x0c, 0x8c], '7': [0x08, 0x88], 
            '8': [0x09, 0x89], '9': [0x0a, 0x8a], '4': [0x05, 0x85], '5': [0x06, 0x86], '6': [0x07, 0x87], 
            '1': [0x02, 0x82], 'end': [0xe04f, 0xe0cf], '2': [0x03, 0x83], '3': [0x04, 0x84], 
            'pgdn': [0x51, 0xd1], '0': [0x0b, 0x8b], 'ins': [0x52, 0xd2], '.': [0x34, 0xb4], 'del': [0x53, 0xd3], 
            '+': [0x0d, 0x8d], 'enter': [0x1c, 0x9c], 'insert': [0xe052, 0xe0d2], 'page up': [0xe049, 0xe0c9], 
            'delete': [0xe053, 0xe0d3], 'page down': [0xe051, 0xe0d1], 'left': [0xe046, 0xe0c6], 
            'right': [0xe04d, 0xe0cd], 'up': [0xe048, 0xe0c8], 'down': [0xe050, 0xe0d0], 
            'esc': [0x01, 0x81], 'f1': [0x3b, 0xbb], 'f2': [0x3c, 0xbc], 'f3': [0x3d, 0xbd], 
            'f4': [0x3e, 0xbe], 'f5': [0x3f, 0xbf], 'f6': [0x40, 0xc0], 'f7': [0x41, 0xc1], 'f8': [0x42, 0xc2], 
            'f9': [0x43, 0xc3], 'f10': [0x44, 0xc4], 'f11': [0x57, 0xd7], 'f12': [0x58, 0xd8], '~': [0x29, 0xa9], 
            '·': [0x29, 0xa9], '<tab>': [0x0f, 0x8f], '!': [0x02, 0x82], 'q': [0x10, 0x90], '@': [0x03, 0x83], 
            'w': [0x11, 0x91], '#': [0x04, 0x84], 'e': [0x12, 0x12], '$': [0x05, 0x85], 'r': [0x13, 0x93], 
            '%': [0x06, 0x86], 't': [0x14, 0x94], '^': [0x07, 0x87], 'y': [0x15, 0x95], '&': [0x08, 0x88], 
            'u': [0x16, 0x96], 'i': [0x17, 0x97], '(': [0x0a, 0x8a], 'o': [0x18, 0x98], ')': [0x0b, 0x8b], 
            'p': [0x19, 0x99], '_': [0x0c, 0x8c], '{': [0x1a, 0x9a], '[': [0x1a, 0x9a], '=': [0x0d, 0x8d], 
            '}': [0x1b, 0x9b], ']': [0x1b, 0x9b], '|': [0x2b, 0xab], '\\': [0x2b, 0xab], 'backspace': [0x0e, 0x8e], 
            'capslock': [0x3a, 0xba], 'l-shift': [0x2a, 0xaa], 'a': [0x1e, 0x9e], 'z': [0x2c, 0xac], 
            's': [0x1f, 0x9f], 'x': [0x2d, 0xad], 'd': [0x20, 0xa0], 'c': [0x2e, 0xae], 'f': [0x21, 0xa1], 
            'v': [0x2f, 0xaf], 'g': [0x22, 0xa2], 'b': [0x30, 0xb0], 'h': [0x23, 0xa3], 'n': [0x31, 0xb1], 
            'j': [0x24, 0xa4], 'm': [0x32, 0xb2], 'k': [0x25, 0xa5], '<': [0x33, 0xb3], ',': [0x33, 0xb3], 
            'l': [0x26, 0xa6], '>': [0x34, 0xb4], ':': [0x27, 0xa7], ';': [0x27, 0xa7], '?': [0x35, 0xb5], 
            '': [0x35, 0xb5], '"': [0x28, 0xa8], '\'': [0x28, 0xa8], 'r-shift': [0x36, 0xb6], 'l-ctrl': [0x1d, 0x9d], 
            'l-alt': [0x38, 0xb8], 'space': [0x39, 0xb9], 'r-alt': [0xe038, 0xe0b8], 'r-ctrl': [0xe01d, 0xe09d], } 
 
 
class SocketKeyBoard(object): 
    def __init__(self, ip, port=8000): 
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 
        print("socket bind " + ip + ":" + str(port)) 
        self.socket.bind((ip, port)) 
        self.socket.listen(128) 
        self.client_socket, self.client_addr = None, None 
        self.auto_thread = Thread(target=self.auto_run, args=()) 
        self.auto_thread.start() 
        self.hold_key_list = {} 
 
    def auto_run(self): 
        while True: 
            time.sleep(0.01) 
            remove_key_list = [] 
            for key in self.hold_key_list.keys(): 
                if self.hold_key_list[key] < time.time(): 
                    self.press_key(key, win32con.KEYEVENTF_KEYUP) 
                    remove_key_list.append(key) 
            for key in remove_key_list: 
                self.hold_key_list.pop(key) 
 
    @staticmethod 
    def press_key(key, code=0): 
        if code == 0: 
            print("press", key, time.time()) 
        else: 
            print("release", key, time.time()) 
        if key in key2code.keys(): 
            if key2scan[key][0] & 0xff00 == 0xe000: 
                win32api.keybd_event(key2code[key], key2scan[key][0] & 0x00ff, 
                                     win32con.KEYEVENTF_EXTENDEDKEY | code, 0) 
            else: 
                win32api.keybd_event(key2code[key], key2scan[key][0], 0 | code, 0) 
        else: 
            print("get unexpected key: ", key) 
 
    def run(self): 
        total_len = 0 
        while True: 
            if self.client_socket: 
                rev_info = self.client_socket.recv(1024) 
                if not rev_info: 
                    self.client_socket = None 
                    continue 
                key_list = re.findall("\"(.*?)\"", rev_info.decode()) 
                key_list = [key.strip().lower() for key in key_list] 
                for key in key_list: 
                    total_len += 1 
                    if 'hold' in key: 
                        key_temp = key.split(':')[0] 
                        keep_time = float(re.findall("hold(.*?)s", key.split(':')[1])[0]) 
                        self.hold_key_list[key_temp] = time.time() + keep_time 
                        key = key_temp 
                    self.press_key(key) 
                time.sleep(0.01) 
                for key in key_list: 
                    if ':hold' in key: 
                        continue 
                    if key in key2code.keys(): 
                        self.press_key(key, win32con.KEYEVENTF_KEYUP) 
            else: 
                print("wait for connecting ...") 
                self.client_socket, self.client_addr = self.socket.accept() 
                print("connect success from: ", self.client_addr) 
 
 
def parse_args(): 
    description = "start web socket keyboard application" 
    parser = argparse.ArgumentParser(description=description) 
    parser.add_argument('--ip', help="the local ip number") 
    parser.add_argument('--port', help='set listen port ', default=8000) 
    args_ = parser.parse_args() 
    return args_ 
 
 
if __name__ == "__main__": 
    args = parse_args() 
    socket_keyboard = SocketKeyBoard(ip=args.ip, port=args.port) 
    socket_keyboard.run() 

先将代码在PC机 上保存为 .py 文件,然后找到文件目录,并键入cmd
在这里插入图片描述

输入如下命令

python3 your_python_file --ip your_ip

在这里插入图片描述
注意:
(1)用python3输入后没有上图所示输出,可用python代替继续尝试
( 2) IP为自己PC机的IP 地址 可以在cmd命令提示符里面 键入 ipconfig 查询
在这里插入图片描述

  • 4 旭日x3派可以运行人体检测代码
    参考旭日X3派官网box应用算法中的手势识别手势识别
    若使用usb摄像头需更改x3派中部分代码

  • 5 最后将下面的python代码保存进自己的旭日x3派中

import rclpy 
import time 
import socket 
import math 
import json 
import argparse 
from rclpy.node import Node 
from ai_msgs.msg import PerceptionTargets 
from threading import Thread 
from collections import Counter 
 
#Action to key mapping 
ACTION2KEY = { 
    'COIN_P1': '5', 
    'COIN_P2': '6', 
    'P1_START': '1', 
    'P2_START': '2', 
    'P1_JPUNCH': 'L-CTRL', 
    'P2_JPUNCH': 'A', 
    'P1_RIGHT': 'RIGHT', 
    'P2_RIGHT': 'G', 
    'P1_LEFT': 'L', 
    'P2_LEFT': 'D', 
    'P1_DOWN': 'DOWN', 
    'P2_DOWN': 'F', 
    'P1_UP': 'UP', 
    'P2_UP': 'R', 
    'P1_SKICK': 'L-SHIFT', 
    'P2_SKICK': 'W', 
} 
""" 
hand gesture: 
0: Background,  // 无手势 
1: FingerHeart,  // 比心 
2: ThumbUp,  // 大拇指向上 
3: Victory, // V手势 
4: Mute,  // 嘘 
10: Palm,  // 手掌 
11: Okay,  //  OK手势 
12: ThumbRight,  //  大拇指向右 
13: ThumbLeft,  //  大拇指向左 
14: Awesome,  //  666手势 
""" 
GESTURE_VICTORY = 3 
GESTURE_OKAY = 11 
GESTURE_AWESOME = 14 
 
class PlayerState(object): 
    """ 
    Player status 
    There are two players by default. 
    left_arm_line: state of left arm 
    right_arm_line: state of right arm 
    up_or_down: control up and down or squat jump 
    kick: kick 
    left_gesture: left hand gesture 
    right_gesture: right hand gesture 
    time_stamp: timestamp 
    """ 
 
    def __init__(self, right_arm_line=False, left_arm_line=False, move_direction=0, kick=False, right_gesture=0, 
                 left_gesture=0): 
        self.left_arm_line = left_arm_line 
        self.right_arm_line = right_arm_line 
        self.move_direction = move_direction 
        self.kick = kick 
        self.left_gesture = left_gesture 
        self.right_gesture = right_gesture 
        self.time_stamp = time.time() 
 
    def __str__(self): 
        return "\rarm " + str(self.left_arm_line) + " " + str(self.right_arm_line) + " move " + str( 
            self.move_direction) + " kick " + str(self.kick) + " gesture " + str(self.left_gesture) + " " + str( 
            self.right_gesture) + " " + str(self.time_stamp) 
 
 
class Player(object): 
    """ 
    This class defines players. 
    state_list The list stores a series of action states of players, which are obtained by matching the results of key point detection with the corresponding actions. 
    Method def get_ Action() filters the player's state_list to get the final action. 
    Filtering is to ensure the stability of output action. 
    """ 
 
    def __init__(self, name): 
        self.state_list = [] 
        self.id = 0 
        self.name = name 
 
    def get_action(self): 
        action_num = {} 
        buffer_time_step = 0.5 
        if len(self.state_list) < 15: 
            return [], {} 
        while self.state_list and time.time() - self.state_list[0].time_stamp > buffer_time_step: 
            self.state_list.pop(0) 
        if not self.state_list: 
            return [], {} 
 
        action_list = [] 
        select_tile = len(self.state_list) // int(buffer_time_step / 0.2) 
        right_arm_list = [state.right_arm_line for state in self.state_list] 
        left_arm_list = [state.left_arm_line for state in self.state_list] 
 
        if right_arm_list[-select_tile:].count(True) - select_tile // 5 > right_arm_list[ 
                                                                          -select_tile * 2:-select_tile].count( 
            True) or left_arm_list[-select_tile:].count(True) - select_tile // 5 > left_arm_list[ 
                                                                                   -select_tile * 2:-select_tile].count( 
            True): 
            action_list.append(self.name + '_JPUNCH') 
            action_num[self.name + '_JPUNCH'] = max(right_arm_list.count(True), left_arm_list.count(True)) 
 
        left_gesture_list = [state.left_gesture for state in self.state_list] 
        right_gesture_list = [state.right_gesture for state in self.state_list] 
        if left_gesture_list.count(GESTURE_VICTORY) > (len(left_gesture_list) / 3 + 1) or right_gesture_list.count( 
                GESTURE_VICTORY) > (len(right_gesture_list) / 3 + 1): 
            action_list.append('COIN_' + self.name) 
            action_num['COIN_' + self.name] = max(left_gesture_list.count(GESTURE_VICTORY), right_gesture_list.count( 
                GESTURE_VICTORY)) 
        elif left_gesture_list.count(GESTURE_OKAY) > (len(left_gesture_list) / 3 + 1) or right_gesture_list.count( 
                GESTURE_OKAY) > (len(right_gesture_list) / 3 + 1): 
            action_list.append(self.name + '_START') 
            action_num[self.name + '_START'] = max(left_gesture_list.count(GESTURE_OKAY), right_gesture_list.count( 
                GESTURE_OKAY)) 
 
        move_direction_list = [state.move_direction for state in self.state_list] 
 
        if move_direction_list.count(1) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_DOWN') 
            action_num[self.name + '_DOWN'] = move_direction_list.count(1) 
        elif move_direction_list.count(2) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_UP') 
            action_num[self.name + '_UP'] = move_direction_list.count(2) 
        elif move_direction_list.count(3) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_RIGHT') 
            action_num[self.name + '_RIGHT'] = move_direction_list.count(3) 
        elif move_direction_list.count(4) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_LEFT') 
            action_num[self.name + '_LEFT'] = move_direction_list.count(4) 
 
        kick_list = [state.kick for state in self.state_list] 
        if kick_list.count(True) > (len(kick_list) / 10 + 1): 
            action_list.append(self.name + '_SKICK') 
            action_num[self.name + '_SKICK'] = kick_list.count(True) 
        return action_list, action_num 
 
class Game(object): 
    """ 
    Instantiate players. 
    Give each player a unique ID. 
    To solve the track of key point detection_ For the problem that the player 
    cannot match after the ID changes, maintain a queue to store the ID. 
    """ 
 
    def __init__(self): 
        self.players = [Player("P1"), Player("P2")] 
        self.show_id_list = [] 
        self.action_buffer = set() 
        self.action_nums = {} 
 
    def push_state(self, hand_match_dict, body_parser_dict): 
        for track_id in body_parser_dict.keys(): 
            right_arm, left_arm, move_direction, kick = self.stat_detection(body_parser_dict[track_id]) 
            self.show_id_list.append(track_id) 
            while len(self.show_id_list) > 100: 
                self.show_id_list.pop(0) 
 
            match_idx = -1 
            for player_idx in range(2): 
                if self.players[player_idx].id == track_id: 
                    match_idx = player_idx 
 
            if match_idx == -1: 
                p0_sum = self.show_id_list.count(self.players[0].id) 
                p1_sum = self.show_id_list.count(self.players[1].id) 
                if p0_sum < 10: 
                    self.players[0].id = track_id 
                    match_idx = 0 
                elif p1_sum < 10: 
                    self.players[1].id = track_id 
                    match_idx = 1 
            if match_idx != -1: 
                right_gesture_id = 0 
                left_gesture_id = 0 
                if (track_id, 'left') in hand_match_dict: 
                    left_gesture_id = hand_match_dict[(track_id, 'left')] 
                if (track_id, 'right') in hand_match_dict: 
                    right_gesture_id = hand_match_dict[(track_id, 'right')] 
                self.players[match_idx].state_list.append( 
                    PlayerState(right_arm, left_arm, move_direction, kick, left_gesture_id, right_gesture_id)) 
        return self 
 
    @staticmethod 
    def stat_detection(points): 
 
        def is_line(a, b, c): 
            pow_len_a_b = (math.pow(a.x - b.x, 2) + math.pow(a.y - b.y, 2)) 
            pow_len_b_c = (math.pow(b.x - c.x, 2) + math.pow(b.y - c.y, 2)) 
            pow_len_a_c = (math.pow(a.x - c.x, 2) + math.pow(a.y - c.y, 2)) 
            cosC = -(pow_len_a_b + pow_len_b_c - pow_len_a_c) / (2 * math.sqrt(pow_len_a_b) * math.sqrt(pow_len_b_c)) 
            return cosC > 0.8 
 
        def is_y_similar(a, b, c): 
            x_a_c = abs(a.x - c.x) 
            if x_a_c < 0.1: 
                print("x is too small") 
                return False 
            y_a_c = abs(a.y - c.y) 
            x_y_ratio = y_a_c / x_a_c 
            return x_y_ratio < 0.4 
 
        def arm_status(): 
            is_right_line = is_line(points[10], points[8], points[6]) 
            is_right_y_similar = is_y_similar(points[10], points[8], points[6]) 
 
            is_left_line = is_line(points[9], points[7], points[5]) 
            is_left_y_similar = is_y_similar(points[9], points[7], points[5]) 
 
            return is_right_line and is_right_y_similar, is_left_line and is_left_y_similar 
 
        def move_direction(): 
            """ 
            0: stand 
            1: down 
            2: up 
            3: left 
            4: right 
 
            :return: 
            """ 
            # down 
            if abs(points[16].y - points[14].y) < abs(points[14].y - points[12].y) / 2 or abs( 
                    points[15].y - points[13].y) < abs(points[13].y - points[11].y) / 2: 
                return 1 
            # up 
            if points[8].y < points[0].y or points[7].y < points[0].y: 
                return 2 
 
            middle_x = (points[5].x + points[6].x + points[12].x + points[11].x) / 4 
            middle_y = (points[5].y + points[6].y + points[12].y + points[11].y) / 4 
            # right 
            if points[10].x < middle_x and points[9].x < middle_x and points[9].y > middle_y: 
                return 3 
            # left 
            if points[10].x > middle_x and points[9].x > middle_x and points[10].y > middle_y: 
                return 4 
            return 0 
 
        def kick(): 
            return (abs(points[16].y - points[13].y) < abs(points[15].y - points[14].y) and 
                    points[14].y < points[12].y) or abs(points[15].y - points[14].y) < abs( 
                points[15].y - points[13].y) and points[13].y < points[11].y 
 
        right_arm_line_, left_arm_line_ = arm_status() 
        move_direction_ = move_direction() 
        kick_ = kick() 
 
        return right_arm_line_, left_arm_line_, move_direction_, kick_ 
 
    def refresh_action(self): 
        for player in self.players: 
            action_list, action_num = player.get_action() 
            for action in action_list: 
                self.action_buffer.add(action) 
            self.action_nums = dict(Counter(self.action_nums) + Counter(action_num)) 
        return self 
 
    def get_action(self): 
        return self.action_buffer, self.action_nums 
 
    def clear_action(self): 
        self.action_buffer.clear() 
        self.action_nums = {} 
 
 
class socketer(object): 
    def __init__(self,ip,port): 
        self.ip = ip 
        self.port = port 
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
        self.connect() 
 
    def connect(self): 
        while True: 
            try: 
                self.socket.close() 
                self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
                self.socket.connect((self.ip, self.port)) 
                print('connect success') 
                return 
            except socket.error as e: 
                print('connect failed', e) 
                time.sleep(1) 
 
    def send(self, data): 
        try: 
            data = json.dumps(data) 
            print(data) 
            self.socket.send(bytes(data.encode('utf-8'))) 
        except socket.error: 
            self.connect() 
        except: 
            assert 0 
 
 
class MinimalSubscriber(Node): 
 
    def __init__(self, args): 
        super().__init__('minimal_subscriber') 
        self.subscription = self.create_subscription( 
            PerceptionTargets, 
            '/hobot_hand_gesture_detection', 
            self.listener_callback, 
            10) 
        self.subscription  # prevent unused variable warning 
        self.socketer = socketer(ip=args.ip, port=args.port) 
        self.game = Game() 
        self.auto_thread = Thread(target=self.auto_step, args=()) 
        self.auto_thread.start() 
 
    def auto_step(self): 
        need_hold_list = ['P1_RIGHT', 'P2_RIGHT', 'P1_LEFT', 'P2_LEFT',  'P1_DOWN', 'P2_DOWN'] #'P1_UP', 'P2_UP', 
        old_action_list=[] 
        old_action_num = {} 
        while True: 
            time.sleep(0.1) 
            action_list, action_num = self.game.get_action() 
            for act in old_action_list: 
                if act in action_list: 
                    action_list.remove(act) 
            old_action_list = action_list.copy() 
            key_list = [] 
            for act in action_list.copy(): 
                key_ = ACTION2KEY[act] 
                print(act, action_num) 
                if act in need_hold_list and act in old_action_num.keys() and (action_num[act] >= old_action_num[act]-20 and action_num[act] > 10): 
                    key_+=':hold0.3s' 
                key_list.append(key_) 
            if key_list: 
                self.socketer.send(key_list) 
            old_action_num = action_num.copy() 
            self.game.clear_action() 
 
    def hand_match(self, hand_parser_list, body_parser_dict): 
        def distance(x1, y1, x2, y2): 
            return math.sqrt(math.pow(x1 - x2, 2) + math.pow(y1 - y2, 2)) 
 
        hand_match_dict = {} 
        for hand_info in hand_parser_list: 
            min_distance = 10000 
            match_flag = (-1, 'left') 
            for track_id in body_parser_dict.keys(): 
                new_distance = distance(body_parser_dict[track_id][18].x, body_parser_dict[track_id][18].y, 
                                        hand_info[1][0], hand_info[1][1]) 
                if hand_info[1][1] > body_parser_dict[track_id][0].y: 
                    continue 
                if new_distance < min_distance and new_distance < 40: 
                    min_distance = new_distance 
                    match_flag = (track_id, 'right') 
                new_distance = distance(body_parser_dict[track_id][17].x, body_parser_dict[track_id][17].y, 
                                        hand_info[1][0], hand_info[1][1]) 
                if new_distance < min_distance and new_distance < 40: 
                    min_distance = new_distance 
                    match_flag = (track_id, 'left') 
            if match_flag[0] != -1: 
                hand_match_dict[match_flag] = hand_info[0] 
        return hand_match_dict 
 
    def get_body_hand_info(self, msg): 
        body_parser_dict = {} 
        hand_parser_list = [] 
        for targeter in msg.targets: 
            track_id = targeter.track_id 
            if len(targeter.points) == 0: 
                continue 
            point_object = targeter.points[0] 
            if point_object.type == 'body_kps': 
                body_points = point_object.point 
                body_points_confidence = point_object.confidence 
 
                if sum(body_points_confidence)/len(body_points_confidence) < 0.7: 
                    break 
                body_parser_dict[track_id] = body_points 
            if point_object.type == 'hand_kps': 
                average_x = 0.0 
                average_y = 0.0 
                hand_points = point_object.point 
                for p in hand_points: 
                    average_x += p.x 
                    average_y += p.y 
                average_x /= len(hand_points) 
                average_y /= len(hand_points) 
                if len(targeter.attributes) > 0 and targeter.attributes[0].type == 'gesture': 
                    gesture_id = targeter.attributes[0].value 
                    hand_parser_list.append((gesture_id, (average_x, average_y))) 
        hand_match_dict = self.hand_match(hand_parser_list, body_parser_dict) 
        return hand_match_dict, body_parser_dict 
 
    def listener_callback(self, msg): 
 
        # By monitoring the detected key points, the status information of the body and hands can be obtained. 
        #fps = msg.fps 
        #print(fps) 
        hand_match_dict, body_parser_dict = self.get_body_hand_info(msg) 
        self.game.push_state(hand_match_dict, body_parser_dict).refresh_action() 
 
        return None 
 
 
def main(args): 
    rclpy.init() 
    minimal_subscriber = MinimalSubscriber(args) 
    rclpy.spin(minimal_subscriber) 
 
    # Destroy the node explicitly 
    # (optional - otherwise it will be done automatically 
    # when the garbage collector destroys the node object) 
    minimal_subscriber.destroy_node() 
    rclpy.shutdown() 
 
def parse_args(): 
    description = "start web socket keyboard application" 
    parser = argparse.ArgumentParser(description=description) 
    parser.add_argument('--ip', help="the local ip number") 
    parser.add_argument('--port', help='set listen port ', default=8000) 
    args_ = parser.parse_args() 
    return args_ 
 
if __name__ == '__main__': 
    args = parse_args() 
    main(args) 

使用说明

  • 1 在windows 端 进入文件保存位置 输入cmd 运行准备工作3中的代码文件(Ip 地址为PC机);

python3 your_python_file --ip your_ip

  • 2 启动下载的游戏(启动之前,一定要先设置输入法为英文模式),启动代码如下,启动方式如之前介绍;

mame -keyboardprovider win32 sfiii3nr1

  • 3 在x3派上,先启动手势检测的示例,然后再新开一个终端,刷新化境变量

source /opt/tros/setup.bash

接着在终端运行保存进旭日x3派章节的python脚本(注意!!IP地址与第一步地址一样)。

python3 your_python_file --ip your_remote_computer_ip

成功后板端有如下输出
在这里插入图片描述
PC端分别如下所示
在这里插入图片描述

注:使用摄像头时,要保证摄像头能够采集全身信息,人体距离摄像头4~5米。

手势说明

投币:
在这里插入图片描述

开始(start):
在这里插入图片描述

上 or 跳:

在这里插入图片描述
在这里插入图片描述

下 or 蹲:
在这里插入图片描述

踢腿:

在这里插入图片描述

出拳:
在这里插入图片描述

左右:

在这里插入图片描述
在掌握手势说明后,就可以开始你沉浸式的体验啦!!

文章参考自
在这里插入图片描述
原文章地址 https://developer.horizon.ai/forumDetail/112555512834430487

  • 2
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值