一种基于OpenPose和DNN的跌倒检测软件系统(中文&英文)

A Fall Detection Software System Based on OpenPose and DNN

Falling may be nothing to the physically healthy people whereas it will cause some groups like the elderly not to stand up hereafter, or even to lose consciousness. If they are not rescued in time, they may miss the best chance of being rescued. Some study shows that falling is the second leading cause of unintentional injury death all over the world, with an estimated 684,000 deaths worldwide each year and with the most fatal falls among those over 60 years.

 One year before, my grandmother fell down at home and fractured her ribs and vertebrae due to the slippery floor. Her suffering made a dramatic impact on me. I realized at that moment that a fall detection device was necessary for the elderly. When an elderly person falls down, the device will send an alarm for help and then the person will get first aid.

Since then, I began to be immersed in fall alarm devices. After collecting a lot of the related materials , I found out that the structure of an alarm device is designed as follows.

The camera  takes videos uninterruptedly ; then the photos of the human bodies will be captured, input into the software system and the calculation or analysis on the human body’s movements will be done in this system. Once the movement is recognized as  "fall down", a real-time signal will be sent to the mobile terminal of the guardians, or sent directly to "911". Alternatively, an alarm bell will be rung in this device. Finally, the receiver can make an immediate medical rescue.

the Software System

Generally the software system for recognizing the human body’s movements is an important part of the whole device. After reviewing, referring to many materials and studying, I designed a fall detection software system based on Openpose and DNN. Openpose captures the key points of the human skeleton from the camera frame by frame, uses tensorflow to link these key points to form actions, and recognise them to determine whether a fall will occur or not. If yes, a real-time alarm is sent.

The flow chart is as follows:

Capture of the  Key Points of the Human Skeleton

There are many methods for human pose estimation, such as Mediapipe and OpenPose. I designed the software system based on OpenPose, an open source library developed by Carnegie Mellon University, because Openpose can detect several people and its accuracy is much higher than Mediapipe. Meanwhile Openpose can detect the position of the key points of the human skeleton, and form a model of the human skeleton by linking the key points of the human skeleton.

 

 output of 25 key points of human skeleton by Openpose

Recognition and Determination of Actions

The project based on DNN (Deep Neural Networks) takes the coordinate value of the key points of human skeleton from each frame generated by Openpose as input to form the movements of human skeleton, and uses the movements as the label for training to obtain the correspondence between the key points of human skeleton and the movements (standing, walking, working, falling), so as to determine whether the person falls down or not.

Software Programming

Setup of an environment

Link: 

https://github.com/LZQthePlane/Online-Realtime-Action-Recognition-based-on-OpenPose

Environmental Requirements:

python >= 3.5

Opencv >= 3.4.1

sklearn

tensorflow & keras

numpy & scipy

the Actual Use Environment :

Python==3.6.5

Opencv==3.4.2

Sklearn==0.19.2

Tensorflow==2.1.0

Keras==2.3.1

Scipy==1.5.3

Anaconda, a release version of Python, is used to manage the Python environment and the various Python install packages. Set up and configure a new python environment by Anaconda; install the various modules, libraries and packages required for Openpose and DNN; set up the input, output and interface of each library and package; and run the code to complete the test.

Rules for Determining “Fall Down” Actions

  1) Decision-making Condition 1 :Descending velocity at the center of the hip joint 

During a sudden fall, the center of gravity of the human body rapidly changes in the vertical direction. The center point of the human hip joint can represent the center of gravity of the human body. Since the time interval from standing to falling down is very short, it is necessary to detect every five adjacent frames with an interval of 0.25 s. A fall is determined if the hip joint goes from normal height to zero.

Decision-making Calculation Formula: 

2) Decision-making Condition 2: The angle between the centerline of the human body and the ground

During a fall, the most obvious feature of the human body is the tilt of the body.  The inclination increases with a fall. In order to reflect the characteristics of the continuous inclination of the human body during the fall, the midpoint of the joint points S10 and S13 is s. The line connecting s and s0 is the centerline, and θ is the angle between the centerline of the human body and the ground. An angle less than 45 ° is determined as a fall.

Decision-making Calculation Formula:

 

3) Decision-making Condition 3: The aspect ratio of the outer contour of the human body

When the movement of the target is detected to change, such as a fall, the ratio of the detection window will change accordingly. When the target only changes in width and height due to its distance from the camera, the aspect ratio of the detection frame does not change. Let the aspect ratio be P. When P ≥ 1, a fall is determined.

Decision-making Calculation Formula:

 

Presentation 

The Project can realize two detection modes: online real-time detection （ webcam) and offline detection (video). The former will be the default option.

Detection Examples

Side View:

Front View:

Recap

This project has successfully implemented fall detection with the help of Openpose and DNN. However, the recognition accuracy of this model is not good enough in dimly lit environments, and so  improving the accuracy of the model in different environments is further studied. In this project only one person can be detected and I will try to explore the experiment with more people in the future.

一种基于 OpenPose和DNN 的跌倒检测软件系统

       跌倒对普通人而言，不是什么大事， 然而对于一些特殊人群，例如老人，跌倒后可能就无法站立起来，严重者甚至失去意识。如果他们没有被及时发现，就会错过最佳救治时机。有证据表明，跌倒是全球意外伤害死亡的第二大原因，全球每年估计684000人死于跌倒，并且60岁以上的老年人跌倒致命次数最多。

       一年多前，因为地面湿滑，我姥姥在家摔了一跤，肋骨和脊椎骨骨折。因为没有及时发现，她受了很多罪。这件事对我的震动很大，我意识到，如果家里有一台自主检测跌倒的的设备，老人在摔倒的时候, 设备会发出报警，老人就可以得到及时救治，从而避免造成严重的后果。

      所以，我开始关注跌倒报警设备。翻阅资料后，设计报警设备的结构如下：

摄像机持续拍摄，将镜头中拍摄到的人的一系列照片输入软件系统，经过软件系统的计算分析，识别出人的动作，并对这些动作进行判定，一旦判定为“跌倒“，则根据需要，实时发出信号。可以发”跌倒“信号给设定的监护人的移动终端，也可以直接发给”120“，当然也可以发出报警铃声，或者其它合适的方式。这样，接收到报警信号的一方就可以及时安排救治。

软件系统

       我们可以看到，对人体动作进行识别的软件系统是整个装置的重要部分，在查阅、参考大量资料后，我设计了基于Openpose和DNN的跌倒检测软件系统。通过Openpose抓取来自于摄像机的一帧帧照片中的人体骨骼关键点，利用tensorflow来将这些关键点联系起来形成动作，并对这些动作进行识别，判定是否跌倒。一旦判定为跌倒，则实时发出报警信号。

骨骼关键点的抓取

       现有很多人体姿态估计的方法，如Mediapipe，OpenPose。我们选用卡耐基梅隆大学团队开发的开源项目OpenPose，因为Openpose可以对多人进行检测，其检测精度远高于Mediapipe。Openpose可以检测人体骨骼关键点的位置，将人体关键点连接起来，形成人体骨架模型。

Openpose输出的25个骨骼关键点

动作的识别、判断

       本项目基于DNN(Deep Neural Networks), 把来自于Openpose生成的各帧中人体骨骼关键点的坐标值作为输入，形成人体骨架动作，以动作作为标签进行训练从而得到人体关键点与动作（站立、行走、工作、跌倒）的对应关系，以此判断被测对象是否跌倒。

软件编程实现

环境搭建: 

项目链接：   

https://github.com/LZQthePlane/Online-Realtime-Action-Recognition-based-on-OpenPose

项目环境要求：

python >= 3.5

Opencv >= 3.4.1

sklearn

tensorflow & keras

numpy & scipy

经测试，实际使用环境：

Python==3.6.5

Opencv==3.4.2

Sklearn==0.19.2

Tensorflow==2.1.0

Keras==2.3.1

Scipy==1.5.3

        采用python编程，利用 Python 的一个发行版本Anaconda 来管理Python环境以及Python的各种安装包。通过Anaconda 创建、配置一个新的python环境，然后安装Openpose和DNN所需的各种模块、库和包，设置各库、包的输入、输出及接口，运行代码后即可完成检测。

人工制定判断跌倒的规则

    1) 决策条件一：髋关节中心处的下降速度 

    在突然跌倒的过程中，人体的重心将迅速在垂直方向上改变。人体髋关节的中心 点可以代表人体的重心。因为从站立到跌倒用时很短，所以需要每隔五个相邻的帧就检测一次， 间隔为0.25 s, 如果髋关节从正常高度到零则判断为跌倒。

  决策计算公式：

2) 决策条件二： 人的中心线与地面的夹角

    在跌倒过程中，人体最明显的特征是身体倾斜。倾斜度会随着跌倒不断增加。为 了反映人体在跌倒过程中连续倾斜的特征，关节点S10和S13 的中点是 s ， s 和s0 的 连线是中心线，θ 是人的中心线与地面之间的角度（如下图所示），当θ小于45°时判断为跌倒。

决策计算公式：

3) 决策条件三：人体外部轮廓矩形的宽高比

   当检测到目标的动作改变, 例如跌倒时，检测框的宽高比会随之发生改变。当目标只是因距离相机的远近发生宽度和高度变化时，检测框的宽高比不会改变。设宽高比为P，当 P≥1，判断为跌倒。

决策计算公式： 

成果展示

 本项目可以实现在线实时检测（webcam）与离线检测（video）两种检测模式，默认为实时检测。

检测实例（本例为离线检测）。

侧面图：

 

正面图：

心得与展望

      本项目成功利用Openpose和DNN实现了跌倒检测。但在昏暗的环境下，本模型的识别精度还不够好，因此可以考虑提升模型在不同环境下的准确率。本项目只是对一人进行跌倒检测，以后还可以尝试对多人进行检测。