SORT 代码阅读笔记

最新推荐文章于 2023-03-14 22:15:39 发布

Tianchao龙虾

最新推荐文章于 2023-03-14 22:15:39 发布

阅读量1.1k

点赞数 4

分类专栏：论文代码阅读笔记文章标签：人工智能神经网络目标跟踪

本文链接：https://blog.csdn.net/wuchaohuo724/article/details/118751563

版权

论文代码阅读笔记专栏收录该内容

10 篇文章 0 订阅

订阅专栏

SORT 代码阅读笔记

代码链接:https://github.com/abewley/sort

论文阅读笔记可以查看这篇博客:https://blog.csdn.net/wuchaohuo724/article/details/118771154

先创建个实例

self.tracker = Sort.Sort(max_age=max_age, min_hits=min_hits)

这里有两个参数，max_age表示在多少帧中没有检测，trackers就会中止。min_hits代表持续多少帧检测到，生成trackers。

检测器每来一帧数据，进行检测，返回检测结果detections= $x_1,y_1,x_2,y_2,score], [....]]$ ，把这个检测结果放入SORT 追踪器上:

self.traker.update(detections)

具体来看一下update函数，输入的是检测器检测的bounding boxes结果，用[x1,y1,x2,y2,score]表示，且输出的是 [x1,y1,x2,y2,ID]。大体的流程图如下:
在这里插入图片描述

使用了Kalman Filter进行预测和Hungarian Algorithm进行检测和预测的匹配。 现在我们用一个例子来继续探讨，我们假设有两个bounding boxes输入到update函数。

self.traker.update([x1,y1,x2,y2,score],[x1,y1,x2,y2,score])


def update(self,dets): # 输入的是检测结果[x1,y1,x2,y2,score]形式
  #首先进行帧计数
  self.frame_count += 1

  #根据当前所有卡尔曼跟踪器的个数创建二维零矩阵。维度为：卡尔曼跟踪器ID个数x 5 (这5列内容为bbox与ID) 但是目前一开始没有跟踪器，所以 len(self.trackers) 为0
  trks = np.zeros((len(self.trackers),5))
  to_del = [] # 存放待删除
  ret = []    # 存放最后返回的结果

  #循环遍历跟踪器
  for t,trk in enumerate(trks): # 循环遍历卡尔曼跟踪器列表
    # predict() 返回两个参数， x 和 p。
    # x :Prior state estimate vector
    # p :Prior covariance matrix
    pos = self.trackers[t].predict()[0] # 用卡尔曼跟踪器t预测对应物体在当前帧中的bbox
    trk[:] = [pos[0], pos[1], pos[2], pos[3], 0] #跟踪器的值更新

    if(np.any(np.isnan(pos))): # 如果预测的bbox为空，那么将第t个卡尔曼跟踪器删除
      to_del.append(t) # 从跟踪器中删除 to_del中的上一帧跟踪器ID

  # 将预测为空的卡尔曼跟踪器所在行删除，最后trks中存放的是上一帧中被跟踪的所有物体在当前帧中预测的非空bbox
  trks = np.ma.compress_rows(np.ma.masked_invalid(trks)) 

  for t in reversed(to_del): # 对to_del数组进行倒序遍历
      self.trackers.pop(t) # 从跟踪器中删除 to_del中的上一帧跟踪器ID


  # 对传入的检测结果与上一帧跟踪物体在当前帧中预测的结果做关联，
  #返回匹配的目标矩阵matched, 新增目标的矩阵unmatched_dets, 离开画面的目标矩阵unmatched_trks
  matched, unmatched_dets, unmatched_trks = associate_detections_to_trackers(dets,trks) 


  # update matched trackers with assigned detections
  for t,trk in enumerate(self.trackers): # 对卡尔曼跟踪器做遍历
    if(t not in unmatched_trks):  # 如果上一帧中的t还在当前帧画面中（即不在当前预测的离开画面的矩阵unmatched_trks中）
      d = matched[np.where(matched[:,1]==t)[0],0] # 说明卡尔曼跟踪器t是关联成功的，在matched矩阵中找到与其关联的检测器d
      trk.update(dets[d,:][0]) # 用关联的检测结果d来更新卡尔曼跟踪器（即用后验来更新先验）

  #create and initialise new trackers for unmatched detections # 对于新增的未匹配的检测结果，创建并初始化跟踪器
  for i in unmatched_dets: # 新增目标
    trk = KalmanBoxTracker(dets[i,:])  # 将新增的未匹配的检测结果dets[i,:]传入KalmanBoxTracker
    self.trackers.append(trk) # 将新创建和初始化的跟踪器trk 传入trackers
  i = len(self.trackers) 
  for trk in reversed(self.trackers): # 对新的卡尔曼跟踪器集进行倒序遍历
    d = trk.get_state()[0]  # 获取trk跟踪器的状态 [x1,y1,x2,y2] 
    if((trk.time_since_update < 1) and (trk.hit_streak >= self.min_hits or self.frame_count <= self.min_hits)):
      ret.append(np.concatenate((d,[trk.id+1])).reshape(1,-1)) # +1 as MOT benchmark requires positive
    i -= 1
    #remove dead tracklet
    if(trk.time_since_update > self.max_age):
      self.trackers.pop(i)
  if(len(ret)>0):
    return np.concatenate(ret)
  return np.empty((0,5))

sort.update函数包含了一个函数和一个类，associate_detections_to_trackers 和 KalmanBoxTracker。 下面分别来讲讲这两个。

一、 KalmanBoxTracker

这个类主要是kalman filter的实现，里面包括kalman filter的两个步骤:预测和更新，也包括bounding box的状态函数。kalman filer的具体原理可以参考这篇博客。下面给出Kalman filter的步骤:

在这里插入图片描述

源码采用了filterpy中的KalmanFilter，我们打开源码来看一下，Kalman Filter是如何进行预测和更新的。

1. Kalman Filter 预测

先来看一下 Kalman Filter的预测 :

def predict(self, u=None, B=None, F=None, Q=None):
        """
        Predict next state (prior) using the Kalman filter state propagation
        equations.

        Parameters
        ----------

        u : np.array
            Optional control vector. If not `None`, it is multiplied by B
            to create the control input into the system.

        B : np.array(dim_x, dim_z), or None
            Optional control transition matrix; a value of None
            will cause the filter to use `self.B`.

        F : np.array(dim_x, dim_x), or None
            Optional state transition matrix; a value of None
            will cause the filter to use `self.F`.

        Q : np.array(dim_x, dim_x), scalar, or None
            Optional process noise matrix; a value of None will cause the
            filter to use `self.Q`.
        """

        if B is None:
            B = self.B
        if F is None:
            F = self.F
        if Q is None:
            Q = self.Q
        elif isscalar(Q):
            Q = eye(self.dim_x) * Q

        # x = Fx + Bu
        if B is not None and u is not None:
            self.x = dot(F, self.x) + dot(B, u)
        else:
            self.x = dot(F, self.x)

        # P = FPF' + Q
        self.P = self._alpha_sq * dot(dot(F, self.P), F.T) + Q

        # save prior
        self.x_prior = self.x.copy()
        self.P_prior = self.P.copy()

可以看到上面的代码对应上面预测步骤:

self.x = dot(F, self.x)
self.P = self._alpha_sq * dot(dot(F, self.P), F.T) + Q

2. Kalman Filter 更新

再来看一下 Kalman Filter的更新 :

def update(self, z, R=None, H=None):
        """
        Add a new measurement (z) to the Kalman filter.

        If z is None, nothing is computed. However, x_post and P_post are
        updated with the prior (x_prior, P_prior), and self.z is set to None.

        Parameters
        ----------
        z : (dim_z, 1): array_like
            measurement for this update. z can be a scalar if dim_z is 1,
            otherwise it must be convertible to a column vector.

        R : np.array, scalar, or None
            Optionally provide R to override the measurement noise for this
            one call, otherwise  self.R will be used.

        H : np.array, or None
            Optionally provide H to override the measurement function for this
            one call, otherwise self.H will be used.
        """

        # set to None to force recompute
        self._log_likelihood = None
        self._likelihood = None
        self._mahalanobis = None

        if z is None:
            self.z = np.array([[None]*self.dim_z]).T
            self.x_post = self.x.copy()
            self.P_post = self.P.copy()
            self.y = zeros((self.dim_z, 1))
            return

        z = reshape_z(z, self.dim_z, self.x.ndim)

        if R is None:
            R = self.R
        elif isscalar(R):
            R = eye(self.dim_z) * R

        if H is None:
            H = self.H

        # y = z - Hx
        # error (residual) between measurement and prediction
        self.y = z - dot(H, self.x)

        # common subexpression for speed
        PHT = dot(self.P, H.T)

        # S = HPH' + R
        # project system uncertainty into measurement space
        self.S = dot(H, PHT) + R
        self.SI = self.inv(self.S)
        # K = PH'inv(S)
        # map system uncertainty into kalman gain
        self.K = dot(PHT, self.SI)

        # x = x + Ky
        # predict new x with residual scaled by the kalman gain
        self.x = self.x + dot(self.K, self.y)

        # P = (I-KH)P(I-KH)' + KRK'
        # This is more numerically stable
        # and works for non-optimal K vs the equation
        # P = (I-KH)P usually seen in the literature.

        I_KH = self._I - dot(self.K, H)
        self.P = dot(dot(I_KH, self.P), I_KH.T) + dot(dot(self.K, R), self.K.T)

        # save measurement and posterior state
        self.z = deepcopy(z)
        self.x_post = self.x.copy()
        self.P_post = self.P.copy()

先计算Kalman增益:

self.S = dot(H, PHT) + R
self.SI = self.inv(self.S)
# K = PH'inv(S)
# map system uncertainty into kalman gain
self.K = dot(PHT, self.SI)

然后更新后验概率分布:

self.x = self.x + dot(self.K, self.y)

# P = (I-KH)P(I-KH)' + KRK'
# This is more numerically stable
# and works for non-optimal K vs the equation
# P = (I-KH)P usually seen in the literature.

I_KH = self._I - dot(self.K, H)
self.P = dot(dot(I_KH, self.P), I_KH.T) + dot(dot(self.K, R), self.K.T)

值得注意的是，filterpy源码没有使用 $P = (I-KC_k)P$ ，而是使用了 $P = (I-KC_k)P(I-KC_k)' + KRK'$ 。

3. KalmanBoxTracker 应用

对kalman实现之后，来看看KalmanBoxTracker


class KalmanBoxTracker(object):
  """
  This class represents the internel state of individual tracked objects observed as bbox.
  """
  count = 0
  def __init__(self,bbox):
    """
    Initialises a tracker using initial bounding box.
    """
    #define constant velocity model
    self.kf = KalmanFilter(dim_x=7, dim_z=4) # state vector with dim_x， measurement vector with dim_z
    # F--> State Transition matrix
    self.kf.F = np.array([[1,0,0,0,1,0,0],[0,1,0,0,0,1,0],[0,0,1,0,0,0,1],[0,0,0,1,0,0,0],  [0,0,0,0,1,0,0],[0,0,0,0,0,1,0],[0,0,0,0,0,0,1]])
    # H --> Measurement function --> numpy.array(dim_z, dim_x)
    self.kf.H = np.array([[1,0,0,0,0,0,0],[0,1,0,0,0,0,0],[0,0,1,0,0,0,0],[0,0,0,1,0,0,0]])
    # R --> Measurement noise matrix -->numpy.array(dim_z, dim_z)
    self.kf.R[2:,2:] *= 10.
    # P --> Current state covariance matrix --> numpy.array(dim_x, dim_x) Any call to update() or predict() updates this variable.
    self.kf.P[4:,4:] *= 1000. #give high uncertainty to the unobservable initial velocities
    self.kf.P *= 10.
    # Q --> Process noise matrix --> numpy.array(dim_x, dim_x)
    self.kf.Q[-1,-1] *= 0.01
    self.kf.Q[4:,4:] *= 0.01
    # x --> Current state estimate.--> numpy.array(dim_x, 1) Any call to update() or predict() updates this variable.
    self.kf.x[:4] = convert_bbox_to_z(bbox)
    
    self.time_since_update = 0
    self.id = KalmanBoxTracker.count
    KalmanBoxTracker.count += 1
    self.history = []
    self.hits = 0
    self.hit_streak = 0
    self.age = 0

  def update(self,bbox):
    """
    Updates the state vector with observed bbox.
    """
    self.time_since_update = 0
    self.history = []
    self.hits += 1
    self.hit_streak += 1
    self.kf.update(convert_bbox_to_z(bbox))

  def predict(self):
    """
    Advances the state vector and returns the predicted bounding box estimate.
    """
    if((self.kf.x[6]+self.kf.x[2])<=0):
      self.kf.x[6] *= 0.0
    self.kf.predict()
    self.age += 1
    if(self.time_since_update>0):
      self.hit_streak = 0
    self.time_since_update += 1
    self.history.append(convert_x_to_bbox(self.kf.x))
    return self.history[-1]

  def get_state(self):
    """
    Returns the current bounding box estimate.
    """
    return convert_x_to_bbox(self.kf.x)

论文中给出了每一个目标的状态定义为:
$x=[u,v,s,r,\dot{u}, \dot{v}, \dot{s}]^T$

所以初始化先定义了卡尔曼模型:

# state vector with dim_x， measurement vector with dim_z
self.kf = KalmanFilter(dim_x=7, dim_z=4)

上面式子中的 $A_k$ 为状态平移矩阵，源码定义为:

#F--> State Transition matrix
self.kf.F = np.array([[1,0,0,0,1,0,0],[0,1,0,0,0,1,0],[0,0,1,0,0,0,1],[0,0,0,1,0,0,0],  [0,0,0,0,1,0,0],[0,0,0,0,0,1,0],[0,0,0,0,0,0,1]])

为何矩阵是这个数值，可以参考[1]。

接下来也定义了测量矩阵 $C_k$ ，源码定义为:

# H --> Measurement function --> numpy.array(dim_z, dim_x)
self.kf.H = np.array([[1,0,0,0,0,0,0],[0,1,0,0,0,0,0],[0,0,1,0,0,0,0],[0,0,0,1,0,0,0]])

测量噪声矩阵 $Q$ ，源码定义为:

# R --> Measurement noise matrix -->numpy.array(dim_z, dim_z)
self.kf.R[2:,2:] *= 10.

状态噪声矩阵 $R$ ，源码定义为:

# Q --> Process noise matrix --> numpy.array(dim_x, dim_x)
self.kf.Q[-1,-1] *= 0.01
self.kf.Q[4:,4:] *= 0.01

二、associate_detections_to_trackers

预测出来的bounding boxes和检测出来的，需要进行关联匹配。源码中，把每个detection和每个prediction进行了IOU计算，并存储在IOU矩阵。输入进linear_assignment函数。这个函数是开源库sklearn中的函数。匈牙利算法是二分图匹配问题，也就是这里是detection 和 prediction的一个匹配。


def associate_detections_to_trackers(detections,trackers,iou_threshold = 0.3): #用于将检测与跟踪进行关联
  """
  Assigns detections to tracked object (both represented as bounding boxes)

  Returns 3 lists of matches, unmatched_detections and unmatched_trackers
  """
  if(len(trackers)==0): # 如果跟踪器为空
    return np.empty((0,2),dtype=int), np.arange(len(detections)), np.empty((0,5),dtype=int)
  iou_matrix = np.zeros((len(detections),len(trackers)),dtype=np.float32) # 检测器与跟踪器IoU矩阵

  for d,det in enumerate(detections):
    for t,trk in enumerate(trackers):
      iou_matrix[d,t] = iou(det,trk) # 计算检测器与跟踪器的IOU并赋值给IoU矩阵对应位置

  # 加上负号是因为linear_assignment求的是最小代价组合，而我们需要的是IOU最大的组合方式，所以取负号
  matched_indices = linear_assignment(-iou_matrix)  # Hungarian Algorithm

  unmatched_detections = [] # 未匹配上的检测器
  for d,det in enumerate(detections):
    if(d not in matched_indices[:,0]): # 如果检测器中第d个检测结果不在匹配结果索引中，则d未匹配上
      unmatched_detections.append(d)
  unmatched_trackers = []
  for t,trk in enumerate(trackers): # 未匹配上的跟踪器
    if(t not in matched_indices[:,1]): #  #如果跟踪器中第t个跟踪结果不在匹配结果索引中，则t未匹配上
      unmatched_trackers.append(t)

  #filter out matched with low IOU 过滤掉那些IOU较小的匹配对
  matches = [] # 存放过滤后的匹配结果
  for m in matched_indices: # 遍历粗匹配结果
    if(iou_matrix[m[0],m[1]]<iou_threshold): # m[0]是检测器ID， m[1]是跟踪器ID，如它们的IOU小于阈值则将它们视为未匹配成功
      unmatched_detections.append(m[0])
      unmatched_trackers.append(m[1])
    else:
      matches.append(m.reshape(1,2)) # 将过滤后的匹配对维度变形成1x2形式
  if(len(matches)==0): # 如果过滤后匹配结果为空，那么返回空的匹配结果
    matches = np.empty((0,2),dtype=int) 
  else: # 如果过滤后匹配结果非空，则按0轴方向继续添加匹配对
    matches = np.concatenate(matches,axis=0)

  return matches, np.array(unmatched_detections), np.array(unmatched_trackers) #其中跟踪器数组是5列的（最后一列是ID）

三、np.ma.masked_invalid

np.ma.masked_invalid(a, copy=True) 用来Mask an array where invalid values occur (NaNs or infs).

import numpy.ma as ma

a = np.arange(5, dtype=float)

a[2] = np.NaN

a[3] = np.PINF

a
array([ 0.,  1., nan, inf,  4.])

masked_array = ma.masked_invalid(a)
masked_array(data=[0.0, 1.0, --, --, 4.0],
             mask=[False, False,  True,  True, False],
       fill_value=1e+20)

numpy.ma.compress_rowcols(x, axis=None)
Suppress the rows and/or columns of a 2-D array that contain masked values.

If axis is None, both rows and columns are suppressed.

If axis is 0, only rows are suppressed.

If axis is 1 or -1, only columns are suppressed.

numpy.ma.compress_rows:
Suppress whole rows of a 2-D array that contain masked values.

numpy.ma.compress_cols:
Suppress whole columns of a 2-D array that contain masked values.

np.ma.compress_rowcols(masked_array, 1)

array([0.0, 1.0, 4.0])

References

https://www.zhihu.com
https://blog.csdn.net/c20081052/article/details/93488032

Tianchao龙虾

关注

4
点赞
踩
19

收藏

觉得还不错? 一键收藏
1
评论
SORT 代码阅读笔记

SORT 代码阅读笔记先创建个实例self.tracker = Sort.Sort(max_age=max_age, min_hits=min_hits)这里有两个参数，max_age表示在多少帧中没有检测，trackers就会中止。min_hits代表持续多少帧检测到，生成trackers。检测器每来一帧数据，进行检测，返回检测结果detections=[[x1,y1,x2,y2,score],[....]][[x_1,y_1,x_2,y_2,score], [....]][[x1,y1,x
复制链接

扫一扫