(上)基于提升HOG特征的自动多斑马鱼跟踪 Automatic multiple zebrafish tracking based on improved HOG features

www.nature.com/scientificreports

scientific reports　二区

摘要

As an excellent model organism, zebrafish have been widely applied in many fields.

The accurate identification and tracking of individuals are crucial for zebrafish shoaling behaviour analysis.

However,multi-zebrafish tracking still faces many challenges.

It is difficult to keep identified for a long time due to fish overlapping caused by the crossings.

Here we proposed an improved Histogram of Oriented Gradient (HOG) algorithm to calculate the stable back texture feature map of zebrafish, then tracked multi-zebrafish in a fully automated fashion with low sample size, high tracking accuracy and wide applicability.

The performance of the tracking algorithm was evaluated in 11 videos with different numbers and different sizes of zebrafish.

In the Right-tailed hypothesis test of Wilcoxon, our method performed better than idTracker, with significant higher tracking accuracy.

 Throughout the video of 16 zebrafish, the training sample of each fish had only 200–500 image samples, one-fifth of the idTracker’s sample size.

Furthermore, we applied the tracking algorithm to analyse the depression and hypoactivity behaviour of zebrafish shoaling.

 We achieved correct identification of depressed zebrafish among the fish shoal based on the accurate tracking results that could not be identified by a human.

斑马鱼作为一种优良的模式生物，在许多领域得到了广泛的应用。
准确识别和跟踪个体对于斑马鱼的鱼群行为分析至关重要。
然而，多斑马鱼的跟踪仍然面临着许多挑战。
由于交叉造成的鱼类重叠，长期难以识别。
本文提出了一种改进的方向梯度直方图(HOG)算法来计算斑马鱼稳定的背纹理特征图，然后采用全自动化的方式对多条斑马鱼进行跟踪，样本量小，跟踪精度高，适用性广。
在11个不同斑马鱼数量和大小的视频中对跟踪算法的性能进行了评估。
在Wilcoxon的右尾假设检验中，我们的方法优于idTracker，跟踪精度显著提高。
在整个16条斑马鱼的视频中，每条鱼的训练样本只有200-500个图像样本，是idTracker样本大小的五分之一。
此外，我们应用跟踪算法分析了斑马鱼浅滩的低度和低活动行为。
通过对人类无法识别的斑马鱼的精确跟踪，实现了对鱼群中抑郁斑马鱼的正确识别。

Zebrafish have been widely applied in many fields as an excellent model organism, for example in biological experiments 1 , drug screens 2 , gene screens 3 , neurotoxicology analysis 4 , and swarm intelligence development 5
 The notable shoaling behaviour of zebrafish has garnered widespread interest among scientists
 It is crucial to achieve accurate and rapid identification and tracking of individuals for relevant mechanistic analyses of and predictions for ecosystems
 However, multi-zebrafish tracking still faces many challenges, such as similar shape, movement and frequent occlusion 6
 A series of computer vision tracking methods were proposed to solve these problems
 These methods generally fall into two categories: the tracking methods based on detection-data association and methods based on identification
 In the first category, researchers extracted the objects from videos first and then associated the positions of the objects continuously to achieve multi-object tracking
 In 2008, Nevatia proposed a multi-level data association tracking framework 7
 Dicle et al
 tracked multi-objects with similar appearances by evaluating the rank similarity of the matrix, which was formed by the trajectory data 6
 This system worked well in numerous fields
 Similar methods also include an extended kalman filter 8
 However, this types of tracking methods is more suitable for tracking objects with immutable movement
 Thus this method cannot be effectively adapted to zebrafish with uncertain mobility
 In the second category, the main idea is to correctly identify individuals
 Then multi-object tracking is per- formed based on the identification
 Researchers have manually labelled the targets, such as with colour labels 9 and visible implantable elastomer (VIE) tags 10 , to achieve identification in the early stage
 However, the process of manual marking is very complicated, and the marks even may affect the social behaviour of the objects
 In 2014, Alfonso Pérez-Escudero et al
 proposed idTracker, which identifies individuals based on fingerprinting features on the back texture of animals 11
 This fully automated tracking method has achieved good tracking performances in many fields
 However, the method has high requirements regarding the quality and quantity of samples, given that the fingerprinting feature is not stable
 

斑马鱼作为一种优秀的模型生物，在生物实验1、药物筛选2、基因筛选3、神经毒理学分析4、群体智能发育5等领域得到了广泛的应用

斑马鱼这种引人注目的浅滩行为引起了科学家们的广泛兴趣

对于生态系统的相关力学分析和预测，实现对个体的准确、快速的识别和跟踪是至关重要的

然而，多斑马鱼的跟踪仍然面临着许多挑战，如形状相似、运动频繁、遮挡频繁等

针对这些问题，提出了一系列计算机视觉跟踪方法

这些方法一般分为两类:基于检测数据关联的跟踪方法和基于识别的跟踪方法

在第一类中，研究人员首先从视频中提取目标，然后将目标的位置连续关联起来，实现多目标跟踪

2008年，Nevatia提出了一个多级数据关联跟踪框架7

蒂格里斯等

通过评价由弹道数据构成的矩阵的秩相似度，对外形相似的多目标进行跟踪

这个系统在许多领域都运行良好

类似的方法还包括扩展卡尔曼滤波器8

但是，这种跟踪方法更适合于跟踪具有不可变运动的对象

因此，该方法不能有效地应用于机动性不确定的斑马鱼

在第二类中，主要是正确识别个体，在此基础上进行多目标跟踪，　研究人员已经对目标进行了人工标记，例如用彩色标签9和可见的可植入弹性体(VIE)标签10来实现早期识别

然而，手工标记的过程是非常复杂的，标记甚至可能影响到对象的社会行为

2014年，Alfonso Perez-Escudero等人

提出了一种基于动物背纹理指纹特征识别个体的idTracker算法

这种全自动跟踪方法在许多领域都取得了良好的跟踪性能

但由于指纹特征不稳定，该方法对样品的质量和数量要求较高

 Experimental videos must be sufficiently long to ensure each zebrafish  has approximately 2500 image samples, which results in a slow processing speed that cannot be widely used in practice.
 Therefore, it is necessary to design new image features to realize a multi-zebrafish tracking method.
 The Histogram of Oriented Gradient (HOG) algorithm is an image descriptor proposed by Dalal et al.
 to detect human 12 .
 The HOG algorithm calculates the histogram of gradient directions or edge orientations in local regions and is stable when tracking humans.
 However, the algorithm is orientation-sensitive to the input images and cannot be applied to zebrafish tracking directly.
 To improve the orientation-sensitive nature of the algorithm and increase stability, we proposed an improved HOG algorithm to calculate the stable back texture feature map of the zebrafish and track multi-zebrafish in a fully automated fashion.
 In this paper, we classified the HOG feature blocks based on the correlation of the zebrafish’s back texture and then outputted the relevant back texture feature blocks spirally from the centre.
 In addition, the training samples were automatically accumulated based on the trajectory analysis and the improved HOG + SVM (Support Vector Machine) classification mechanism.
 Finally we established the data association strategies based on the identifica- tion of multiple zebrafish and obtained the final whole trajectories.
 To verify the validity of the improved HOG features, we compared the proposed improved HOG algorithm with the previous HOG method by identifying multiple zebrafish.
 Then we applied the tracking algorithm to identify 3 types of animal models, and all of them were successfully identified.
 A plurality of zebrafish in the growth cycle more than a month was also identified.
 Furthermore, eleven videos of different numbers and different sizes of zebrafish were processed.
 The tracking results showed that the accuracy rates were considerably increased compared with state-of-the-art tracking methods.
 Statistics of 8 tracking videos revealed that the average accuracy rate of the pro- posed method is 99.
27%, 4.
33% higher than idTracker 11 .
 In the Right-tailed hypothesis test of Wilcoxon, our method performed better than idTracker, with significant higher tracking accuracy.
 Throughout the video of 16 zebrafish, the training sample of each fish had only 200–500 image samples, one-fifth of the idTracker’s sample size.
 The proposed method can also calculate the stable feature of zebrafish larvae, and have a good tracking performance.
 Finally, we applied the tracking algorithm to analyse the depression and hypoactivity behaviour of the zebrafish shoaling.
 We achieved correct identification of depressed zebrafish among the fish shoal based on accurate tracking results that could not be identified by a human.
 In summary, the proposed method based on improved HOG features tracks multiple zebrafish in a fully automated fashion with low sample size, high tracking accuracy and wide applicability.

实验视频必须足够长，保证每条斑马鱼大约有2500个图像样本，这导致处理速度较慢，不能广泛应用于实际。

因此，有必要设计新的图像特征来实现多斑马鱼跟踪方法。

方向梯度直方图(Histogram of Oriented Gradient, HOG)算法是Dalal等人提出的一种图像描述符。

来检测人类12。

HOG算法计算局部区域梯度方向或边缘方向的直方图，在跟踪人类时是稳定的。

然而，该算法对输入图像的方向敏感，不能直接应用于斑马鱼的跟踪。

为了提高算法的方向性和稳定性，我们提出了一种改进的HOG算法来计算斑马鱼稳定的背纹理特征图，并以完全自动化的方式跟踪多条斑马鱼。

本文根据斑马鱼背纹理的相关性对猪的背纹理特征块进行分类，然后从中心向外螺旋输出相关的背纹理特征块。

此外，基于轨迹分析和改进的HOG + SVM (Support Vector Machine)分类机制，自动累积训练样本。

最后建立了基于多重斑马鱼识别的数据关联策略，得到了最终的整体轨迹。

为了验证改进后的HOG特征的有效性，我们通过对多种斑马鱼的识别，将改进后的HOG算法与之前的HOG算法进行了比较。

然后应用跟踪算法对3种动物模型进行了识别，均成功识别。

多种斑马鱼在生长周期中也被鉴定出一个多月。

此外，还对11个不同数量和大小的斑马鱼视频进行了处理。

跟踪结果表明，与最先进的跟踪方法相比，该方法的准确率有较大提高。

通过对8段跟踪视频的统计，表明该方法的平均准确率为99。

27%,4。

比idTracker 11高出33%。

在Wilcoxon的右尾假设检验中，我们的方法优于idTracker，跟踪精度显著提高。

在整个16条斑马鱼的视频中，每条鱼的训练样本只有200-500个图像样本，是idTracker样本大小的五分之一。

该方法还可以计算斑马鱼幼虫的稳定特征，具有良好的跟踪性能。

最后，我们应用跟踪算法分析了斑马鱼浅滩的低度和低活动行为。

我们根据人类无法识别的精确跟踪结果，在鱼群中准确识别出了抑郁的斑马鱼。

综上所述，该方法基于改进的HOG特征，以完全自动化的方式跟踪多条斑马鱼，样本量小，跟踪精度高，适用性广。

>>>Methods and Materials

The tracking algorithm consists of three modules: the Preprocessing module, the Feature extracting module, and the Tracklets classification and matching module (Fig .
 1) .
 In the Preprocessing module, we gathered ROIs of single zebrafish as image samples for classifier and matched the zebrafish positions into initial tracklets, which are short trajectory segments for each individuals with timing relations .
 In the Feature extracting module, we pro- posed the improved HOG algorithm and then extracted the improved HOG features of zebrafish image samples .
 In the Tracklets classification and matching module, we trained 2 types of classifiers by applying the improved HOG + SVM mechanism .
 For the first type of classifier, we trained numerous classifiers by the initial tracklets before and after the same crossings .
 Then, we lengthened the initial tracklets by this type of classifiers to enrich the image samples .
 The second type of classifier was trained by the longest tracklets of each zebrafish among the lengthened tracklets .
 We used the only classifier to obtain the final trajectory of each zebrafish .

  >>>Preprocessing module .
  The Preprocessing module contains two parts: zebrafish ROI image samples collection and initial tracklets acquisition .
 On one hand, we built the background model and intercepted the zebrafish ROIs as image samples .
 On the other hand, we matched the zebrafish positions into initial tracklets for each individual as long as possible .

  >>>Zebrafish ROI collection .
  The flow chart of zebrafish ROI collection is presented in Fig .
 2 .
 We used a static background model for motion detection, given that the laboratory environment is relatively stable .
 The average intensity values of the first f frames were used to calculate the intensity of the background model:

方法和材料

跟踪算法由预处理模块、特征提取模块和轨迹分类匹配模块三部分组成。

1 在预处理模块中，我们采集单个斑马鱼的roi作为分类器的图像样本，并将斑马鱼的位置匹配到初始轨迹，初始轨迹是每个个体具有定时关系的短轨迹段。

2 在特征提取模块中，提出了改进的HOG算法，并提取了斑马鱼图像样本的改进HOG特征。

3 在Tracklets分类匹配模块中，我们采用改进的HOG + SVM机制训练了两类分类器。

第一类分类器，我们通过相同交叉前后的初始轨迹训练了大量分类器。

然后，我们通过这类分类器延长了初始轨迹，以丰富图像样本。

第二种分类器, 是由每条斑马鱼中最长的航迹训练而成。

我们使用唯一的分类器来获得每条斑马鱼的最终轨迹。

> > >预处理模块。

预处理模块包括两部分:斑马鱼ROI图像样本采集和初始轨迹采集。

一方面，建立背景模型，截取斑马鱼roi作为图像样本。

另一方面，我们尽可能长时间地将斑马鱼的位置匹配到每个个体的初始轨迹中。

>>>斑马鱼ROI采集。

斑马鱼ROI采集流程图如图所示。

考虑到实验室环境相对稳定，我们使用静态背景模型进行运动检测。

使用前f帧的平均强度值计算背景模型的强度:　　intensity这里是亮度

where G t (x,y) is the intensity at point (x,y) on frame t.
 We extracted the whole zebrafish region by applying the  background subtraction method.
 Then we obtained the minimum enclosing rectangles of the zebrafish in the  whole zebrafish region by using an elliptic fitting algorithm, and calculated the centre coordinate and the tilt angle  θ of the enclosing rectangle.
 The initial value of θ is ranged from −90° to 90°.
  We intercepted the head region of the zebrafish as the zebrafish ROI given that the whole zebrafish is a  non-rigid object and the shape may change as it swims.
 To obtain the stable HOG feature, we set a threshold Thr e  of the aspect ratio of the enclosing rectangle to exclude the zebrafish ROIs where the fish bodies are significantly  deformed or overlapping occurs, shown as Figs 1 and 2 in Supplement s1.
 Finally, we rotated the zebrafish ROI to  orient the fish head to the right, which greatly eliminated the orientation-sensitity of the HOG algorithm.
 

其中G t (x,y)是坐标系t上(x,y)点的强度。

采用背景减法提取整个斑马鱼区域。

然后我们获得的最小包围矩形斑马鱼整个地区的斑马鱼通过使用一种椭圆拟合的算法,计算中心坐标和封闭矩形的倾角θ。

的初始值θ是介于−90°- 90°。

我们截取了斑马鱼头部区域作为斑马鱼的ROI，因为整个斑马鱼是一个非刚性物体，它的形状可能会随着游动而改变。

为了获得稳定的HOG特征，我们设置了一个外方矩形长宽比的阈值Thr e来排除鱼体发生明显变形或重叠的斑马鱼roi，
如图1和图2所示，在附录s1中。PS:这个长宽比指的头部还是全身呢？？

最后，我们旋转斑马鱼ROI使鱼头向右定位，这大大消除了HOG算法的定位敏感性。

ＰＳ:
1 截取头部
2 外接长方形长宽比固定（头部外接正方形）
3 向右旋转定位

算法：
椭圆拟合算法

fig1

Figure 1.
  Overview of the proposed multiple zebrafish tracking system.
 There are three main modules: the  Preprocessing module, the Feature extracting module, the Tracklets classification and matching module.
 In  the Preprocessing module, we gathered ROIs of single zebrafish as image samples for classifier and matched  the zebrafish positions into initial tracklets.
 In the Feature extracting module, we proposed the improved  HOG algorithm, then we extracted the improved HOG features of zebrafish image samples.
 In the Tracklets  classification and matching module, we trained 2 types of classifiers by applying the improved HOG+SVM  mechanism.
 The initial tracklets before and after the same crossings were further lengthened based on the first  type of classifier to enrich the image samples.
 The second type of classifier was trained by the longest tracklets of  each zebrafish among the lengthened tracklets to obtain the final trajectory.

图1所示。
提出的多斑马鱼跟踪系统概述。
主要有三个模块:预处理模块、特征提取模块、轨迹分类匹配模块。
在预处理模块中，我们采集单个斑马鱼的roi作为分类器的图像样本，并将斑马鱼的位置匹配到初始轨迹中。
在特征提取模块中，提出了改进的HOG算法，并对斑马鱼图像样本进行了改进的HOG特征提取。
在Tracklets分类匹配模块中，我们采用改进的HOG+SVM机制训练了两类分类器。
在第一类分类器的基础上，对相同交叉口前后的初始轨迹进行进一步加长，以丰富图像样本。
第二种分类器由每条斑马鱼的最长航迹在加长航迹中进行训练，得到最终的航迹。

fig2

Figure 2.  The flow chart of zebrafish ROIs collection.
We intercepted the head region as the zebrafish ROI by calculating the minimum enclosing rectangles, the center coordinate and the tilt angle θ of the whole zebrafish region.

图2　斑马鱼ROIs采集流程图。
我们截获的头部区域的斑马鱼ROI计算最低封闭矩形的中心坐标和倾角θ斑马鱼整个地区。

 

As shown in Fig. 3, the steps of zebrafish ROI extraction are listed as follows:

    (1)     Determine the direction of the zebrafish body.
 　　　The zebrafish was oriented towards the left and right if  θ ∈ (−45°, 45°) (regions 3 and 4 in Fig. 3F).
 　　　Otherwise, the zebrafish was oriented up and down (regions  1 and 2 in Fig. 3E).
      (2)     Calculate the position of the zebrafish head, pHead.
 　　　The head is positioned on the side with the larger sum  of pixels in the external rectangle region, as shown in Fig. 3G,H.
      (3)     Recalculate the tilt angle of the zebrafish in the range of [0, 360°] based on the head direction as shown in  Fig. 3I,J.

 

如图3所示，斑马鱼ROI提取步骤如下:

(1)确定斑马鱼身体的方向。

斑马鱼是面向左和右如果θ∈(−45°、45°)(区域3和4在)图3 f

否则，斑马鱼的方向是向上和向下的(区域1和2)图3E

(2)计算斑马鱼头部的位置，pHead。

头部位于外部矩形区域像素和较大的一侧，如图3G,３H所示。

(3)根据头部方向重新计算[0,360°]范围内的斑马鱼倾斜角度，如图3I,3J所示。

fig3

    (4)     Rotate the enclosed rectangle region clockwise to orient the fish head to the right based on the modified tiltangle θ.
    (5)     Intercept the head to aggregate the zebrafish ROIs collection.

 

(4)顺时针旋转封闭的矩形区域,鱼头向右的基础上修改tiltangleθ。

ＰＳ：上述４类正好是鱼头朝向１，３，４，２象限的四种情况

(5)截去头部集合斑马鱼ROIs集合

>>>Initial tracklets acquisition.

  We applied a heuristic strategy based on the minimum distance to obtain the  initial tracklets without object overlapping because the swimming displacement（位移） of fish is limited in consecutive  frames.
 Thus, the length of initial tracklets is related to the crossing frequency in approximately 50–300 frames.
  Let O i,t-1 and O j,t denote a zebrafish on frame t-1 and frame t, respectively, and N t-1 and N t are the numbers of  objects on frame t-1 and frame t, respectively.
 D(O i,t-1 , O j,t ) denotes the Euclidean distance between O i,t-1 and O j,t .
  Then the matching results are:

 

由于连续帧中鱼的游动位移受限制，我们采用了一种基于最小距离的启发式策略来获得无目标重叠的初始轨迹。
(启发式策略让初始段轨迹无目标重叠)

因此，初始轨迹的长度与大约50-300帧的交叉频率有关。

设O i,t-1,　 O j,t分别表示帧t-1和帧t上的斑马鱼，N t-1和N t分别表示帧t-1和帧t上的物体数。

D(O i,t-1, O j,t)表示O i,t-1到O j,t之间的欧氏距离。

则匹配结果为:

上一帧一个目标i，与下一帧中Ｎt个目标都做匹配,最小的那个匹配上

 

If two objects O i,t-1 and O k,t-1 on frame t-1 matched with the same object O j,t on frame t according to equation (3),
we set:

根据式(3), 如果在帧t-1上的两个物体Oi,t -1和O k,t-1与　t帧的同一物体O j,t匹配.PS：上一帧的两个对象与下一帧同一个对象匹配上

我们设:

 

Additionally, the numbers of objects would change if the zebrafish were crossing, so the crossing determination criteria is defined as follows:

另外，如果斑马鱼发生交叉情况, 对象的数量会发生变化，所以交叉判定标准定义如下:

In particular, C(O i,t-1 , O j,t ) = 0 while both O i,t-1 and O k,t-1 matched with O j,t , which may be caused by the stationary  fish or the special matching distance.
 Then we matched O i,t-1 and O j,t if the difference between D(O i,t-1 , O j,t ) and  D(O k,t-1 , O j,t ) is greater than threshold Thr d :

其中，C(O i,t-1, O j,t) = 0，而O i,t-1和O k,t-1都与O j,t匹配，这可能是由于静止的鱼(???)或特殊的匹配距离(距离相等)。

如果D(O i,t-1, O j,t)与D(O k,t-1, O j,t)之差大于阈值Thr D，则匹配O i,t-1和O j,t:   (i比k更像j,超出阈值的像)

这里Ｄ是什么？？

We discarded the short tracklets (<10 frames) caused by crossings to avoid introducing the possible overlapping errors.

为了避免可能的重叠错误，我们丢弃了交叉导致的短轨(<10帧)。ps:短暂的交叉就不要了

>>>Feature extracting module.
  In the Feature extracting module, we proposed the improved HOG algorithm  to calculate the stable back texture feature map of zebrafish, which greatly increased the certainty of identification.
  Then we extracted the improved HOG features of zebrafish image samples.

 

>>>Improved HOG feature extracting.
  The basic idea of the HOG algorithm is that local object appearance  and shape can often be characterized rather well by the distribution of local intensity gradients or edge directions 12 .
 It is implemented by dividing the binary image into small spatial regions (“cells”), for each cell accumulating a local 1-D histogram of gradient directions or edge orientations over the pixels of the cell.
 There is no  overlapping between the cells.
 For better invariance to illumination, shadowing, etc.
, a certain number of cells are  configured to form the larger spatial regions (“blocks”).
 Then a measure of local histogram “energy” is generated  over blocks and the results are used to normalize all of the cells in the block.
 Overlapping between the blocks is  noted.
 Finally, the feature vectors are output by sliding windows.
 However the algorithm has orientation-sensitity  to the input images.

 

>>>特征提取模块。

在特征提取模块中，我们提出了改进的HOG算法来计算稳定的斑马鱼背纹理特征图，大大提高了识别的确定性。

然后提取改进后的斑马鱼图像样本的HOG特征。
>>>改进的HOG特征提取。

HOG算法的基本思想是通过局部强度梯度或边缘重定向的分布可以很好地表征局部对象的外观和形状。

12 [hog论文]Dalal, N. & Triggs, B. Histograms of oriented gradients for human detection. IEEE Computer Society Conference on Computer Vision
& Pattern Recognition. 1, 886–893 (2005).

它是通过将二值图像分割成小的空间区域(“单元”)来实现的，每个单元在单元的像素上累积一个局部的梯度方向或边缘方向的一维直方图。

细胞之间没有重叠。

为了更好的不变性，光照，阴影等。

，一定数量的单元格cell被配置成更大的空间区域block(“块”)。

然后在块上生成局部直方图“能量”的度量，并将结果用于对块中的所有单元格进行标准化。

注意到块之间的重叠。

最后，通过滑动窗口输出特征向量。

然而，该算法对输入图像具有方向性敏感。

To improve the orientation-sensitive nature of the algorithm and reduce the feature dimension, we proposed  the improved HOG algorithm.
 We scaled the zebrafish ROIs to a specified size as the image sample, such that  the input images of the improved HOG algorithm with same size are consistent in resolution, which improves  the adaptability of image resolution.
 

Then we classified the HOG feature blocks based on the correlation of the  zebrafish’s back texture.
 Block 0 indicates there is no correlation between the block and the back texture, block  1 indicates they are relevant.
 Then, a fixed number of blocks marked as 1 was outputted spirally from the centre  to obtain the improved HOG feature with low dimension but high stability.
 Figure 4 presents the process of the  improved HOG algorithm.
 

为了提高算法的方向性，减小特征维数，提出了改进的HOG算法。

我们将斑马鱼的ROIs缩放到指定的尺寸作为图像样本，使得相同尺寸的改进HOG算法的输入图像分辨率一致，提高了图像分辨率的适应性。

然后根据斑马鱼背部纹理的相关性对HOG的特征块进行分类。

Block 0表示Block和back纹理之间没有相关性，Block 1表示它们是相关的。

然后从中心向外螺旋输出一定数量的标记为1的块，得到维度小但稳定性高的改进HOG特征。

图4给出了改进的HOG算法的过程。

 

fig4

下面这里写反了no correlation

Figure 4.
  The process of the improved HOG algorithm.
 First we applied the HOG algorithm to convert the  zebrafish image samples to the HOG feature map.
 Then we classified the HOG feature blocks based on the  correlation of the zebrafish’s back texture.
 The fixed number of blocks marked as 1 were outputted spirally from  the centre.
 Finally we obtained the improved HOG feature with low dimensionality but high stability.

 

图4改进的HOG算法的过程。
首先应用HOG算法将斑马鱼图像样本转换为HOG特征图。
然后根据斑马鱼背部纹理的相关性对hog的特征块进行分类。
标记为1的固定块数从中心螺旋输出。
最后得到了低维、高稳定性的改进HOG特征。

 The steps are as follows:

1)     Scale the zebrafish ROIs to the specified size P r as the image sample.
       2)     Binarize the image samples.
 The foreground pixels and the background pixels are marked as 1 and 0 re-  spectively.
 Let Bi t (x,y) be the binary value of point (x,y) on frame t:  where Bm is the background model in equation (1).
       3)     Apply the HOG algorithm to convert the zebrafish image samples to the HOG feature map.
       4)     Calculate the pixel number with the value of 1 in each cell of the binary image.
 The cell is marked as 0 if the  pixel number is less than a fixed threshold Thr c .
       5)     Group all the labelled cells into blocks.
 The block is marked as 0 if the sum of cells in the block is 0.
 Other-  wise, the block is marked as 1.
       6)     Output the fixed number of blocks marked as 1 spirally to calculate the final HOG feature as shown in  Fig.
 5.
 The fixed number of blocks is set to P b .

步骤如下:

1)将斑马鱼roi缩放到指定的尺寸P r作为图像样本。

2)对图像样本进行二值化。

将前景像素和背景像素分别标记为1和0。

设Bi t (x,y)为t帧上点(x,y)的二进制值，

其中Bm为式(1)中的背景模型。

3)应用HOG算法将斑马鱼图像样本转换为HOG特征图。

4)计算二进制图像中每个单元(cell)中值为1的像素数量, 像素数量大于固定阈值Thr c， cell(单元格)被标记为1
如果像素数量小于固定阈值Thr c，则单元格被标记为0。

5)将所有标记细胞cell分组为block。

如果块中的单元格和为0，则该块被标记为0。块中的单元格和为大于０，则该块被标记为１．ps: 说明这是背景

另一方面，块被标记为1。

6)输出标记为1的固定块数，螺旋式地计算最终的HOG特征，如图5所示。

固定块的数量, 设为。

fig5

The order of output blocks in the HOG feature map.
 We spirally output the fixed number of blocks marked as 1 to calculate the final HOG feature, which contains more texture features and less background.
 The size of the feature is small compared with the previous method.

HOG特征图中输出块的顺序。

我们螺旋输出固定数量的块标记为1来计算最终的HOG特征，它包含更多的纹理特征和更少的背景。

与前一种方法相比，该特征的尺寸较小。

>>>Tracklets classification and matching module.
  In this module, we trained 2 types of classifiers by  applying the improved HOG + SVM mechanism.
 The initial tracklets before and after the same crossings were  further lengthened based on the first type of classifier.
 We aimed at stitching the tracklets as long as possible here  to enrich and accumulate the image samples.
 A plurality of classifiers was trained in this manner.
 The other type  of classifier was trained to calculate the final trajectory of each zebrafish.
 It trained by the longest tracklets belonging to each zebrafish without overlapping among the lengthened tracklets.

>>>轨迹,分类&匹配模块。

在该模块中，我们使用改进的HOG + SVM机制训练了两类分类器。

在第一类分类器的基础上，进一步延长了相同交叉的前后的初始轨迹。

我们的目标是在这里拼接轨迹尽可能长，以丰富和积累图像样本。

用这种方法训练了多个分类器。

训练另一种分类器来计算每条斑马鱼的最终轨迹。

它由属于每条斑马鱼的最长航迹训练而成，，与前面的加长航迹之间没有重叠。

Let N be the number of zebrafish tracked in the tracking video.
 T t  = {T it | i = 1, 2, 3...N} denotes a group of  tracklets sharing the same frame t.
 NF t  = {NF it | i = 1, 2, 3...n} denotes the length of tracklets in group T t .
 In the  training of the first type of classifiers, we first calculated the shortest tracklets with length of min(NF t ) in the  two groups of tracklets before and after crossing.
 Then we compared the lengths of the shortest tracklets of two  groups to obtain the longer tracklets.
 Third, we used group T t with the longer tracklet to train the SVM classifier.
  Finally, we classified the tracklets in the other group by the classifier and associated the tracklets with the same  identities.
 As shown in Fig.6,

设N为跟踪视频中被跟踪的斑马鱼的数量。

{T | i = 1, 2, 3…N}表示一组共享同一帧t的tracklet。

{NF| i = 1, 2, 3…n}为T组中tracklet的长度。

we first calculated the shortest tracklets with length of min(NF t ) in the  two groups of tracklets before and after crossing.

在第一类分类器的训练中，我们首先计算了在交叉前后的两组轨迹中最短的两组tracklets with min(NFt)。
ps: 分别是图６的min(NF)A, min(NF)B,　取较长min(NF)A来训练

然后我们比较了两组中最短轨迹的长度，得到了较长的轨迹。

然后，我们使用更长轨迹的组Tt来训练SVM分类器。

最后，通过分类器对另一组的tracklets进行分类，并将具有相同身份的tracklets关联起来。ps:同样的操作再来一遍

如图６所示。

fig6

 

group A is selected as the training set, and group B is employed as the test set as min  (NF) A  > min(NF) B .
Then, we stitched the initial tracklets with the same identity.
In this manner, the samples were  accumulated by the lengthened tracklets, which greatly increases the certain of identification.
For training the  second type of classifier, many tracklets groups are waiting to be matched, as shown in Fig.7.
We chose one group  as the training set for the SVM classifier.
First, we calculated the shortest tracklets with a length of min(NF t ) in all groups.
Then we compared these shortest tracklets to obtain the longest tracklets with a length of max(min(NF t )).
Finally the tracklet group T t , which contained the longest tracklet, was selected to train the classifier.

选择A组作为训练集，B组作为测试集min(NF) A > min(NF) B。

然后，我们用相同的身份拼接了最初的tracklets。

这样，样品通过加长的小轨积累，大大提高了鉴定id的确定性。

为了训练第二种分类器，许多tracklets组等待匹配，如图7所示。

我们选择一组作为SVM分类器的训练集。

首先，我们计算了所有组中长度为min(NF t)的最短轨迹。ps:各组取最小

然后我们对这些最短的轨迹进行比较，得到max(min(NF t ))长度最长的轨迹。ps:各组最小值比较取最大

最后选择包含最长轨迹的轨迹组Tt对分类器进行训练。

fig7

Let p ij be the predicted probability of Tit classified as the class T it+n .
 The probabilities matrix is noted as follows:  

 Select N p ij in equation (8), no two of which lie in the same row or column.
 We used the Hungarian algorithm  to match the tracklets group T t and T t+n based on p ij .
 If the sum of N p ij was maximal and all of the p ij were greater  than 1/N, matching was performed.
 The predicted label of the tracklet T it (i =1, 2, 3.N) are calculated as follows:

设pij为Tit的预测概率，分类为T it+n类。

概率矩阵记为:

在公式(8)中选择（Ｎpij），其中没有两个位于同一行或同一列。

我们使用匈牙利算法来匹配基于pij的tracklet组(Tt和T t+n)。

如果npij的和最大，且所有的pij都大于1/N，则进行匹配。

对T (i = 1,2,3 . n)轨迹的预测标号predicted label计算如下:

which indicated the matching association is successful if the tracklets before and after the interval have the same label.

如果间隔前后的tracklet具有相同的标签，则表示匹配关联成功。

We also provided the manual error correction function to correct tracking errors and improve the accuracy  of tracking results.
 To reduce manual checking, the possible tracking errors are located in the image sequence  automatically.
 The error correction is accomplished by convenient human-computer interaction.
 We assume that  the matching error may occur in tracklets when the matching probability value is less than a threshold Thr p .
  The error correction function automatically locate these points.
 Then the user can view the tracking results of  adjacent frames and correct the wrong matching results easily by clicking the mouse.
 This function is detailed in  Supplement s1.

我们还提供了手动纠错功能，以纠正跟踪误差，提高跟踪结果的准确性。
为了减少人工检查，可能的跟踪误差自动定位在图像序列中。
通过方便的人机交互，实现了误差的校正。
我们假设当匹配概率值小于阈值Thr p时，tracklets可能会发生匹配误差。
纠错功能自动定位这些点。
然后用户可以查看相邻帧的跟踪结果，并通过单击鼠标轻松纠正错误的匹配结果。
这个函数在附录s1中有详细说明。