[论文精读] 实时目标追踪算法SORT

忘是亡心i 2023-05-29 03:53 23阅读 0赞

# 题目：SIMPLE ONLINE AND REALTIME TRACKING #

## 摘要 ##

This paper explores a pragmatic approach to multiple object tracking where the main focus is to associate objects efficiently for online and realtime applications. To this end, detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%. Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260Hzwhich is over 20x faster than other state-of-the-art trackers.

Index Terms— Computer Vision, Multiple Object Tracking, Detection, Data Association

本文探讨了一种实用的多目标跟踪方法，旨在有效地关联目标做在线和实时地应用。为此，目标检测质量被认为是影响跟踪性能的关键因素，其中改变检测器可以改善追踪率高达18.9%。尽管仅使用了一种将为人熟知的卡尔曼滤波和匈牙利算法作为跟踪组建的初级结合，该方法实现了与现有最先进跟踪技术相当的精度。此外，由于本文的跟踪方法简单化，跟踪器以260Hz的速率做更新，比其他先进跟踪器快了20倍。

关键词——计算机视觉，多目标跟踪，检测，数据关联

## 引言 ##

This paper presents a lean implementation of a tracking-by-detection framework for the problem of multiple object track-ing (MOT) where objects are detected each frame and repre-sented as bounding boxes. In contrast to many batch based tracking approaches \[1, 2, 3\], this work is primarily targeted towards online tracking where only detections from the pre-vious and the current frame are presented to the tracker. Additionally, a strong emphasis is placed on efficiency for facilitating realtime tracking and to promote greater uptake in  
applications such as pedestrian tracking for autonomous vehicles.

本文提出一种针对多目标跟踪问题的基于检测的追踪框架的精益实现，其中每帧对象以边界框形式检测出。与许多基于批处理的跟踪方法相比，这项工作主要针对在线跟踪，其中只有前一帧和当前帧的检测被呈现于跟踪器。此外，重点要强调的是促进实时跟踪的效率，并促进更大程度的应用，如自动驾驶中的行人跟踪。

The MOT problem can be viewed as a data association problem where the aim is to associate detections across frames in a video sequence. To aid the data association process, trackers use various methods for modelling the motion \[1, 4\] and appearance \[5, 3\] of objects in the scene. The methods employed by this paper were motivated through observations made on a recently established visual MOT benchmark \[6\]. Firstly, there is a resurgence of mature data association techniques including Multiple Hypothesis Track-  
ing (MHT) \[7, 3\] and Joint Probabilistic Data Association (JPDA) \[2\] which occupy many of the top positions of the MOT benchmark. Secondly, the only tracker that does not use the Aggregate Channel Filter (ACF) \[8\] detector is also the top ranked tracker, suggesting that detection quality could be holding back the other trackers. Furthermore, the trade-off between accuracy and speed appears quite pronounced, since the speed of most accurate trackers is considered too slow for realtime applications (see Fig. 1). With the prominence of traditional data association techniques among the top online and batch trackers along with the use of different detections  
used by the top tracker, this work explores how simple MOT can be and how well it can perform.

MOT问题可看做是数据关联问题，其目的是将视频序列帧中的检测结果相关联。为了帮助数据关联过程，跟踪器使用各种方法来构建场景中目标们的运动模型和外观模型。本文采用的方法是通过建立在视觉 MOT benchmark上的观测结果上的。首先，有很多较为成熟的数据关联技术，如多假设跟踪（ MHT）和联合概率数据关联（JPDA）占据了很多MOT benchmark的前几名。其次，唯一没有使用聚合通道滤波（ACF）检测器也是排名靠前的跟踪器，暗示检测质量可以压过其他跟踪器。此外，需要权衡准确性和速度，因为大多数准确性高的跟踪器都面临实时性低的问题。如图1所示。本文旨在探索如何简化MOT以及性能如何优化的方法。

Keeping in line with Occam’s Razor, appearance features beyond the detection component are ignored in tracking and only the bounding box position and size are used for both motion estimation and data association.Furthermore, issues regarding short-term and long-term occlusion are also ignored,as they occur very rarely and their explicit treatment introduces undesirable complexity into the tracking framework.We argue that incorporating complexity in the form of object re-identification adds significant overhead into the tracking framework – potentially limiting its use in realtime applications.

与奥姆特剃刀原理保持一致(读者备注：奥姆特剃刀原理是指如无必要，勿增实体)，在跟踪中检测部位之外的其他外观特性将会无视，而仅仅边界框位置和尺寸大小将作为运动估计和数据关联。此外，短期和长期的遮挡问题也将忽略掉，因为它们发生几率非常小，而且对它们的处理会给框架带来不必要的复杂性。我们认为以对象重识别的形式引入复杂性会给跟踪带来很大的开销——进而潜在的限制了在实时性应用程序的使用。

This design philosophy is in contrast to many proposed visual trackers that incorporate a myriad of components to handle various edge cases and detection errors \[9, 10, 11, 12\].This work instead focuses on efficient and reliable handling of the common frame-to-frame associations.Rather than aiming to be robust to detection errors, we instead exploit recent advances in visual object detection to solve the detection problem directly.This is demonstrated by comparing the common ACF pedestrian detector \[8\] with a recent convolutional neural network (CNN) based detector \[13\].Additionally, two classical yet extremely efficient methods, Kalman filter \[14\] and Hungarian method \[15\], are employed to handle the motion prediction and data association components of　the tracking problem respectively.This minimalistic formulation of tracking facilitates both efficiency and reliability for online tracking, see Fig. 1.In this paper, this approach is only applied to tracking pedestrians in various environments, however due to the flexibility of CNN based etectors \[13\], it naturally can be generalized to other objects classes.

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk_size_16_color_FFFFFF_t_70]

这种设计理念与许多视觉跟踪器形成了对比，后者包含了大量的组件来处理各种边缘情况和检测错误。本文侧重于高效可靠地处理常见的帧 到帧的关联，利用目标检测的最新进展来解决检测问题，不做目标检测误差针对性解决。此外，两种经典而又及其有效的方法，卡尔曼滤波和匈牙利算法分别用作处理跟踪中的运动预测和数据关联 本文方法目前只应用在跟踪多种环境中的行人，考虑基于CNN的灵活性，自然能够用于其他目标类别。

The main contributions of this paper are:  
• We leverage the power of CNN based detection in the context of MOT.  
• A pragmatic tracking approach based on the Kalman filter and the Hungarian algorithm is presented and evaluated on a recent MOT benchmark.  
• Code will be open sourced to help establish a baseline method for research experimentation and uptake in collision avoidance applications.

本文主要贡献：

*  本文利用基于CNN的检测器；
 *  呈现了一种基于卡尔曼滤波和匈牙利算法的实用性跟踪方法；
 *  将开放源代码来帮助建立一个基准方法。

## 文献回顾 ##

Traditionally MOT has been solved using Multiple Hypothe-sis Tracking (MHT) \[7\] or the Joint Probabilistic Data Association (JPDA) filters \[16, 2\], which delay making difficult decisions while there is high uncertainty over the object assignments.The combinatorial complexity of these approaches is exponential in the number of tracked objects making them impractical for realtime applications in highly dynamic envi-  
ronments.

传统多目标检测是采用MHT或JPDA, 当目标分配存在高度不确定性时，会延迟做出困难的决定。而且这些方法的组合复杂性伴随着跟踪对象的数量呈现指数级增长，这使得它们在高度动态环境中的实时应用变得不切实际。

Many online tracking methods aim to build appearance models of either the individual objects themselves \[17, 18, 12\] or a global model \[19, 11, 4, 5\] through online learning.In addition to appearance models, motion is often incorporated to assist associating detections to tracklets \[1, 19, 4, 11\]. When considering only one-to-one correspondences modelled as bipartite graph matching, globally optimal solutions such as the Hungarian algorithm \[15\] can be used \[10, 20\].

许多在线跟踪方法旨在通过在线学习建立单个对象的外观模型或全局模型。除了外观模型，运动轨迹常常能用于帮助关联检测物体。当仅考虑一对一的联系采用二分图匹配时，全局优化算法如匈牙利算法能够使用。

The method by Geiger et al. \[20\] uses the Hungarian algorithm \[15\] in a two stage process.First, tracklets are formed by associating detections across adjacent frames where both geometry and appearance cues are combined to form the affinity matrix.Then, the tracklets are associated to each other to bridge broken trajectories caused by occlusion, again using both geometry and appearance cues.  
This two step association method restricts this approach to batch computation. Our approach is inspired by the tracking component of \[20\], however we simplify the association to a single stage with basic cues as described in the next section.

Geiger等人提出的方法在两阶段过程中使用了匈牙利算法\[15\]。首先，轨迹是通过关联相邻帧之间的检测而形成的，在这些帧中几何结构和外观特征结合在一起形成关联矩阵。其次，轨迹间彼此关联来连接由遮挡造成的断裂，再次使用几何结构和外观特征。这种两阶段关联方法限制了该方法的批量计算能力。本文方法受到了跟踪组件的启发，但我们将关联简化为一个阶段。

## 方法 ##

The proposed method is described by the key components of detection, propagating object states into future frames, associating current detections with existing objects, and managing the lifespan of tracked objects.

该方法通过检测、将对象状态传播到未来帧、将当前检测与现有对象相关联、管理跟踪对象的生命周期等关键组件来描述。

### 3.1 检测 ###

To capitalise on the rapid advancement of CNN based detection, we utilise the Faster Region CNN (FrRCNN) detection framework \[13\]. FrRCNN is an end-to-end framework that consists of two stages. The first stage extracts features  
and proposes regions for the second stage which then classifies the object in the proposed region. The advantage of  
this framework is that parameters are shared between the two stages creating an efficient framework for detection. Addi-  
tionally, the network architecture itself can be swapped to any design which enables rapid experimentation of different ar  
chitectures to improve the detection performance.

为了利用基于CNN的检测器的快速发展，本文使用Faster RCNN 检测器。  
Faster RCNN 检测器是一个端到端的两阶段检测框架：第一个阶段提取特征然并提供区域给第二个阶段，第二个阶段在这些区域上做目标分类。这个框架的优势是两个阶段参数是共享的，从而创建一个有效的检测框架。此外，网络结构本身可设计性强，可快速试验不同架构来提高检测性能。

Here we compare two network architectures provided with FrRCNN, namely the architecture of Zeiler and Fergus (FrRCNN(ZF)) \[21\] and the deeper architecture of Simonyan and Zisserman (FrRCNN(VGG16)) \[22\]. Throughout this work, we apply the FrRCNN with default parameters learnt for the PASCAL VOC challenge.As we are only interested in pedestrians we ignore all other classes and only pass person detection results with output probabilities greater than 50% to the tracking framework.

这里，我们比较两个网络结构FrRCNN(ZF)和更深层的FrRCNN(VGG16)。我们使用在Pascal VOC上训练得到的参数作为默认参数。由于我们只对行人感兴趣，可以无视其他类别，并且将那些检测概率置信度高于50%的行人检测结果输出给跟踪框架。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk_size_16_color_FFFFFF_t_70 1]  
In our experiments, we found that the detection quality has a significant impact on tracking performance when comparing the FrRCNN detections to ACF detections.This is demonstrated using a validation set of sequences applied to both an existing online tracker MDP \[12\] and the tracker proposed here.Table 1 shows that the best detector (FrRCNN(VGG16)) leads to the best tracking accuracy for both MDP and the proposed method.

实验发现，对比Faster RCNN和ACF检测，检测质量对跟踪质量有显著的影响。这个现象在使用视频序列验证集的现有在线跟踪器MDP和本文使用的跟踪器中均存在。表1 显示最好的检测器Faster RCNN（VGG16）会使得MDP和本文提出的方法的跟踪精度都达到最佳。

### 3.2. 估计模型 ###

Here we describe the object model, i.e. the representation and the motion model used to propagate a target’s identity into the next frame.We approximate the inter-frame displacements of each object with a linear constant velocity model which is independent of other objects and camera motion.The state of each target is modelled as:

这里我们描述了目标模型，也即是说外观模型和运动模型将会被传到下一帧中用作目标身份识别（ID)。我们近似认为每个目标的帧间位移满足线性恒速模型，并且每个目标间的运动是独立的，和相机的运动也是独立的。那每个目标的状态模型可以描述为：

x = \[u, v, s, r, u̇, v̇, ṡ\] T ,

where u and v represent the horizontal and vertical pixel location of the centre of the target, while the scale s and r represent the scale (area) and the aspect ratio of the target’s bounding box respectively. Note that the aspect ratio is considered to be constant. When a detection is associated to a target, the detected bounding box is used to update the target state where the velocity components are solved optimally via a Kalman filter framework \[14\]. If no detection is associated to the target, its state is simply predicted without correction using the linear velocity model.

其中u和v代表目标中心的水平和垂直的像素位置；s和r分别表示目标的尺度（面积）和目标框bbox的比例值（注：比例r被认为是常数）；  
当一个目标被检测时，检测框将用于更新目标的状态，其中速度分量将用kalman滤波框架来解决。如果没有检测到与目标相关联，目标的状态用线性速度模型来预测，无矫正过程。

### 3.3. 数据关联 ###

In assigning detections to existing targets, each target’s bounding box geometry is estimated by predicting its new location in the current frame.The assignment cost matrix is then computed as the intersection-over-union (IOU) distance　between each detection and all predicted bounding boxes from the existing targets. The assignment is solved optimally using the Hungarian algorithm.  
Additionally, a minimum IOU is imposed to reject assignments where the detection to target overlap is less than IOU min .

在为现有目标分配检测时，每个目标框的bbox几何框都是通过预测当前帧新的位置估计得到的。assignment cost matrix 分配代价矩阵通过每个检测结果和所有现有目标的预测框 间的IOU距离计算得到。分配方法通过使用匈牙利算法得到最佳优化。此外，检测到的IOU与预测目标物间IOU小于IOUmin阈值时，检测的物体将被拒绝分配。

We found that the IOU distance of the bounding boxes implicitly handles short term occlusion caused by passing targets.  
Specifically, when a target is covered by an occluding object, only the occluder is detected, since the IOU distance appropriately favours detections with similar scale.This allows both the occluder target to be corrected with the detection while the covered target is unaffected as no assignment is made.

我们发现边界框的IOU距离能够潜在解决因目标移动造成的短时间遮挡问题。具体地说，当目标物被遮挡物遮挡时，只有遮挡物被检测出来，由于IOU距离适当地支持具有类似尺度目标的检测。这使得遮挡物目标需要通过检测得到矫正，而被遮挡的目标不受影响，因为没有分配任务。

3.4 创建与输出跟踪ID  
When objects enter and leave the image, unique identities need to be created or destroyed accordingly. For creating trackers, we consider any detection with an overlap less than IOU min to signify the existence of an untracked object. The tracker is initialised using the geometry of the bounding box with the velocity set to zero. Since the velocity is unobserved at this point the covariance of the velocity component is initialised with large values, reflecting this uncertainty. Additionally, the new tracker then undergoes a probationary period where the target needs to be associated with detections to accumulate enough evidence in order to prevent tracking of false positives.

Tracks are terminated if they are not detected for T Lost frames. This prevents an unbounded growth in the number of trackers and localisation errors caused by predictions over long durations without corrections from the detector. In all experiments T Lost is set to 1 for two reasons. Firstly, the constant velocity model is a poor predictor of the true dynamics and secondly we are primarily concerned with frame-to-frame tracking where object re-identification is beyond the scope of this work. Additionally, early deletion of lost targets aids ef-  
ficiency. Should an object reappear, tracking will implicitly resume under a new identity.

当目标进入和离开图像时，唯一的ID需要创建或者销毁。用于创建跟踪器时，我们认为任何检测结果重叠小于IOUmin时，存在没有被跟踪的对象。 使用边界框的几何图形来初始化跟踪器，并使速度设置为0。由于速度此时速度未被观测到，初始速度分量的协方差很高，反应了这种不确定性。此外，新的跟踪器需要经历试用期，即目标物需要与检测结果相关联积累到足够才能防止误追踪。

当跟踪器未被检测到TLost帧时，将终止这个跟踪器。这么做可以防止跟踪器数量的无线增长，以及长时间未通过检测来得到矫正的局部误差增长。在所有实验中，TLost设置为1有有两个原因：第一，恒定速度模型在真实动力学模型中是个很差的预测模型；第二，我们主要关注帧和帧之间的跟踪，而目标REID超出本工作范围。此外，早期删除目标有助于提高效率。如果目标重新出现，则会隐式分配新的ID来跟踪。

## 4 实验 ##

在MOT数据集上做评估。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk_size_16_color_FFFFFF_t_70 2]

### 4.1 指标 ###

• MOTA(↑): Multi-object tracking accuracy \[25\].

• MOTP(↑): Multi-object tracking precision \[25\].

• FAF(↓): number of false alarms per frame.

• MT(↑): number of mostly tracked trajectories. I.e. target has the same label for at least 80% of its life span.

• ML(↓): number of mostly lost trajectories. i.e. target is not tracked for at least 20% of its life span.

• FP(↓): number of false detections.

• FN(↓): number of missed detections.

• ID sw(↓): number of times an ID switches to a different previously tracked object \[24\].

• Frag(↓): number of fragmentations where a track is interrupted by miss detection

↑ ：表示越高越好，↓ :表示越低越好；

### 4.2 性能评估 ###

使用MOT基准\[6\]测试服务器评估跟踪性能，其中11个序列的地面真实值是保留的。表2比较了提出的方法排序与其他基线跟踪器。为了简单起见，只列出了最可靠的跟踪器，即在精度方面最先进的在线跟踪器，如(TDAM \[18\]， MDP\[12\])，最快的基于批量的跟踪器(DP NMS\[23\])，和所有的附近在线方法(NOMT\[11\])。此外，还列出了激发这种方法的方法(TBD\[20\]、ALEx- TRAC\[5\]和SMOT\[1\])。与其他方法相比，SORT在在线跟踪器上获得了最高的MOTA分数，可以与最先进的NOMT方法相媲美，后者要复杂得多，并且在不久的将来就会使用框架。另外，由于SORT的目标是集中于帧与帧之间的关联，所以丢失目标的数量(ML)是最少的，尽管与其他跟踪器有类似的假阴性结果。此外，由于SORT侧重于通过帧与帧之间的关联来增长tracklets，因此与其他方法相比，它丢失目标的数量最少。

### 4.3 实时性 ###

大多数MOT解决方案的目标是将性能推向更高的准确性，但往往以运行时性能为代价。虽然在离线处理任务时可以容忍较慢的运行速度，但对于机器人和自动驾驶车辆来说，实时性能是必不可少的。图1显示了MOT基准\[6\]上的许多跟踪器的速度和准确性。这表明方法实现最好的ac -副牧师的职务也往往是最慢的(图1中右下角)。在另一端的频谱的最快方法较低精度(图1中左上角)。结合了两个可取的属性,速度和准确性,没有典型的缺点在图1(右上角)。跟踪组件运行在260 hz英特尔i7 2.5 ghz的单核机16 GB的内存。

## 参考文献 ##

论文地址：[https://arxiv.org/abs/1602.00763][https_arxiv.org_abs_1602.00763]  
github地址：[https://github.com/abewley/sort][https_github.com_abewley_sort]

https://blog.csdn.net/c20081052/article/details/86500352  
https://blog.csdn.net/zjc910997316/article/details/83962462

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191208220332942.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191208224211232.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191208232430113.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA0MTQ1ODk=,size_16,color_FFFFFF,t_70
[https_arxiv.org_abs_1602.00763]: https://arxiv.org/abs/1602.00763
[https_github.com_abewley_sort]: https://github.com/abewley/sort