Spatio-Temporal_Point_Process_for_Multiple_Object_Tracking__GoldenDB.pdf

tinge

12页

0次

2023-10-26

免费下载

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023 1777

Spatio-Temporal Point Process for

Multiple Object Tracking

Tao Wang , Kean Chen, Weiyao Lin , Senior Member, IEEE, J ohn See , Senior Member, IEEE,

Zenghui Zhang , Senior Member, IEEE,QianXu ,andXiaJia

Abstract— Multiple object tracking (MOT) focuses on

modeling the relationship of detected objects among consecutive

frames and merge them into different trajectories. MOT remains

a challenging task as noisy and confusing detection results

often hinder the ﬁnal performance. Furthermore, most existing

research are focusing on improving detection algorithms and

association strategies. As such, we propose a novel framework

that can effectively predict and mask-out the noisy and confusing

detection results before associating the objects into trajectories.

In particular, we formulate such “bad” detection results as a

sequence of events and adopt the spatio-temporal point process

to model such events. Traditionally, the occurrence rate in a

point process is characterized by an explicitly deﬁned intensity

function, which depends on the prior knowledge of some speciﬁc

tasks. Thus, designing a proper model is expensive and time-

consuming, with also limited ability to generalize well. To tackle

this problem, we adopt the convolutional recurrent neural net-

work (conv-RNN) to instantiate the point process, where its

intensity function is automatically modeled by the training data.

Furthermore, we show that our method captures both temporal

and spatial evolution, which is essential in modeling events for

MOT. Experimental results demonstrate notable improvements

in addressing noisy and confusing detection results in MOT

data sets. An improved state-of-the-art performance is achieved

by incorporating our baseline MOT algorithm with the spatio-

temporal point process model.

Index Terms— Multiple object tracking, recurrent neural

networks, spatio-temporal point processes.

I. INTRODUCTION

ULTIPLE object tracking (MOT) is one of the funda-

mental problems in computer vision, which is important

in many applications like intelligent video sur veillance, behav-

ior analysis, automatic driving, and robotics. MOT constitutes

Manuscript received 15 October 2019; revised 1 March 2020 and 3 April

2020; accepted 14 May 2020. Date of publication 8 June 2020; date of

current version 5 April 2023. This work was supported in part by the China

Major Project for New Generation of AI under Grant 2018AAA0100400,

in part by the National Natural Science Foundation of China under

Grant 61971277, in part by ZTE Industry-Academia-Research Cooperation

Funds and State Key Laboratory of Mobile Network and Mobile Multimedia

Technology, and in part by CREST Malaysia under Grant T03C1-17.

(Corresponding author: Weiyao Lin.)

Ta o Wang, Kean Chen, Weiyao Lin, and Zenghui Zhang are with

the Department of Electronic Engineering, Shanghai Jiao Tong Uni-

versity, Shanghai 200240, China (e-mail: wang_tao1111@sjtu.edu.cn;

ckadashuaige@sjtu.edu.cn; wylin@sjtu.edu.cn; zenghui.zhang@sjtu.edu.cn).

John See is with the Faculty of Computing and Informatics, Multimedia

Uni versity, Cyberjaya 63100, Malaysia (e-mail: johnsee@mmu.edu.my).

Qian Xu and Xia Jia are with ZTE Corporation, Shenzhen 518057, China

(e-mail: xu.qian5@zte.com.cn; jia.xia@zte.com.cn).

Color versions of one or more of the ﬁgures in this article are available

online at https://ieeexplore.ieee.org.

Digital Object Identiﬁer 10.1109/TNNLS.2020.2997006

Fig. 1. Example of a noisy object detection. (a) Detection results, where a

potential noisy detection (red box) occurs in the scene. (b) As a result, the red

trajectory is unnecessarily tracked. This is an incorrect result. (c) Our approach

can predict the area which is likely to contain the noisy detections and avoid

them in the association process. (d) Tracking result using our method, where

the dashed line boxes are generated by linear interpolation. (e) Prediction is

generated by the proposed point process model.

the task of modeling the relationship of detected objects among

consecutive frames and then merging them into d ifferent

trajectories [1]–[3]. This task remains challenging, one major

issue is that some “bad” detection results always hinder the

performance of MOT. In general, such “bad” results can be

divided into two different types: 1) noisy detection results, i.e.

false positives in object detection, and 2) confusing detection

results, i.e. two highly overlapping objects with similar appear-

ances. Two such examples are shown in Figs. 1 and 2. It can be

seen that both the noisy and confusing objects had misguided

the matching process. Furthermore, we observe that most of

the failure examples in MOT are caused b y them, directly or

indirectly. Hence, the main problem in MOT that needs to be

addressed can realistically be reduced to: how to effectively

handle these “bad” detection results?

Some existing methods in MOT such as [1] and [3]–[10]

improved the tracking performance by introducing more robust

object association strategies. More speciﬁcally, some advanced

techniques for cost function and corresponding optimization

algorithm have been developed to associate objects in different

frames. However, these methods do not explicitly model

the “bad” detection results, so they can be confused by

the objects with high similarity in appearance and motion.

Other methods including [11]–[16] focused their attention

on different feature representations and metrics for detected

objects. These methods have better accuracy in normal scenes

but are still affected by the noisy detection results. Some works

See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: ZTE CORPORATION. Downloaded on October 26,2023 at 06:35:56 UTC from IEEE Xplore. Restrictions apply.

1778 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 4, APRIL 2023

Fig. 2. Example of a confusing object detection. (a) Detection results.

(b) Incorrect tracking result due to objects that are overlapping in close

proximity (yellow box and blue box). (c) Our approach can predict the

area which is likely to contain the confusing objects and avoid them in the

association process. (d) Tracking result by our method, where the dashed line

boxes are generated by linear interpolation. (e) Prediction is generated by the

proposed point process model.

like [17]–[22] adopted more accurate object detectors in

attempt to reduce these noisy detections, while the perfor-

mances are still hindered by confusing detections such as the

highly overlapping objects.

To tackle this issue in MOT, we propose a framework that

can effectively predict and mask-out these “bad” detection

results before associating the objects into trajectories. First,

we note that the “bad” detections can be formulated as

a sequence o f events that happen in d ifferent frames and

locations. Thus, we need to infer when and where these events

are likely to happen, given the feature of current frame and

the historical behavior of the detector as prior information.

More speciﬁcally, we model these events based on the pix-

els that are inside the bounding boxes of “bad” detections.

Such events are distributed across the spatio-temporal tube

of a video and are generated based on complicated latent

mechanisms [23], [24], which is hard to capture through

simple modeling. Because these events happen in the motion

of objects, so there exists relationship between these events

which actually reﬂects the motion information of the objects.

In other words, we can detect these bad detections more

accurately and improve the performance of MOT by obtaining

the relationship between these events. Moreover, we make the

following observations.

1) Noisy object detections are more likely to appear in the

area where there are already some noisy detections in

the previous frames.

2) If there are confusing detections among people in a

group who walk closely or dress similar to others,

confusing d etections are more likely to appear in these

people in the subsequent frames.

In this article, we introduce the use of spatio-temporal point

process to deal with such events.

Point process [25] is a powerful tool for modeling the real-

world sequ e ntial data, which has lots of applications in many

ﬁelds, such as ﬁnance [26], equipment maintenance [27], [28],

and social n etwork [29], [30]. A point process is

characterized by its conditional intensity function, which

presents the occurrence rates of some class of events con-

ditioned on the historical events. Traditionally, the intensity

function can be explicitly deﬁned based on prior knowledge of

event data and latent mechanisms of the process. The intensity

function usually consists of two parts [31]: an exogenous

intensity that describes factors driven by the inherent and

often time-varying occurrence rate for a type of events; an

endogenous intensity which describes the triggering effect

from the previous events. This parametric strategy has been

widely adopted in many classic models, such as Poisson

processes [32], Hawkes processes [33], and self-correcting

processes [34]. However, there are three issues that need to

be handled in the case of MOT.

1) Designing a proper parametric model is expensive and

time-consuming, since it requires expert domain knowl-

edge and experience if it were to be modeled manually.

Besides, the generalization ability is also limited.

2) It is difﬁcult to properly capture the dynamics of inﬂu-

ence from historical information given the complicated

nature of data patterns in MOT task. On the other

hand, it is also hard for traditional point processes

to incorporate other heterogeneous data, such as time

series [31] associated with event sequences.

3) Furthermore, the detection events in MOT occur in dif-

ferent frames and different locations in the scene, which

requires capturing both temporal and spatial evolution.

To tackle these problems, we adopt the spatio-temporal

point process to model the detection events, where a convo-

lutional recurrent neural network (conv-RNN) is proposed to

instantiate the point process. The main characteristics of our

method are shown below and in Fig. 3.

1) In our method, the intensity function is automatically

modeled by the training data, without requiring expert

knowledge and experience.

2) We propose a two-stream RNN framework to handle

two different inputs, i.e., time series and event sequence,

with a n ovel time-evolving mechanism to align and

merge these two input data. This enables our model

to capture the complex dynamics of inﬂuence from

historical information.

3) We incorporate the use of convolution operation in

RNN, which enables spatial diffusion for the historical

information of events.

Coupled with the capacity of RNN in modeling temporal

dependence, the proposed conv-RNN-based point process is

equipped with the capability of capturing both temporal and

spatial evolution. The main contributions of this article are

summarized as follows.

1) We propose a novel framework in MOT that can effec-

tively predict and mask-out the noisy and confusing

detection results before associating the objects into tra-

jectories. The “ bad” detection results are formulated as

a sequence of events and are modeled by the spatio-

temporal point process.

2) We introduce a two-stream pipeline to handle two syn-

chronous and asynchronous inputs (i.e., time series and

event sequence) with a novel-evolving mechanism for

merging.

Authorized licensed use limited to: ZTE CORPORATION. Downloaded on October 26,2023 at 06:35:56 UTC from IEEE Xplore. Restrictions apply.

of 12

免费下载

goldendb

关注

评论