RGB-D Tracking: Depth Scaling Kernelised Correlation Filters DS-KCF

Massimo Camplani, Sion Hannuna,  Majid Mirmehdi, Dima Damen, Adeline Paiement, Lili Tao, Tilo BurghardtReal-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling
British Machine Vision Conference, September 2015.

AN EXTENDED VERSION OF THE PAPER AND CODE CAN BE FOUND HERE 
THE GITHUB REPOSITORY OF THE PAPER CAN BE FOUND HERE

The recent surge in popularity of real-time RGB-D sensors has encouraged research into combining colour and depth data for tracking. The results from a few, recent works in RGB-D tracking have demonstrated that state-of-the-art RGB tracking algorithms can be outperformed by approaches that fuse colour and depth, for example [1, 3, 4, 5]. In this paper, we propose a real-time RGB-D tracker which we refer to as the Depth Scaling Kernelised Correlations Filters (DS-KCF). It is based on, and improves upon, the RGB Kernelised Correlation Filters tracker (KCF) from [2]. In this paper, we propose a real-time RGB-D tracker which we refer to as the Depth Scaling Kernelised Correlations Filters (DS-KCF). It is based on, and improves upon, the RGB Kernelised Correlation Filters tracker (KCF) from [2]. KCF is based on the use of the ‘kernel trick’ to extend correlation filters for very fast RGB tracking. The KCF tracker has important characteristics, in particular its ability to combine high accuracy and processing speed as demonstrated in [2, 6].

DS-KCF

The proposed DS-KCF tracker1extends the RGB KCF tracker in three ways: (i) we employ an efficient combination of colour and depth features (ii) we propose an efficient a novel management of scale changes and (iii) occlusions handling. The improvements we implement provide higher rates of accuracy while still operating at better than real time frame rates (35fps on average ). In particular, depth data in the target region is segmented with a fast K-means approach to extract relevant features for the target’s depth distribution. Modelled as a Gaussian distribution, this data allows to identify scale changes and efficiently model them in the Fourier domain. The advantage of the proposed approach is that only a single target model is kept and updated. Furthermore, region depth distribution enables the detection of possible occlusions identified as sudden changes in the target region’s depth histogram, and recovering lost tracks by searching for the unoccluded object in specifically identified key areas. During an occlusion, the model is not updated and the occluding object is tracked to guide the target’s search space.

Block diagram of the proposed DS-KCF tracker


Results on Princeton Dataset [4] (validation set)

We compare tracking performance by reporting the precision value for an error threshold equal to 20 pixels (P20), the area under the curve (AUC) of success plot measure, and the number of processed frames per second (fps). Table 1 shows that the proposed DS-KCF tracker outperforms the baseline KCF leading to better results both in terms of AUC and P20 measures. DS-KCF also outperforms the other two RGB-D trackers tested, Prin-Track [4] and OAPF [3]. Furthermore, the average processing rate in the Prin-Track (RGB-D) is 0.14fps and 0.9fps for the OAPF tracker in striking contrast to 40fps for DS-KCF. Example results of the trackers are shown in the videos below. Results on the test set of the Princeton dataset can be found here

Trackers’ performance on Princeton Validation Dataset [4]

Average precision plot (left) and average success plot (right) for Princeton Dataset [4]


Qualitative examples of the DS-KCF's performance on Princeton Dataset [4] and BoBoT-D dataset [1] are shown below.

   
  

DS-KCF Matlab code

The Matlab code of the DSKCF tracker may be used on the condition of citing our paper “Real-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling, BMVC2015″ and the SPHERE project. The code is released under BSD license and it can be downloaded here

GITHUB DS-KCF code  

The C++ and MATLAB version of the code are also available in GITHUB here. Code of the DSKCF tracker may be used on the condition of citing our paper  “Real-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling, BMVC2015″ and the SPHERE project. The code is released under BSD license.

 

References

[1] G. García, D. Klein, J. Stückler, S. Frintrop, and A. Cremers. Adaptive multi-cue 3D tracking of arbitrary objects. In Pattern Recognition, pages 357–366. 2012.
[2] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High-speed tracking with kernelized correlation filters. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2015.
[3] T. Meshgi, S. Maeda, S. Oba, H. Skibbe, Y. Li, and S. Ishii. Occlusion aware particle filter tracker to handle complex and persistent occlusions. Computer Vision and Image Understanding, 2015 to appear.
[4] S. Song and J. Xiao. Tracking revisited using RGBD camera: Unified benchmark and baselines. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 233–240, 2013.
[5] Q. Wang, J. Fang, and Y. Yuan. Multi-cue based tracking. Neurocomputing, 131(0):227 – 236, 2014.
[6] Y. Wu, J. Lim, and M. Yang. Online Object Tracking: A Benchmark. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 2411–2418, 2013.