DS-KCF: A real-time tracker for RGB-D data
S. Hannuna, M. Camplani, J. Hall, M. Mirmehdi, D. Damen, T. Burghardt, A. Paiement, L. Tao,
DS-KCF: A real-time tracker for RGB-D data
Journal of Real-Time Image Processing
C++ and Matlab Code Available
We propose an RGB-D single object tracker, built upon the extremely fast RGB-only KCF tracker that is able to exploit depth information to handle scale changes, occlusions, and shape changes. Despite the computational demands of the extra functionalities, we still achieve real-time performance rates of 35-43 fps in Matlab, and 187 fps in our C++ implementation. Our proposed method includes fast depth-based target object segmentation that enables, (i) efficient scale change handling within the KCF core functionality in the Fourier domain, (ii) the detection of occlusions by temporal analysis of the target's depth distribution, and (iii) the estimation of a target's change of shape through the temporal evolution of its segmented silhouette allows. Finally, we provide an in-depth analysis of the factors affecting the throughput and precision of our proposed tracker and perform extensive comparative analysis. Both the Matlab and C++ versions of our software are available in the public domain.
This paper contains an extension of the DS-KCF proposed in . In this section we provide a detailed description of the core modules comprising DS-KCF, which extend the KCF tracker in a number of different ways. We integrate an efficient combination of colour and depth features in the KCF-tracking scheme. We provide a change of scale module, based on depth distribution analysis, that allows to efficiently modify the tracker's model in the Fourier domain. Different to other works that deal with change of scale within the KCF framework, such as [2,3], our approach estimates the change of scale with minimal impact on real-time performance. We also introduce an occlusion handling module that is able to identify sudden changes in the target region's depth histogram and to recover lost tracks. Finally, a change of shape module, based on the temporal evolution of the segmented target's silhouette, is integrated into the framework. To improve the tracking performance during occlusions, we have added a simple Kalman filter motion model.
A detailed overview of the modules of the proposed tracker is shown in the figure below. Initially, depth data in the target region is segmented to extract relevant features for the target's depth distribution. Then, modelled as a Gaussian distribution, changes in scale guide the update in the target's model. At the same time, region depth distribution is deployed to enable the detection of possible occlusions. During an occlusion, the model is not updated and the occluding object is tracked to guide the target search space. Kalman filtering is used to predict the position of the target and the occluding object in order to improve the occlusion recovery strategy. Further, segmentation masks are accumulated over time and used to detect significant changes of shape of the object.
Block diagram of the proposed DS-KCF tracker
Results on Princeton Dataset  (validation set)
Table below summarizes the results obtained by all the different algorithms , showing the average AUC obtained for all 95 videos and for each of the different video categorizations. Moreover, we report an average ranking of the algorithms (second column) by considering the individual rankings under the different categorizations. The methods in the table are ordered according to the best performance obtained, i.e. by their average rank. The representation in Table is as proposed in . However, as the number of sequences in each category is different and some videos belongs to more than one category, this ranking is not a perfect valid summary of the results. The combined analysis of AUC and Avg. Rank would be more appropriate for a detailed look (see below).
Our proposed method is ranked 3rd (in Matlab), 4th (without shape handling in Matlab) and 5th (without shape handling in C++). The small difference between the accuracies obtained by the MATLAB and C++ implementations of DS-KCF arises since the two implementation are completely independent and use different libraries for completing all the operations, such as the Fourier transformations, matrix operations, and also the computation of the fast segmentation approach. Only the HoG feature extraction module belongs to the same software library.
Trackers’ performance on Princeton Validation Dataset 
Overall, only two approaches, OAPF  and RGBDOcc+OF , obtain a higher Avg. Rank (by approximately 0.6% more) and a higher average AUC value (by approximately 1.4% more). However, as reported by their authors, these approaches achieve a very low processing rate of less than 1 fps. Our proposed method has an average processing rate ranging from 35 to 43 fps for its Matlab implementation and 187 fps for its C++ version.
Updated results can be found in the dataset webpage
C++ DS-KCF code (no shape Module)
The C++ code of the new version of the DSKCF tracker may be used on the condition of citing our paper “DS-KCF: A real-time tracker for RGB-D data. Journal of Real-Time Image Processing″ and the SPHERE project. The code is released under BSD license and it can be downloaded here. Please note that this does not contain the shape-handling module
Matlab DS-KCF code (with shape Module)
The Matlab code of the new version of the DSKCF tracker may be used on the condition of citing our paper “DS-KCF: A real-time tracker for RGB-D data. Journal of Real-Time Image Processing″ and the SPHERE project. The code is released under BSD license and it can be downloaded here
GITHUB DS-KCF code
The C++ and MATLAB version of the code are also available in GITHUB here. Code of the DSKCF tracker may be used on the condition of citing our paper “DS-KCF: A real-time tracker for RGB-D data. Journal of Real-Time Image Processing″ and the SPHERE project. The code is released under BSD license.
The datset can be used for research purposes on the condition of citing our paper “DS-KCF: A real-time tracker for RGB-D data. Journal of Real-Time Image Processing″ and the SPHERE project. The dataset can be downloaded from the University of Bristol official repository data.bris
 Massimo Camplani, Sion Hannuna, Majid Mirmehdi, Dima Damen, Adeline Paiement, Lili Tao, Tilo Burghardt. Real-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling.
 Danelljan M, Haager G, Shahbaz Khan F, Felsberg M (2014) Accurate Scale Estimation for Robust Visual Tracking. In: BMVC 2014
 Li Y, Zhu J (2015) A scale adaptive kernel correlation Filter tracker with feature integration. In: ECCV Workshops, vol 8926, pp 254-265.
 S. Song and J. Xiao. Tracking revisited using RGBD camera: Unified benchmark and baselines. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 233–240, 2013.
 Meshgi K, ichi Maeda S, Oba S, Skibbe H, zhe Li Y, Ishii S. An occlusion-aware particle Filter tracker to handle complex and persistent occlusions. Computer Vision and Image Understanding 150:81-94, 2016