Abstract: Active contour-based tracking has been widely used for tracking contours of objects with moving cameras. However, traditional methods are sensitive to noise, partial occlusions, background disturbance and some other factors. With respect to this point, we propose a two-layer hierarchical level set-based tracking framework in which color information and shape are fused sequentially. In the first layer, the initial contour is evolved only with the color feature. A penalty term measuring the correlations between neighboring pixels is added to improve the general region-based level set speed model. Then, the Mahalanobis distance-based discriminant criterion is adopted to determine whether the shape model is needed. If the shape model is not needed, the contours obtained in the first layer are considered as the final tracking results; otherwise, the obtained contours are evolved with the shape constraint continuously in the second layer. For the second layer, a weighted shape distance term (WSDT) is introduced to fuse the global shape information and the local color information. Principal component analysis (PCA) subspace of shape samples is trained off-line and updated using an incremental learning algorithm. The experimental results have demonstrated the effectiveness and robustness of our method.