Abstract: Content-based video retrieval (CBVR) has attracted increasing interest in recent years. In this paper, we propose a new interactive video retrieval framework using semantic matching. The main contributions are three-fold: 1) We define a novel high-level feature named semantic-matching histogram (SMH) to reflect videos' semantic information. 2) We set up an unsupervised learning-based retrieval mechanism using the dominant set clustering for the sake of low on-line complexity and high retrieval efficiency. 3) We establish a new interactive mechanism called semantic-based relevance feedback (SBRF) working together with SMHs to improve retrieval performances. Experimental results on a database of sports videos show the effectiveness and efficiency of the proposed framework.