A novel multimodal clustering framework for images with diverse associated text

Abstract

With the enormous growth in the number of images on the web, image clustering has become an essential part of any image retrieval system. Since web images are often accompanied by related text or tags, both visual and textual features can be exploited to improve the precision of web image clustering. Existing clustering methods either utilize them separately in a specific order, or use them simultaneously, but independently. In this work, we propose a new framework, Multimodal Hierarchical Clustering for Images (MHCI), which exploits the coexistence of both visual and textual patterns to establish a relationship between them. We propose textual and visual weights to quantify the relationship established between images and their features. The proposed framework can be applied to a wide variety of image datasets with different characteristics, viz., search results with noisy surrounding text, and tagged images. It can also cluster image search queries and their corresponding clicked images. The respective datasets used include image search results, Flicker (NUS-WIDE), and Clickture (Bing query-log). The proposed framework is shown to be versatile on Clickture dataset, which has not been examined by any of the previous approaches. The experimental results show that MHCI significantly improves the quality of image clusters as compared to existing methods.