Exploiting Visual and Textual Neighborhood Information to Improve Image-Tag Relevance

Abstract

Many applications, such as image searching, image indexing, and image label recommendations, have started using tagged images to benefit from user input. However, tags tend to be imprecise, incomplete, and ambiguous. Moreover, tags are also biased towards the user's perspective which degrades the performance of tag-based systems. Most of the existing methods use visual neighborhoods and/or tags to estimate image-tag relevance. We improve image-tag relevance by combining visual neighborhood of images and textual neighborhood of tags. By doing this, we boost the ranking of informative tags of an image. Most of the image-tag relevance measures work well when large supporting data is available, which is typically not sufficient in real datasets. This problem of Void of Information (VoI) is addressed by exploiting tags of visual neighbors of the images. We also exploit external resources like Wikipedia and WordNet to strengthen the tags. The proposed approach, TVNTag (Textual Visual Neighborhood based Tag) exhibits up to 46.1% relative improvement in tag ranking and 79.5% in image ranking, with respect to the current state-of-the-art methods. The experiments are conducted for different tasks and evaluation scenarios on benchmarked social data, such as MIRFlickr, NUS-WIDE, and train10k.