PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Cross-Media Alignment of Names and Faces
Phi T. Pham, Marie-Francine Moens and Tinne Tuytelaars
IEEE Transactions on Multimedia Volume 12, Number 1, pp. 13-27, 2010.


In this paper we report on our experiments on aligning names and faces as found in images and captions of online news websites. Developing accurate technologies for linking names and faces is valuable when retrieving or mining information from multimedia collections. We perform exhaustive and systematic experiments exploiting the (a)symmetry between the visual and textual modalities. This leads to different schemes for assigning names to the faces, assigning faces to the names, and establishing name-face link pairs. On top of that, we investigate generic approaches to the use of textual and visual structural information to predict the presence of the corresponding entity in the other modality. The proposed methods are completely unsupervised and are inspired by methods for aligning phrases and words in texts of different languages developed for constructing dictionaries for machine translation. The results are competitive with state of the art performance on the ”Labeled Faces in the Wild” dataset in terms of recall values, now reported on the complete dataset, include excellent precision values, and show the value of text and image analysis for identifying the probability of being pictured or named in the alignment process.

EPrint Type:Article
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Machine Vision
Multimodal Integration
ID Code:6588
Deposited By:Tinne Tuytelaars
Deposited On:08 March 2010