Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

Xiaoting Wu, Xueyi Zhang, Xiaoyi Feng, Miguel Bordallo Lopez, Li Liu

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Facial kinship verification refers to automatically determining whether two people have a kin relation from their faces. It has become a popular research topic due to potential practical applications. Over the past decade, many efforts have been devoted to improving the verification performance from human faces only while lacking other biometric information, for example, speaking voice. In this article, to interpret and benefit from multiple modalities, we propose for the first time to combine human faces and voices to verify kinship, which we refer it as the audio-visual kinship verification study. We first establish a comprehensive audio-visual kinship dataset that consists of familial talking facial videos under various scenarios, called TALKIN-Family. Based on the dataset, we present the extensive evaluation of kinship verification from faces and voices. In particular, we propose a deep-learning-based fusion method, called unified adaptive adversarial multimodal learning (UAAML). It consists of the adversarial network and the attention module on the basis of unified multimodal features. Experiments show that audio (voice) information is complementary to facial features and useful for the kinship verification problem. Furthermore, the proposed fusion method outperforms baseline methods. In addition, we also evaluate the human verification ability on a subset of TALKIN-Family. It indicates that humans have higher accuracy when they have access to both faces and voices. The machine-learning methods could effectively and efficiently outperform the human ability. Finally, we include the future work and research opportunities with the TALKIN-Family dataset.
Original languageEnglish
Pages (from-to)1-14
Number of pages14
JournalIEEE Transactions on Cybernetics
DOIs
Publication statusE-pub ahead of print - 2022
MoE publication typeA1 Journal article-refereed

Keywords

  • Adversarial learning
  • audio-visual
  • benchmark dataset
  • kinship verification
  • multimodal fusion

Fingerprint

Dive into the research topics of 'Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach'. Together they form a unique fingerprint.

Cite this