Publikation
Listening for Speaker Identity in Multilingual Speech Models: Phonetic Similarity Predicts Transfer in Cross-Lingual Speaker Verification
AriaRay Brown; Yusser Al Ghussin; Dietrich Klakow
In: Mourad Abbas; Abed Alhakim Freihat (Hrsg.). Proceedings of the 9th International Conference on Natural Language and Speech Processing (ICNLSP-2026). International Conference on Natural Language and Speech Processing (ICNLSP), University of Trento, Trento, Italy, Pages 437-447, ISBN 979-8-89176-458-3, Association for Computational Linguistics, 2026.
Zusammenfassung
Multilingual self-supervised transformer
speech models build shared representations
to encode speech, but how speaker identity is
shared across languages, and what modulates
its transfer, remains unclear. We probe XLS-R
and WavLabLM for speaker information in the
task of cross-lingual speaker verification using
layer-weighted speech embeddings from ten
languages spanning four families. Motivated
by the Language Familiarity Effect (LFE)
in human speech perception, we investigate
whether phonetic similarity modulates the suc-
cess of cross-lingual transfer. We introduce a
model-internal method for computing phonetic
similarity: applying the Sinkhorn distance
between per-language distributions of contex-
tualized phone embeddings at a phonetically
prominent model layer. A linear mixed-effects
model across all 90 language pairs shows that
this phonetic distance significantly predicts
cross-lingual speaker-verification accuracy in
both XLS-R (β=−3.13, p < 0.0001) and
WavLabLM (β=−1.22, p<0.0001). These
findings suggest that phonetic similarity is an
important factor for facilitating cross-lingual
transfer of speaker information in multilingual
self-supervised speech models.