Publikation
VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale
Christoph Schuhmann; Robert Kaczmarczyk; Gollam Rabby; Felix Friedrich; Maurice Kraus; Gijs Wijngaard; Kourosh Nadi; Huu Nguyen; Kristian Kersting; Sören Zepezauer
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2609.32016, Pages 1-31, arXiv, 2026.
Zusammenfassung
Expressive speech synthesis has outpaced expressive speech perception: systems now ren-
der fine-grained vocal performances that no public benchmark can score. Most benchmarks
for this inverse problem stop at six to nine basic emotion categories, largely on acted speech.
This paper introduces VOICENET, a human-annotated representation-level benchmark for voice
performance understanding on permissively-licensed in-the-wild speech. VOICENET has two
subsets: VOICENET-EMO applies a 40-emotion taxonomy with three expert ratings per item, and
VOICENET-EXT, a preliminary subset, scores 57 talking-style attributes including speaking rate,
vocal tension, breathiness, and register. The paper also releases EMOLIA, an emotion-annotated
version of the Emilia corpus, with a curated rebalanced subset enriched by dense MOSS-Audio
Thinking annotations. Two voice-text contrastive baselines train on this data: a 110M-parameter
VOICECLAP-SMALL for fast large-scale data filtering and a 7B VOICECLAP-LARGE for state-
of-the-art performance. Both outperform existing CLAP baselines, which sit near chance on
VOICENET-EMO. On VOICENET-EMO, VOICECLAP-LARGE aligns more closely with the
aggregate expert consensus than individual experts agree with one another: a comparison against
the majority label rather than evidence of surpassing human emotion perception. All systems
evaluated here are voice-text embedding models: VOICENET scores representation-level attribute
recognition and retrieval, not end-to-end spoken-dialogue behaviour. Clustering and filtering
uncurated speech corpora into subsets that span diverse talking styles and emotions remains
an open challenge; VOICECLAP embeddings offer a promising tool for this task. VOICENET,
EMOLIA, and VOICECLAP are publicly available for research use.