Cross-Architecture Knowledge Distillation of WavLM for Lightweight Speaker Verification[link]
Jungwoo Heo, Hyun-Seo Shin, Chan-Yeong Lim, Kyowon Koo, Seung-Bin Kim, Jisoo Son, Kyung Wha Kim, Ha-Jin Yu
abstract
Self-supervised learning (SSL) has advanced speech representation learning, but most encoders rely on Transformers, limiting deployment due to high computational cost. While model compression and distillation mitigate these issues, they largely preserve Transformer backbones. Previous work, such as SV-Mixer, showed that attention-free students can remain competitive under aggressive compression, yet it relied on distillation methods designed for homogeneous architectures. In this work, we present the first systematic framework for cross-architecture distillation in speaker verification, transferring knowledge from Transformer teachers to Mixer students. Our Task-Guided Learning and Proxy-Aligned Distillation achieves improvements on VoxCeleb, VoxSRC, and VOiCES while maintaining a balance between efficiency and accuracy.
Cross-Architecture Knowledge Distillation of WavLM for Lightweight Speaker Verification[link]
Jungwoo Heo, Hyun-Seo Shin, Chan-Yeong Lim, Kyowon Koo, Seung-Bin Kim, Jisoo Son, Kyung Wha Kim, Ha-Jin Yu
abstract
Self-supervised learning (SSL) has advanced speech representation learning, but most encoders rely on Transformers, limiting deployment due to high computational cost. While model compression and distillation mitigate these issues, they largely preserve Transformer backbones. Previous work, such as SV-Mixer, showed that attention-free students can remain competitive under aggressive compression, yet it relied on distillation methods designed for homogeneous architectures. In this work, we present the first systematic framework for cross-architecture distillation in speaker verification, transferring knowledge from Transformer teachers to Mixer students. Our Task-Guided Learning and Proxy-Aligned Distillation achieves improvements on VoxCeleb, VoxSRC, and VOiCES while maintaining a balance between efficiency and accuracy.