Publication
On the N-gram Approximation of Pre-trained Language Models
Aravind Krishnan; Jesujoba Alabi; Dietrich Klakow
In: Proceedings of Interspeech 2023. Conference in the Annual Series of Interspeech Events (INTERSPEECH-2023), August 20-24, Dublin, Ireland, Interspeech, 2023.
Abstract
Large pre-trained language models (PLMs) have shown remarkable performance across various natural language understanding (NLU) tasks, particularly in low-resource settings. Nevertheless, their potential in Automatic Speech Recognition (ASR)
remains largely unexplored. This study investigates the potential usage of PLMs for language modelling in ASR. We compare the application of large-scale text sampling and probability conversion for approximating GPT-2 into an n-gram model.
Furthermore, we introduce a vocabulary-restricted decoding method for random sampling, and evaluate the effects of domain difficulty and data size on the usability of generated text. Our findings across eight domain-specific corpora support the
use of sampling-based approximation and show that interpolating with a large sampled corpus improves test perplexity over a baseline trigram by 15%. Our vocabulary-restricted decoding method pushes this improvement further by 5% in domain-specific settings.