Neural audio codecs: how to get audio into LLMs

Speech LLMs currently lack true audio understanding, typically working by converting speech to text and back rather than processing audio natively, limiting their ability to detect emotions or vocal nuances. Neural audio codecs solve this by compressing raw audio into discrete tokens that language models can process, using techniques like residual vector quantization to reduce 160,000 audio samples per second into manageable representations while preserving speech quality.