SEAMLESSEXPRESSIVELM Unifies Semantic & Acoustic Modeling for Efficient Speech Translation

SEAMLESSEXPRESSIVELM is a new language model for style-transferred speech-to-speech translation. It uses HuBERT and EnCodec to convert speech into discrete units, preserving both semantic and acoustic information. The model's architecture includes embedding layers and combines autoregressive and non-autoregressive components. During training, it uses an acoustic prompt and chain-of-thought approach. At inference, the model decodes semantic units with beam search and generates acoustic units with temperature sampling. This technology could potentially improve speech translation while maintaining speaker style and intonation.