Evaluating Audio Reasoning with Big Bench Audio

A new dataset, Big Bench Audio, evaluates audio language models' reasoning abilities. Tests reveal a "speech reasoning gap" between text and audio performance for models like GPT-4o, with accuracy dropping from 92% on text-only questions to 66% for Speech to Speech. The dataset comprises 1,000 audio questions in four categories, testing logical reasoning and language comprehension. Currently, traditional pipeline approaches surpass native Speech to Speech models in complex reasoning tasks.