Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons

A new architecture for large language models tackles the challenge of processing extremely long contexts efficiently. By combining state space blocks, multi-resolution convolutions, a recurrent supervisor, and retrieval-augmented memory, the model avoids attention mechanisms that typically cause quadratic computational growth. This approach potentially enables handling contexts of hundreds of thousands or even millions of tokens with near-linear scaling.