Making Transformers Smarter: A Memory Boost for Symbolic Tasks

The paper presents "address attention," a new mechanism enhancing sequence models' capacity for longer inputs. This approach employs physical pointers and an address bank to emulate symbolic rule-based operations, improving generalization to longer sequences. Address attention can adapt to growing memory during inference and is content-independent, unlike traditional attention methods. The proposed technique shows potential for improving length extrapolation across benchmarks while maintaining standard training procedures.