You Could Have Come Up with Kimi Delta Attention

Standard softmax attention is powerful but slow, requiring memory that grows with sequence length. Linear attention fixes this by compressing history into a fixed-size matrix, but naively adds new information rather than replacing old associations. Kimi Delta Attention (KDA) solves this by first predicting what the memory already stores for a given key, then writing only the correction, while allowing each memory channel to forget at its own independent rate — making it more precise and flexible than earlier approaches.