You Could Have Come Up with Kimi Delta Attention

Standard softmax attention requires storing every past key and value, creating memory that grows with sequence length. Linear attention fixes this by compressing history into a fixed-size matrix, but blindly adding new information corrupts old associations. DeltaNet solves this by writing only the prediction error instead of raw values, while Kimi Delta Attention extends that idea further by allowing each memory channel to independently control how much old information it retains, enabling more precise, fine-grained forgetting across the hidden state.