Understanding LSTMs – Part 4: How LSTM Decides What to Forget

LSTMs use a forget gate mechanism to control how much long-term memory is retained during processing. When input values are processed through a sigmoid activation function, the output determines what percentage of existing memory to keep—positive inputs near 1 retain most memory, while large negative inputs near -10 produce outputs close to 0, essentially erasing long-term memory. This selective forgetting represents the first critical stage of LSTM operation, allowing the network to decide which information is worth preserving.