Why Adam May Be Hurting Your Neural Network’s Memory
SMRTR summary
Research reveals that the Adam optimizer significantly worsens catastrophic forgetting in neural networks, where models lose previously learned information when learning new tasks. Testing multiple optimizers across various metrics showed SGD consistently outperformed Adam, suggesting practitioners should avoid Adam when catastrophic forgetting is a concern.
SMRTR provides this summary for quick context. The original article belongs to Hacker Noon.
Read the original article