Does the Adam Optimizer Amplify Catastrophic Forgetting?
Researchers investigated whether the choice of optimization algorithm affects catastrophic forgetting in neural networks, finding that modern optimizers like Adam actually cause more forgetting than simpler methods like vanilla SGD. The study also revealed that different metrics for measuring catastrophic forgetting can lead to dramatically different conclusions about algorithm performance.