Catastrophic forgetting sounds like a dramatic name for a technical problem. It earns it.
When a neural network learns something new, it doesn't file the new information away in a separate folder. It adjusts the same weights that encode everything it already knows. New learning and old learning compete for the same space. When the new learning wins — which it tends to, because that's what the training process is optimizing for — the old knowledge gets overwritten. Not partially. Often entirely.
The instinctive fix is to make the network more resistant to change. If the weights that encode old knowledge are harder to modify, new training can't erase them. This works, up to a point. But a network that's resistant to changing its weights is also resistant to learning. You've solved the forgetting problem by creating a rigidity problem.
This is the stability-plasticity dilemma, and it's not a quirk of any particular architecture. It's a fundamental tension in how learning systems work. A system optimized for stability — for retaining what it knows — will struggle to incorporate genuinely new information. A system optimized for plasticity — for learning quickly from new data — will keep forgetting what it learned before. You can tune the dial in either direction, but you can't easily have both at once.
The dilemma isn't unique to AI. Neuroscientists have studied versions of it in biological brains for decades, and the human brain's solution — consolidating memories during sleep, using different systems for different types of memory, gradually integrating new information rather than overwriting old — is impressively sophisticated. AI systems don't yet have anything equivalent that works reliably at scale.
This is why continual learning, lifelong learning, and online learning are active research areas rather than solved problems. Each represents a different angle on the same underlying challenge: how do you build a system that can keep learning without losing what it already knows? The approaches are creative. The problem remains genuinely hard.
