Theoretically, I'd agree. But "memory" is an extremely fuzzy definition. Perhaps the memory is not 100% identical, but missing a tiny detail about the color of the triangle on the guy's shirt in the background of the TV.
There is also a lot of redundancy in neural networks. Large swaths of neurons fire together in concerted patterns, and while removing one may tweak the pattern slightly, it is equally likely that the other neurons will just adjust their weights to keep the pattern constant.
There is also a lot of redundancy in neural networks. Large swaths of neurons fire together in concerted patterns, and while removing one may tweak the pattern slightly, it is equally likely that the other neurons will just adjust their weights to keep the pattern constant.