If that were the case, you wouldn’t just remember things a little wrong, you’d try and recall your name and instead be remembering a field trip you took in 3rd grade.
You are implying that the process must be error-prone? I don’t see how that follows.
Ok, so given that memory is error prone, value makes more sense than reference, because errors of reference would be more errory.
That makes assumptions about how the referred-to stuff is arranged. It assumes no organization. That memories about lunch would be kept right next to the Chemistry lessons.
So, to step away from that assumption, maybe memory-components are more organized. Gradients of meaning, say.
And maybe memory-components are less chunky. Instead of a memory of lunch, it’s memory of sandwich, table and chewing, arranged appropriately in the referencing data.