I went when I was three. I have about 5 flashy memory snippets. They are in order of most depressing to most awesome:
1) Me throwing a tantrum in the Village Haus. This is what my parents never let me forget. I had never done a major tantrum in my life and never did since. It figures the one day I would was the day I visited Mickey.
I think they didn't have anything I liked for lunch very much for some reason. I was entirely overstimulated and probably too young to have been there actually, my senses are heightened, especially hearing and that caused me to have a lot of problems as a kid.
2) Peter Pan's flight. Mainly because this and Small World were all I wanted to go on. I remember mainly seeing everyone in the nursery, then my mom was trying to point to Peter Pan's shadow so I'd see him, then the "flying" up out of the window (we were really flying, you know) and seeing London. This is one of two memories I remember as "real". The good news is the ride this past January still gave me amazing warm fuzzies, particularly flying up and over London.
3) I remember a bit of Journey into Imagination, mainly because I was going through a phase with Exit signs at the time. So my mom was a little frustrated because she was pointing at Figment to me, and I kept looking for Exit signs. Yep...
4) I remember the dancing ghosts in the Haunted Mansion, and a sad, big ghost with a frowning face sitting on my lap at the end of the ride. I was apparently fine with the ride up until then, but the ride announcing there was a ghost following us home and having him sitting on me was too much and I lost it a bit, I'm told.
5) I didn't want to go on pirates. I tried resisting, I remember my mom promising it wasn't scary and we'd go back on Peter Pan next but could we do something Daddy wanted to do? The memory I actually remember in the ride itself though is awesome. I remember being in the middle of the ocean, with a real island in flames, and it was so black and vast, and there was a pirate ship out in the water...
So don't be too frustrated with your kids, I guess. Those rides can be real to a little one. I'm wondering if that's why they stuck with me so well.