When we went to DLR as a family of 10 in November, it was the first trip to anything Disney for 9 of them. I was the only one who'd been before, and the last time I went was in high school, 20 years ago. It was just as magical for my 59 year old FIL as it was for me 6 year old nephew. My 39 year old husband who had been poo-pooing Disney for years, saying there was no reason to go and it couldn't be that great, started talking about our next trip before that one was even over! It really is magic.
We had an amazing time, and I loved watching the kids (and other adults!) experience the magic for the first time, but my favorite day was our last day, when we all broke off into our nuclear families. That meant just DH and I, and it was fantastic. We got tired, hot and hungry, so we left midday to go back to the rental house to eat, nap and cool off, then headed back after a while. Getting away from the crowd, having quiet time - it was fantastic, and not at all something we could have done with kids.
I guess that means I agree with all those who said the best part is being able to escape and be the one who makes all the decisions without having to worry about what anyone else wants to do!