A couple of places DH and I enjoyed on our adults-only trip:
Kona Cafe at the Poly - A nice, quiet lunch. There were not many children around the day we ate there, although I don't know if that's typical.
Chefs de France in Epcot - We thought this was a very romantic meal. There were children there, but they weren't ours and they didn't bother us at all!
DH and I considered having a night out without the kids on our next trip, so I looked into some of the quieter restaurants with more of an adult feel. It seemed that most of the restaurants at the Swan/Dolphin fit that description. Palio at the Swan would be a nice choice if you also want romantic. If you want more adult, but not necessarily romantic, consider Shula's or Blue Zoo at the Dolphin.