Well, you HAVE to be out in time for them to setup for late seating, so I think your plans are okay.
We are late seating folks because we like a leisurely meal. DS was 16 and into doing magic tricks on our Magic cruise. We were rarely out of the dining room before 10:15pm. He would do a trick for the servers, then one of the servers would do a magic trick and try and one up him.....a couple days into the cruise we had a pretty large group of passengers AND servers around our table after dessert every night, I think one night we were there until almost 11 pm!