I really can't fathom getting annoyed because some group took their time instead of whatever someone else thinks is appropriate. Just who thinks their rules should apply to all? Really do not understand that concept, that the time YOU feel is appropriate is appropriate for all guests. Like that guest is going to seriously impact your day because they took longer than you allot for yourself. You aren't talking about 30 min or anything. It's 2, 3 min more, tops.
I've encountered this situation a number of times myself. I can't fathom a situation where a family would say no, they can't go in front of us.
Like you, I enjoy watching other groups interact with the characters too.