I'd do early if you want to just enjoy the feeling of being less crowded in the morning, as well as while it may not be much, but very slightly cooler weather start in the morning compared to if you go later in the morning, or afternoon first thing. Also, I personally am the type to like to go first thing, enjoy for awhile, go back to the hotel when it gets too intensely hot, then go back again in the later afternoon / evening.
Also, if your family would struggle with the idea of EE, I wouldn't suggest doing Hagrid's first. As I would imagine you'd have to get there beyond early in order to be at a decent place to wait in line in order to get in, if you do not want to wait forever in the actual parks, and that would be a lot earlier than just EE.
Besides, my typical move for long wait rides regardless of what park I'm in, is waiting until it's at the very end of the day, before the park closes, and get on right when the night time shows start. That way most of the people are gone for the evening, or watching the night shows while you're in the queue. Now there will still be people waiting in line, but it'll be a lot less then first thing in the morning, or during the course of the day.
