The only answer to this is DON'T LEAVE ON FRIDAY! You can easily add three hours to your trip in just the stretch between the DE bridge and Richmond.
If you must leave Friday, do it early. "Rush hour" traffic in DC over Labor Day weekend starts around 2:00 p.m. You don't say where in NJ you are, but try to plan it so you get to the end of the turnpike and cross the bridge no later than 10:00, earlier if you can.
If that is not an option, just plan to be delayed and prepare the family. It's frustrating to roll along the freeway at 20 mph, but you may have to do that if you hit the Baltimore/DC area in the late afternoon.
Hope this helps!