We did the CG brunch this past Sunday. This was our second time doing the brunch and is still, in our opinion, hands down the BEST dining experience we've ever had at WDW! As far as how dressed up people are, we saw the full gamut... from dresses and sport coats to shorts and sneakers, so if that's what is preventing you from doing the brunch, don't worry about that at all! You do want to take your time with the CG brunch as it is a full experience. The GF Cafe will definitely be much shorter (about an hour still) than the brunch (you set the pace, so it all depends, but we took more than an hour- about 2) and definitely be less expensive (literally by hundreds of dollars).
Some additional items to consider: if you're driving, CG brunch includes valet parking. Now, obviously its intended to only be for the brunch, but we actually hopped on the monorail to look at the Christmas decorations at the other resorts and picked up our car after doing the full-circle trip with no issues. I suspect people also use the perk to park for a day at the park, but you didn't hear that from me
Also, with your CG brunch receipt, you are able to return at night to watch HEA fireworks from the CG observation deck, which is a nice perk! Some people prefer to watch HEA in the park because of the castle projections though, so that's something to keep in mind.
...FYI- the CG brunch is now $90 a person. We were on the
DDP, which uses 2 TS meals and includes a cocktail in addition to the unlimited mimosas. Not sure if you would want to indulge that much before going to a park, but it may make your park experience "more magical!" lol