For me, it's just the crowdedness of space in the parks (i.e. walkways) as well as lack of availability of seating in places like benches, CS restaurant eating areas, and buses. I prefer a day where I can walk without being stuck behind people or being bumped into. And doing things like taking transportation, eating lunch, and taking a rest are so much nicer when the space isn't crawling with other people. If I want to grab a seat along a wall to look at my park map and take a drink, I like not to search for five minutes; if I want to walk from Tomorrowland to Adventureland, I like to move at a reasonable pace and not be pinned in by people; and if we get lunch at CHH, I like to be able to find a table (yes, we eat at off hours, and yes, I know it has a second story) without standing around with my tray waiting for others to finish.
As other posters mentioned, I also hate listening to other groups complain and argue amongst themselves. That ruins my mood and does tend to be more noticeable when the weather is hot and the parks crowded.
Long ride lines aren't a huge issue for me because we're fairly good at utilizing FP+ and aren't super picking about needing to get on certain attractions. That said, the last time I went on a VERY busy day was July 4, 2009, and that was when we lived in Celebration and were AP holders, so we were super chill about crowds. Now that we live out of state, we have only been twice since 2012 and have tried picking dates that don't put us in the way of crazy crowds.