I don't know if I would say I'm in the 'opposite' camp, but this is my whole thing...I am no where near interesting enough for someone to want to secretly listen into my conversations. We use our Google Home Mini mainly for weather, set timers, check the time, play music, and ask for an occasional recipe. Even if this thing was listening to me while I was just talking to my wife, no one would care what I have to say about what happened to me at work, how I woke up with a stomach ache, what TV show we should start, etc. I'm simply not interesting enough. I'm sure there are people in the world that are interesting enough to tap...I just think the VAST majority of people aren't.
And all of this, of course, does not invalidate anyone else's opinion. I'm just giving my perspective on why these things don't bother me.