Here is my 2c..
Pajamas are never appropriate in public. That looks sloppy and trashy. Sweatpants are fine if they are nice sweatpants (Abercrombie, American eagle, Aerie, Hollister, etc), not the Hanes ones. Yoga pants are also fine.
For myself, I remember wearing them for elem school for gym days only. MS would wear Adidas track pants sometimes, but I always felt that they looked bad so I wouldn't wear the, too often. I went to a HS with a uniform so I never wore sweats then, not even gym days because we had gym uniforms as well. I wore sweats in college all the time for undergrad, but never pajamas. For grad school every once in a while I will wear togas, but not very often. I find it just as easy to put jeans on than sweats and they are just as comfortable.
I will say that I once wore a pair of jeans to school and my friends made fun of them so I never wore them again. MS girls are blunt and mean, often without realizing it.