Living in NC, you see a fair number of television ads for the Biltmore and for Asheville. The time of year to visit, really depends on what you want to do. I have been during the summer, late fall/early winter and then late winter (before things started blooming for Spring).
During the November/December period, the Biltmore has the Candlelight tour and all of the Christmas decorations up. It is really breathtaking. Also there is the Gingerbread display at the Grove Park Inn during the Christmas Holidays. During the Summer, it can be a little escape from heat and humidity, depending on where you are coming from. If you like doing outside things, then Spring or even early Fall (with the color changes in the Blue Ridge) can be fun. The gardens at Biltmore are beautiful when all of the flowers are blooming.
Downtown Asheville is really nice and there are many places to eat/drink. You may want to go review the official Asheville travel website to see what events are going on through out the year and let that help plan your trip. This may cause the opposite affect of making things busier, if you want to escape some crowds.
There is so much to do in Asheville outside of the Biltmore. But I would suggest doing the Biltmore at least once. I have done a couple of the special tours and they were really nice and you got to go to rooms and areas typically not available to the public. If you ever watched Downton Abbey, then you can see similarities with the Biltmore.
