Hollywood is closer to DL than SD. If you were going to go there I would do that before heading to SD.
There is plenty to do in SD. There are two zoos. As adults with no whining children I suggest the Safari Park. It's less zoo more free range. There are some great tours you can get as adults at the safari park.
Seaport village is a nice walk around the water. It is next to the USS Midway. Balboa park has some great museums (natural history, air and space, sports, a couple art museums, science, miniature train) Balboa park is also home to the actual SD zoo. If you aren't into beaches so much as saying you went and saw/walked on a beach... Check out Coronado island or La Jolla. I think both are more adult oriented then kids. La Jolla also has some great restaurants that are again geared more towards kids. Nightlife in the gas lamp district downtown might be interesting for you to check out also. There is also old town. Old town has museums, shops, real Mexican food, and bars.
You might want to check out the red trolley. It has hop on and off stops all over the place. If I think of more I will come back.