On whether or not it is worth for you giving up a Disney day to go really depends on your relationship with Disney, how long you'll be there, what you want to do, if you'll be going back to Disney, etc.
Now on the merits of Universal: for a Harry Potter fan it is an absolute must. The two worlds in the parks are amazing in their detail and immersiveness. I especially like just wandering around Diagon Alley and could spend hours there even without doing the rides. Just seeing the dragon burst into flame, drinking butterbeer, walking through knockturn alley, exploring the shops, etc. There's just so much to explore and discover. Park hoppers and Hogwarts Express (in both directions) are a must for a Harry Potter fan and it can be ridden and enjoyed by all ages.
With little kids, I'd also highly recommend Camp Jurassic in Islands of Adventure. It's a giant playground with lots of nooks and crannies to explore, including caves, rope bridges, slides, etc. There's also the Pterandon Fliers which can only be ridden by kids under a certain height.
Someone else also mentioned ET - this is a really great ride that would also be doable by younger audiences. Minions also has a newly expanded land if your kids are interested in that IP.
So there's definitely lots to do that would be worth going to for a day even for a family with small kids. Taking one day to go while already in town for Disney would allow you to see it to check it out without the risk of planning a full vacation there. Now whether it is worth exchanging a Disney day for is something that only you can decide.