If it were up to me trust me I'd be spending more days in the parks lol I considered going there on our own without the tour but we don't speak Japanese at all and we concluded we're not that adventurous yet
I know this is the ugly American in me, but I've never had too much of a problem going anywhere and not speaking the language. Yeah, it is a bit more difficult, but you can pretty much get across what you need to in most situations. You can find out online what places have English translations, subtitles etc. Some of the shows don't, but from what I've read you can get the gist of it either way. Cities like Tokyo, where they've had the Olympics before, tend to be very accommodating to English speakers, because oftentimes they have both the Japanese and English written out at major points of interest, especially public transportation.
It's really up to you though. I've been struggling myself with the concept of spending most of my time in Japan at Disney. Yes, it is most of what I want to see there, as I live in South Korea right now, so I can see similar temples and shrines on day trips from my house, but I don't want to waste a trip to Japan. I know that everywhere has distinct cultures and their own things to offer as travel destinations. Originally, I wanted to go and spend close to 2 weeks in Japan and see quite a bit more, but we ran into some financial trouble and I'm having a hard enough time justifying going at all. As of now, I'm trying to work out a budget under $3000 for 2 adults and a toddler, and it's hard if I plan on traveling outside of Tokyo at all. The only thing that has me continuing to consider it is the fact that I hate jet lag and it will only be more expensive in the future. It would be 10 times easier and likely cheaper to do it from here with one child than it will be in the future from the States with possibly more kids. However, I'm giving up some of the other things I want to do. Maybe one day I'll be able to go back and do those things, but I don't know. It's hard to decide whether I should go now for a glimpse or hold off until I can do it justice. We're not doing a guided tour, because it's more expensive, and can be rough with a toddler.
I didn't make it to Disneyland Paris while I was in Europe and I'm having a hard time letting that happen with Tokyo Disney. At least the Paris park is considered the worst of the Disney parks...it's easier to justify skipping than the best.
Anyways, since you're already planning a trip to Japan, I think it's worth it to only do a half day at DisneySea. If Disney is a huge factor for you, then maybe it would be worth it to save up for a while longer, until you can afford to do more. Who knows? Your glimpse might make you decide that you want to return in the future.