I agree totally, I'm afraid. Yes, there could be plenty of people out there who are in the incubation period without knowing but I couldn't
knowingly go somewhere with the knowledge that I could potentially cause a child to become ill or endanger a pregnant woman and her child. You were understandably upset that the child who passed it on continued to go to nursery despite knowing they had chickenpox but you'd potentially be doing the same to someone else.
I know it's a rotten thought to have to cancel your holiday but for me, I wouldn't risk anything other than that option. At the very least your little 'un will be ill and miserable which isn't fair on him or any of the family as you won't be able to enjoy your holiday. At the worst, someone who has a weakened immune system or is pregnant becomes seriously ill. I just couldn't settle and enjoy my holiday knowing that risk is there.
I know it's probably not what you want to hear and that you'll probably go anyway but just my tuppence worth