On our Baltic cruise, we were required to take our passports with us at most ports. On one of our Med cruises, we were required to take our passports with us in Athens and then had to turn passports in to be held when we next went to Turkey, then picked them up after returning to the ship. Our next port was Mykonos, and I don't recall needing the passport when we got off there. Apparently, turning the passport in for our Turkey stop had to do with going to Turkey in between two stops in Greece - why, I can't say. Basically, our practice is to only take them when required. I remember having to turn in passports to be held at one port on another Med cruise, but I don't recall which one.
IMO, you're more likely to have your passport pickpocketed or otherwise stolen than miss the ship and not be able to retrieve it from the port agent. And definitely if we're staying in a foreign country, we never take the passports out of the safe until we're heading for the airport to go home.