I have a pair of Vibrams and mine are made for light hiking and such so they handle getting wet very easily (I recommend those as they have the 5 independent toes and are a much better fit). They are not water-resistant but they are perfect for walking at the parks because they dry very quickly (They recommend washing them in the shower by putting soap in them and mashing your feet around and sitting out to dry which is great when you use a great smelling body wash!)
But my only thought process is that the bottom of most of them are designed for traction, regardless of the use. If your foot slips onto the side, you run the risk of potential doing your body some damage, especially on the fiberglass slides. I know it is a worst-case scenario but think about it...you foot rubs the side and suddenly, the traction of the shoe causes you foot to "stop" and you have at the least, wrenched your ankle, or even worse, broken it...
food for thought. not trying to be a debi downer but it could happen. Maybe wear them to the ride and remove them at the bottom and put them in a locked backpack?