Some people always stay at deluxe resorts, and have experienced multiple problems with guests from moderate and value resorts who swim in whatever pool they want, causing overcrowded conditions for those who pay to stay in the deluxe resorts.
I don't think I've EVER read someone wanting to swim in the AS pools while staying at the Polynesian. It's because generally speaking, the pools a the deluxe resorts are nicer. That's one of the reasons the deluxe resorts cost more money.
I don't think that Disney should allow pool hopping, except as a perk of DVC ownership and when the main pool at a resort is closed or undergoing substantial renovations.
It's simply not fair to those who pay a lot of money to stay at resorts with higher levels of amenities to have to suffer overcrowding due to "visitors".
If someone wants to swim in a particular resorts pool they should rent a room at that resort. It's the right and fair thing to do. If you don't like the pool at the resort you are staying at, or want a better pool, no one is stopping anyone from staying at a different resort.
Anne