As for the 4 adults and one child situation, I'd be surprised if a CM gave or accepted two adults and a child with a RS pass. Clearly not the intent, and I could see them saying no without hesitation.
Why would the fact that it's two adults and a child be an issue? I'm really confused by this. Are you saying no more than one adult should ever be allowed to stay back with a child? We never had an issue with this on our last trip as a party of 4 adults, 1 tall enough child and 1 too short toddler. 2 adults would ride with the child using fastpass while the other 2 adults stayed back with the toddler and did a different ride. Then those 2 adults could ride the first ride later with the rider swap. I would imagine many people do it like this when they travel as two couples or parents and grandparents with a too short kid. It makes it easier having a second set of hands to help out with the little one(s).
There's a simple solution to this -- don't get FPs for rides your whole party can't use. If you're worried about the 3-year-old being able to maximize their FP, then only get FPs for rides the whole family will enjoy. There are plenty of them. Then use the standby line for your RSs. You still have one parent a d child set waiting in line, but that's better than both of you having to do it. That way the 3-year-old gets to have fun with the whole family, and no other family is penalized by losing time to people who combine FP with RS.
it's not as though not having a FP for a ride negates ones ability to ride it. So this way there's no potential confusion at all with RS, the young kid gets just as many FPs as any and every other guest.
The problem with this is, especially for the really long standby lines (which are most often going to be the ones with height requirements that would necessitate a rider swap), it's not really going to be realistic for families. Most families choose to fastpass over standby these lines so they don't have to be separated from the rest of their family for hours at a time while someone is in line. Family time is valuable, especially at Disney. Look at something like FOP which can have 3+ hr standby lines. Then for the second parent to ride with the rider swap, they could still be gone for another 30-45 mins. That's 3.5+ hrs separated just to ride one ride. With using fastpasses in 2 groups on our last trip, it took about an hour and 15 mins for everybody to ride FOP. So in the standby scenario, no, the toddler is not getting to have fun with the whole family when they are separated for half the day just trying to ride one ride. Why shouldn't families be able to use fastpasses for the rides that generally have long standby queues just like every other person who tours the parks wants to do? By getting a rider swap for the person who has to stay behind, they're still having to wait in the fastpass line twice as long as any other person with a fastpass in order to get both parents through.
Sure, the family could just forego all height-restricted rides, but the entire purpose of rider swap was so families could have the opportunity to ride things and not have to essentially wait twice. If a family uses the standby line and rider swap, it still takes longer for the whole party to ride than for a group of people who don't need the swap. The same is true for fastpass and rider swap. It still takes longer for the whole group to ride than for a group utilizing fastpass without rider swap.
In regards to others being penalized by losing time to people who use fastpass and rider swap, how exactly is this happening? How is this making your wait any longer than if all of them used fastpass without rider swap or someone did standby and got a riderswap for the other person? Either way, the same number of people are going on the ride. No matter what, it takes them longer to ride, but I don't see how it's impacting your wait.