I have one of the first software patents ever issued, have had teams of up to 500 developers working for me at one time, and currently run development for a Fortune 25 company. So, tell me again how I don't know how software is developed or how large IT organizations are staffed?
The mere fact that the system crashes under heavy load shows us, unequivocally, that there is a serious problem. One can develop code that doesn't "break" or return incorrect responses under load. One can model the performance of code under stress. One can use those performance metrics and anticipated volume of use to spec out the amount of bandwidth and server capacity one needs to handle the volume. One can throttle down volume using any number of means (including "Please return later" messages when the number of connections to the server farm exceeds some arbitrary number).
Let me give you an example of a "fail" caused by using inexperienced developers: When the volume of traffic goes up, the site responds that there is no availability. When the volume goes down, the site responds with availability. Let's look at the parts of this problem: database, middle ware (APIs), and website. In order to prevent the database from being overwhelmed, one sets a maximum query time. When that's exhausted, the database returns an error code to the calling code. The calling code (we'll assume it's an API, because that's how modern web development is done) needs to recognize the error. It then needs to pass an intelligent response to the website code (which called it). The website code must then do something useful with it. But, let's assume that the API wasn't built understanding this condition may occur. It didn't get any results back, so it sends a "no results" response back to the website, which says "No availability". Ok, let's now give the API developers some credit and assume they understand the error and send back no results to the website code along with an error code. Now, the website developer has to decide what to do with that response. Because your most fresh-out-of-trade-school-developers are writing website code, they'll write themselves into a box. They expect a certain response, and getting no results means "No Availability". Or, they may be trapping the error, and may choose to not display a "try again later" message to the user, because they're developing ahead of the implementation of the actual API, or because their test database doesn't have data in it, or because their boss told them that "we don't display errors". Now, the code is done. It goes to testing. The regime for testing includes various standardized scripts designed to simulate the use-cases, and ordinarily also includes tests for error trapping on inputs, browser incompatibilities, use on mobile devices, and operation under heavy load. It's clear that there is insufficient testing, or the test results under heavy loads are being ignored. While it's a good idea to implement some throttling processes (such as the "system is busy, you're in the queue" recently implemented for paying dues), when the user gets on, they expect the site to work. Even with the throttle down to limit the number of simultaneous users, the recent fiasco that was paying dues shows us that they still don't have a grasp as to where their site breaks under load.
In the case of the dues payment system, it appears that they didn't bother to test at all. Virtually everyone is getting the message that their dues amounts don't equal their payment amount when the user clicks on the "pay balance" button and the user has multiple contracts. When the system is busy, the payment processing system gets overwhelmed, and the website shows you a gold dot and no error message and tells you to correct your input.
