At this point, there is no question that the Healthcare.gov site has failed in part due to insufficient testing of the integrated parts. This is technology 101 – test your systems in real world situations prior to release, and you’ll eliminate the embarrassment of downtime and a poor user experience.
Case in point: As integration test manager in the Air Force, one of my first lessons and assignments was for a communication satellite. Geosynchronous satellites are placed at an orbit where their rotation matches that of the earth’s, so they appear stationary from a point on the ground. These satellites are placed 22,236 miles above sea level making it impossible to reach them for in-place repairs. I learned, early on, the importance of testing and the skills required to design these satellites to last 10 or more years without problems – experience that I brought to HOSTING for effective solutions.
Our primary contractor for the communication satellite was Lockheed but the actual components were built by dozens, probably even hundreds, of contractors. The government wouldn’t dare to be responsible for the integrated testing of all these components. If they did so they would be right in the middle of a bunch of finger-pointing (sound familiar?). More importantly they simply were not set up to do the job well. Lockheed had tested satellites and spacecraft at the systems level since the beginning of the space program and had men and women with decades of experience and skills to do the job right – the perfect partner.
The second lesson I learned was that you have to test all the time and at all levels: the component level, the device level, the subsystem level and the full system level. You test every information pathway possible in the entire satellite, and we literally tested the satellite system for years. Finally, you need to take your testing to real-world scenarios. I even took my satellite components to laboratories and put them in vacuum chambers and zapped them with cosmic rays to simulate the dangers of space. At HOSTING, we apply real-world scenarios for mission critical websites and support in terms of load-testing where we simulate the load of “X” number of simultaneous users coming into the site. The key is that “X” should be what you would reasonably expect plus a margin for additional load.
Coupling my experience in the Air Force and how the technology and cloud markets test, I firmly believe the mistakes with the healthcare.gov website could have easily been avoided. Stay tuned for some additional insight by my team on best practices for effectively supporting mission-critical websites and applications.