Tuesday/ saving healthcare.gov

TIME magazine’s latest issue features a cover story called ‘Code Red’ that describes how the Obamacare website was saved.  I read it with interest – after all, I have been on the inside of oh, a dozen or so large SAP implemenation project go-lives.  To be sure, with SAP as the core solution, one works with tried-and-tested and packaged software,  and not with program code that’s written from scratch.  Still, other things can go wrong : the hardware could be inadequate and not handle the onslaught of new users, or massive errors in the data migration could be discovered late, after go-live.

In the case of the healthcare.gov website though, after $300 million had been spent on the project through its Oct 1 go-live, the rescue team contemplated on Oct 17 if they should kill everything that had been done, and just start over.  There was no dashboard for the website, so it was impossible to track its performance, the number of concurrent users, and other critical parameters.  Frequently used parts of the enormous database had not been cached (copied and stored off-line), a fundamental design flaw that hit the performance of the server with every new user logging on.   They decided they could salvage most of it. Facing extreme time pressures, they had to do hot fixes (releasing new code while the web-site was up and running), and make software and hardware changes at the same time. (Under normal conditions a bad practice, since one then doesn’t know what the root cause of a new problem or error is).

Anyway, even though some challenges remain, the article says that the website now works. It ends with the fundamental long-term question, though ‘Will Obamacare work?’ (Will enough people sign up?  Will the healthcare industry players be able to adjust to the new law?)

Code Red article sm
A page from TIME magazine’s cover story on saving the healthcare.gov website. Check out the ‘Rules’ for the A-team, as defined by Mikey Dickerson. His title at Google is ‘site-reliability engineer’. I love Rule No 2!

Leave a Reply

Your email address will not be published. Required fields are marked *