November Meeting Review: Loretta Morgan
warned LSC: "Stuff will happen...be prepared"

by Jim Korth, PR Committee member

Fifty million electric customers in the Northeast U.S. are without power for many hours with authorities initially clueless about the cause. In a separate incident, Bank of America’s automated teller machines crash and remain down for three days. In yet a third major snafu, backup generators fail to activate at the large 365 Main data center in San Francisco, bringing down Craigslist and other major Internet sites. In each case, poor software change management played a central role in the incident.

In her career, Lorretta Morgan has seen many such disasters caused by failures of process discipline and poor implementation. Her presentation, “IT Application Lifecycle from a Change Management Perspective," dealt with the need for organizational rigor and methodology when planning for, testing, and rolling application changes into production in an enterprise setting.

“Change affects every aspect of the software development lifecycle including documentation,” Ms. Morgan said before the meeting. “Managing change in the application lifecycle reduces documentation failure and is critical to successful IT Service Delivery.” She defined Application Lifecycle Management as “the process of delivering software as a continuously repeating cycle of inter-related steps.”

Ms. Morgan is a proponent of the Information Technology Infrastructure Library (ITIL), a set of concepts and techniques for managing technology infrastructure, development, and operations. ITIL has evolved partly in response to the need for large organizations to eliminate the risks that lead to the types of disasters she described. These kinds of cascade failures are common in complex, tightly coupled systems and are often identified and then promptly dismissed as being "nearly impossible." Unfortunately, the impossible can become reality.

Ms. Morgan and ITIL advocate setting up specific test environments that are managed to ensure that applications and their component changes are properly validated and tested before release. The goal is to avoid installing software into an organization’s production environment before it has been fully stress tested in a lab setting that closely mimics the production environment. The basic idea is to avoid surprises and the resulting down time and business embarrassment. No software is rolled into production before it has been subject to well-defined change management procedures. Back out plans must be part of the process to roll the environment back in the event a change has unintended consequences.

ITIL has now earned international recognition by the business process community and is widely accepted by most large enterprises. Microsoft used ITIL during preparations for the year 2000 and the Microsoft Operations Framework was the result of that effort. ITIL has evolved into a cohesive set of best practices drawn from the public and private sectors internationally. ITIL’s goal for change management is that any change to a production environment has been tested, approved, communicated, validated, assessed for potential risks, and finally, documented at every step. Ms. Morgan compared the application and documentation lifecycles and found them similar. Each requires identification, preparation, validation, revision, and publication.

major objective for all professionals in corporate IT is to prepare for the big failure that everyone says could never happen. The Y2K situation and its potential for disaster was understood and planned for well. Nothing happened. It is the unforeseen combination of events and risks that we must be ready for. Companies must be prepared for rapid recovery around whatever unfathomable event may occur. But as Ms. Morgan cautioned, even with the best possible management of application lifecycle, stuff will happen.