If you work in the IT Service Management industry, and even if you don't, the late-February Amazon Web Services (AWS) S3 "event" won't have escaped your attention. Where a human error - a typo while members of the S3 team were debugging a billing system - caused other Amazon Web Services to stop functioning plus the services of customer that rely on them. For many, hopefully it was a timely reminder of the need for formalized change management activities in IT.
And perhaps we need that reminder right now. Why? Because the growing interest in, and adoption of DevOps while warranted, might be blinkering people to the origins and necessity of the ITIL-espoused change management best practice. This change management best practice is ultimately there to protect the business and its operations.
ITIL Change Management isn't a Bad Thing
Or at least it was created with good intentions. With the origins of change management best practice reflecting the need to protect both IT and business operations from the adverse effects of one or more of:
- Unauthorized changes
- Badly planned and/or designed changes
- Failed changes (which can still happen with effective change management).
For many organizations, change - well, failed change - continues to be a major cause of business-affecting incidents. Not only making change a common source of customer and employee "pain" but also exposing the organization to lost productivity and revenues, rework costs, and the potential for negative brand impact if the issue is very public (as was the AWS S3 "event").
Change management also aids corporate governance and compliance (internal and external compliance) efforts. And the lack of formalized change management can contribute to a number of compliance stumping blocks, in terms of lacking adequate internal controls in areas such as:
- Standardization and the application of appropriate rigor - is there an agreed process for tracking and managing the progress of critical changes in particular?
- Business impact assessment - is change impact sufficiently assessed outside of the immediate change area, including the wider business impact?
- Appropriate authorization - are changes considered, approved, and prioritized by appropriate personnel?
- Segregation of duties - it's an auditor favorite, is there an appropriate segregation of responsibilities between the key stages of the change management process?
- Suitable audit trails - is everything adequately documented with associated audit trails that can be easily accessed when need?
The Issues with ITIL Change Management
However, in 2017, this might all sound a little draconian in light of what you can read about, and might already be doing, related to the ease of DevOps' continuous integration, continuous delivery, and continuous deployment. But ITIL-espoused change management has played its part - and a big part - in protecting the business and still can. It's all about identifying the associated risks and applying sensible, and acceptable controls (and at this point I need to admit to being an internal auditor for five years at the outset of my work life journey).
That change management needs to be sufficient to optimize the risks related to:
- Change-related incidents (and their impact)
- Unauthorized changes (and their impact)
- "Rush jobs" to deploy changes, including "emergency" changes. Then "rush jobs" to deploy fixes
- The unexpected and unwanted side-effects of "correctly" deployed changes
- The inability to back out changes
- Change scheduling conflicts (and their impact)
- A lack of cross-business input to assess change impact or change priority
- The lack of change awareness, especially at the service desk
…In the context of your organization's appetite for risk, i.e. how much they are prepared to tolerate instead of increased bureaucracy, delay, and cost.
In Light of the Risks, Change Management is Still Important
The same-old benefits of change management are still there to be reaped, including:
- The financial benefits associated with minimizing change-related incidents and problems, such as: the business cost of business-critical service downtime and delays to important or mandated changes, and the IT costs of dealing with change-related incidents and problems, and backing out or deploying change fixes
- Improved visibility, control, and sometimes speed through: the communication of the schedule of changes, taking a stage-gate approach to change approval (with appropriate authority levels), and change prioritization and fast tracking (including change models and emergency changes)
- Business-wide collaboration benefits such as better risk and impact assessment, and improved change prioritization and scheduling
The important thing is to understand how to best "blend" DevOps and ITIL change management best practice for the optimal mix of velocity and risk management. For example, Jaime Spector's recent post on Business2Community.com outlines how DevOps and the ITIL CAB (change advisory board) can work together for business benefit, including the need for:
1."Integrating the DevOps tool chain and pipeline with ITSM tools. Integrating notifications, adding decision points that require an email acceptance, and working together.
2.Recognizing that changes that come via the DevOps pipeline are inherently less risky by themselves and of a high quality.
3.Focusing the CAB's efforts - understanding how change by the DevOps pipeline (deploy to production) impacts the wider business, moving from a "stop" sign to a "give way" sign."
There's a lot more advice in Jaime's article if you have time to read it.
And, as mentioned before, this isn't a prescriptive thing - it's about finding what's best, and what works best, for your organization.
So, don't throw out ITIL-espoused change management best practice baby "with the bath water." As ever with ITIL, it's a case of adopt-and-adapt to find the best way to blend DevOps and ITIL (plus anything else your organization wishes to use) to find the best solution for your organization and its management of change risk.