High availability and operational resilience are the goals of every CIO. They know that hardware fails, networks go down, software has bugs, and people make mistakes.
Strategies are put in place to mitigate against such incidents, but there are times when IT operations management is betrayed by the very systems that are supposed to help keep technology and services running. That point was made dramatically in July of 2024, when a respected cybersecurity vendor pushed a routine update to its subscribers. Unfortunately, that update contained an error, resulting in a data mismatch that caused more than 8 million systems to crash. Operations at hospitals, banks, municipal services, airlines, and other organizations were disrupted for more than 12 hours, and by the time the error was diagnosed, and fixed the interruption cost subscribers nearly $6 billion in lost revenue.
The story made headlines because of its scope and steep price tag, but it is hardly an isolated incident. Configuration and patching errors are routine in IT operations, persistent headaches for CIOs, and a constant risk to service availability. The paradox for IT operations managers is that the automations that help them keep pace with and efficiently run their vast and complex technology infrastructures also threaten them with catastrophe when they propagate errors.
Certainly, automation is a key aspect of maintaining system and service availability, but it can run counter to the goal of maximum uptime without an awareness of the risk and the right architecture in place.
When there are errors in patches, bug fixes, and routine maintenance updates intended to keep infrastructure running at peak efficiency, those automations cause errors to spread rapidly, crashing mission-critical workloads and causing systems and services to go offline.
Overcoming the High-Availability Automation Paradox
We know how automation works in support of fast, efficient IT operations, including the use of clusters to ensure service availability and rapid recovery, and we’ve seen how automation can undermine high availability by propagating errors across systems, especially during patching and maintenance exercises. How do we also use automation to minimize the risk of error propagation and maximize availability during unexpected incidents and planned maintenance? There are three basic steps to take:
- Identify and eliminate single points of failure through redundancy.
- Establish software-based clusters with synchronous and asynchronous replication.
- Automate failover and switchover processes with zero or near-zero data loss.
Let’s dig a little deeper into how to evaluate your infrastructure and build a solution that ensures high availability and rapid recovery.
Are Five Nines Really Necessary?
When it comes to high availability, it doesn’t take long for the conversation to turn to a question of how many “nines of availability” the enterprise needs. After all, 99% uptime sounds good until you realize those two nines mean more than 3 days of downtime each year. Three nines, however, is just under 9 hours of downtime, four nines equals about 45 minutes, and five nines is about 5 minutes. That’s a big improvement, but the cost of those 40 minutes, and each additional 9, can be substantial in both dollars and complexity. Failures happen, but not every enterprise needs five nines of availability.
What every enterprise does need is fast, predictable recovery when downtime hits. And that’s where you get the most bang for your high-availability buck.
Designing Recovery for Your Infrastructure
Because every organization’s IT estate is different (and changing), every solution for achieving high availability and rapid recovery needs to be architected to meet the needs of the enterprise it is intended to support. The needs of an organization operating from a single campus with its own data center and some cloud assets are vastly different from a geographically distributed organization with heavy cloud reliance.
No matter how large or complex, whether on-prem or in the cloud, it’s vital to identify all single points of failure that put operations at risk. Physical assets you own, virtual assets on someone else’s infrastructure, software, cloud services, data flow chokepoints, databases, load balancers, DNS, active directory, or anything else that—should it fail—would disrupt operations and make critical workflows unavailable must be accounted for.
Once you’ve taken an inventory of these single points of failure, you can build the redundancy needed to mitigate the risk of failure and create resilience. And then you can approach the task of architecting for fast, automatic recovery in preparation for when failures occur.