Setting and Meeting (RTO and RPO) Objectives
Recovery time objectives (RTO) and recovery point objectives (RPO) are important parameters to set for achieving high availability and rapid recovery. When quantifying RTO and RPO, you need to determine how long of an outage, how much cost the organization can tolerate, and how much data the organization can afford to lose before operations are critically affected.
Once those thresholds are established, RTO and RPO objectives should be set so that recovery occurs before they are met.
While the ideal RTO and RPO might be zero downtime and zero data loss, the actual answer (and it may well be zero and zero) is what drives the architecture decisions you will make. If RTO is measured in seconds, for example, you’ll need failover clustering and automation to handle incident detection and recovery processes.
And if RPO is zero, there needs to be synchronous data replication in place. But, as with most things, there are restrictions and trade-offs that determine how much you can do before impacting performance of storage throughput, for example.
Planning for Failover and Switchover
The true test of resilience, redundancy, RTO, and RPO comes when something breaks, and your failover process must move operations to a backup system or secondary resource. If you got it right, users may not even notice. But planned maintenance and patching cannot be overlooked, because that is where many outages originate. These events put your switchover process to the test, and if routine maintenance still requires downtime, your high-availability solution is not delivering its full value.
Failover is what happens when something goes wrong. Switchover is what happens when everything is working as intended, but it is time to move a workload, patch an operating system, upgrade an application, perform scheduled hardware maintenance, or prepare for work affecting an entire cloud region. During a switchover, the resource is brought offline, its dependencies are released in the proper order, and everything is then brought online on a secondary node.
Because the process is planned, each step can be completed in a controlled sequence. That may sound graceful, but scheduled maintenance can be just as disruptive as an unplanned outage if the switchover process is not properly designed and tested.
Think about Patch Tuesday, that day of the month when operating systems, hypervisors, applications, firmware—everything—need updates. Years ago, that usually meant sending out an email that said, “The system will be unavailable Saturday from midnight till 4:00 a.m.” That’s unacceptable today. Users expect 24x7 access to AWS, Microsoft 365, Netflix, and their own organization’s resources. This is accomplished through rolling updates; instead of taking everything offline, nodes are updated one at a time. Take a server out of rotation, patch it, reboot it, verify its health, then move traffic back, and go onto the next server. Whether these transactions happen unexpectedly or intentionally, they require high-availability clusters and automation.
What Are High-Availability Clusters, and How Do They Work?
A high-availability cluster is a group of independent servers, called nodes, that work together to keep an application or service available. Rather than connecting to a specific server, users connect to the clustered service. The cluster determines which node should run the workload and coordinates the resources it requires. If the active node fails, the cluster moves the workload to another node, like runners passing a baton in a relay race. Cluster software continuously monitors nodes and resources, exchanges health information, and coordinates failover when a failure occurs.
Behind the scenes, the cluster uses quorum to determine which nodes are allowed to continue operating when communication is disrupted. This prevents separate nodes from simultaneously bringing the same clustered workload or shared data online, a condition known as split brain.
By establishing which portion of the cluster has the authority to operate, quorum protects workload integrity and helps prevent a communication failure from becoming a data corruption event.
Protecting Systems, Data, and the Organization
Ultimately, clusters help ensure service availability and the rapid restoration of systems and services. However, responsibility for protecting application data depends on the cluster architecture. In a traditional, shared-storage cluster, the SAN must provide the redundancy and resilience needed to protect the data. In a SANless cluster, host-based replication maintains copies of the data on separate storage attached to different cluster nodes. In either architecture, the availability of the workload depends on both cluster orchestration and a resilient data storage strategy.
There is no one-size-fits-all solution for achieving high availability and rapid incident recovery. Getting there means understanding the automation paradox, identifying and addressing risks to service and data availability, and designing a solution that meets the specific needs of the enterprise.
Successfully architecting the right solution is within reach when the organization establishes reasonable, quantifiable objectives and selects the appropriate clustering, storage, and replication technologies.