In North America, the American Automobile Association (AAA) assists drivers when things go wrong on the road. Locked your keys in the car? Blew a tire? Crashed on a remote country road? Whether you are experiencing an intensely aggravating but minor annoyance or a cataclysmic crash, someone at AAA will answer the phone. The value isn’t that the company has a locksmith or an ambulance on speed dial. The value is in the assurance that timely, reliable help is just a phone call away.
AAA is a convenient alliteration and useful comparison for something just beginning to get due consideration in AI system development: planning, explicitly, for what to do when things go wrong. In the context of system development, this comes down to three interconnected capacities: assessment, assurance, and audit.
Assessment: Determining Required Assurances
To assess is to make a judgment as to a system’s potential or realized value, priority, fit, or fitness. Impact assessments, which study the potential effects of planned actions, are common in business and software development. What existing assessments often briefly touch on—if they address them at all—are unplanned actions, unexpected outcomes, unanticipated errors, or unseen threats. In the case of AI systems, these are varied and may come in the form of purposeful assaults perpetuated with or without AI. Or they might be unintentional but ruinous events triggered by users pursuing an expected goal in an unexpected way. The dynamism of AI systems is, after all, only outmatched by the unpredictability of human behavior.
Despite the complexity, AI system assessments boil down to three core considerations:
- Are expected outcomes assured?
- Can risks be suitably mitigated?
- What errors, if any, are tolerable?
These are simple questions without simple answers. Addressing them decisively—at a level that allows for robust process and system design—requires a deep understanding of what is in the organization’s control, as well as the humility to recognize what cannot be controlled. It mandates an accurate accounting of the capabilities and limits of the technology, the data, and the process, along with a reckoning with one’s appetite for risk and tolerance for error. It can’t merely be a passing, “We think that’s OK.” It must be a deliberate accounting of the risks you are willing to court and the mistakes you are prepared to accept, categorically, with caveats, or not at all.
The easy, upfront work in assessment is identifying desired outcomes and predictable, desired user behaviors. The hard work, and a significant gap today, comes in finding and minding the gaps between decision-making authority and accountability, business and system incentives, predicted and actual behaviors, and internal versus external threats. Nevertheless, deliberative assessment is the only path to assurance.
Assurance: Making It So
According to Cambridge Dictionary, assurance is “a promise that something is or behaves as expected.” This necessarily positions assessment as an essential precursor to assurance. Without knowing what is expected, confidently delivering a system and assuaging any doubts harbored by executives, users, and customers alike is nigh on impossible.
There is another aspect of assurance that’s neatly encapsulated in a category of insurance by the same name. Assurance here is a type of insurance that will pay an amount of money at the end of an agreed period rather than only if something unexpected happens. In regard to AI, we might flip the script a bit and define AI assurance as the mechanism(s) to deliver the expected payout, even if something unexpected happens.
So what is the hallmark of assurance? It is not whether a system conforms to expectations in a predictable, well-prescribed environment. The hallmark of assurance is resiliency: the ability for the system to respond and, if necessary, recover when challenged. As such, assurance is not about checking off a list of requirements. Instead, it’s the set of complementary, reinforcing mechanisms that promote or bolster expected system behaviors and outcomes. When this is not possible, it responds in a way that allows the business to reasonably recover when an error occurs. Importantly, whether an error or issue is caused by naive misuse or flagrantly delinquent intent is of little consequence given the speed and scale at which AI systems act today. Response management is, as Chris Marshall, VP of enterprise AI strategies at IDC, recently suggested, the new rail of risk management.
All that said, what constitutes assurance? The specific safeguards, fail-safes, and guardrails will vary by system. Assurance most certainly is not a static policy on a page. It may entail prescribed system boundaries and strictly managed access rights, deterministic rules to keep AI responses or actions firmly in bounds, and human oversight aligned with the system’s visibility, impact, and risk. In other words, clarity about which loops people are on, in, or over is necessary. Decision gates that are aligned with authority and accountability are must-haves. System designs should not present recommendations that are recommendations in name only. It’s important to design in healthy friction when deliberation—not speed—matters. Methods to monitor usage and unusual behavior originating within and without the enterprise are also essential, as are fail-safes that quickly shut the system down. And be mindful not to forget the old standbys: ongoing production monitoring and validation, robust cybersecurity, etc.
Audit: Resting Assured
To differentiate audit from assurance, I initially thought to say that audits were final tests. However, when it comes to AI systems, there is no finality. Once is never enough, since the models, data, users, and environments in which they intersect are all dynamic.
That aside, audits are formal, methodical examinations of a system executed with specific ends in mind. Audits are not system fail-safes; those get implemented through assurance. Audits are a check that fail-safes are installed and working as intended. Think of them a bit like that annoying test button on your smoke alarm, which, if you’re anything like me, you don’t hit nearly enough—OK, ever.
While audits are typically associated with regulatory compliance, forward-looking enterprises will recognize their value as integrated health checks. Importantly, they must be done by those without any relation or allegiance to the system or its owner. As it is not easy to disintermediate internal resources from overarching corporate goals or pressures, audit has typically been a third-party function. That speaks to the need for internal auditors to be expressly independent of the owners and developers of any given system. This includes the vendor providing the raw algorithmic parts such as a foundational model or the integration partner deploying it.
In addition, AI systems challenge historical auditing paradigms in two important ways. The first is via reliance on preplanned, scheduled testing (e.g., quarterly, annually, or when a predetermined trigger is tripped). The second is the execution of strictly bounded, pre-prescribed testing procedures. AI systems require both actions and then some. The dynamic nature of AI systems necessitates more frequent inspection and a mandate for auditors to bring their creativity to bear during testing. Even in the pre-AI era, talented, persistent, or unruly users found ways to confound the most scrupulously designed system. Natural language interfaces—which do not jive with prescriptive inputs—merely make this challenge more acute. The auditor’s playbook must evolve accordingly.
AAA Delivers What Matters Most: Resilience
Why does any of this matter? Unless you sit in the unfathomably privileged seat of a “frontier model” provider, your organization will be held liable for the actions of its AI systems. Liability may be regulatory, legal, reputational, or operational. Regardless of the door through which accountability comes to call, outcomes, not intention, will matter most. The triple A’s of AI (assessment, assurance, and audit)—orchestrated with deliberate intent—ensure that systems are both functional and resilient. It is resilience that, in a rapidly changing environment, may prove to be the most important feature of all.