The term appropriate reliance has become popular in AI literacy circles. In the context of AI, it refers to a user’s ability to accurately judge the accuracy or applicability of a model’s prediction whether that output is generated text or a recommended action.
The focus on appropriate reliance comes about amidst rising concern over our innate tendency to defer to the judgments of AI systems, right or wrong. Such overreliance or misplaced trust is due in part to misunderstandings about how AI/analytics systems work and in part to purposefully misleading design and marketing.
LLMs such as ChatGPT, Claude, or Gemini are especially disorienting given the resonance of natural language. Countering our natural tendency to anthropomorphize and automation bias are elemental challenges for AI governance and literacy.
The concept of appropriate reliance also has broader applicability within AI, analytics, and data governance programs. Namely, as a measure of how well your operating model is, well, operating. The single hardest element of governance is getting the decision-making framework right. If decision rights are too restrictive, those doing the work have no agency. This results in stodgy, top-heavy business practices. Or, more likely, folks simply doing what they feel they need to do via tricky end runs or by downplaying issues, if they raise them at all. Alternatively, decision rights that are too loose can reward short-term fixes over long-term value, resulting in inefficiencies and duplication as well as heightened risk. Finding the right balance is surprisingly tricky.
Designing your decision-making framework through the lens of appropriate reliance provides an alternate method of finding that balance. A healthy operational governance program is arguably good at one thing: distributing decision-making throughout the organization. This should, optimally, enable decision-making as close to the point of action or impact as possible.
As such, evaluating the level of independence and interdependence between roles is a simple yet nuanced method to right-size governance decision making. Appropriate reliance values both independence and interdependence. It emphasizes what can be known singularly and what cannot or should not. Discovering (aka asking) what decisions different stakeholders make and don’t make is empowering. But going beyond asking what decisions are being made by whom is required. Such inquiries must emphasize not just what but why decisions are—or are not—being made in the manner they are.
For each stakeholder group, include questions such as:
- What decisions do you believe you should be able to make?
- What decisions do you believe you should not make? Why or why not?
- What decisions are you currently asked to make that you are not comfortable making?
- For each: How does having or not having that authority impact your work?
The assessment process here is not to merely codify existing decision-making chains although that may be a logical outcome.
The objective is to identify what can be done to more evenly and appropriately distribute decision making between those directing and those doing the work. This means thinking collectively about the root cause of why decisions are or are not made with confidence at varying levels. In some cases, this may be due to a misalignment of decision with authority or expertise. In others, it may be a lack of simple supports or guidance. To that end, we must also ask:
- Who should make this decision and why?
- What other information would help them make this decision confidently?
And, to avoid the inevitable “if we can’t answer it simply, simply escalate it” dilemma:
- How often are issues escalated?
- What is the purpose of this escalation?
- Does this escalation serve or protect a business need?
- What percentage of escalations originate from similar root causes?
- Can this escalation be eliminated with additional guidance, standards, or other support?
The questions posed above are intended as an illustrative starting point, they are not all inclusive. The questions used while designing your program will also differ slightly from those posed during ongoing tune-ups. (Emphasis on ongoing. Your organization and the business environment you operate in are not static. Your governance operating model must also change with the times.) That small digression—which I never fail to take—aside, viewing your decision-making framework through the lens of appropriate reliance offers a tantalizing new way to clarify and sharpen your governance focus.