Trust and trustworthiness are two concepts receiving increased focus as narratives around generative AI (GenAI) and agentic AI continue to dominate tech headlines and corporate aspirations. But what are they really about, and why is governance the connective tissue between the two?
Strip away the fancy definitions and metrics, and the delta between trust and trustworthiness is a matter of perception. How does party A believe party B will act (trust), and will party B act accordingly (trustworthiness)? This is not to say governance is about PR or ethics-washing, green-washing, or pick-your-pill washing. (Although it can be, and sadly has been, leveraged as such.) Rather, that our perception—whether of a person, an organization, or a tool—influences how we engage with or behave toward them/it. Therefore, ensuring that our perceptions are appropriately vested is the core of governance. This dynamic plays out at every level of the governance stack.
Consider business or corporate governance, which asks this: How do we want to show up in the world? And for whom? What matters to us? What matters to them? Are the actions we take, products we build, and services we deliver in alignment with that aspiration? Most importantly, do they think that is true? (They as in our employees, customers, shareholders, and/or those we serve.) It is that last question—the delta between what we and they perceive—that is the true purview of governance. It is the gap between trust and trustworthiness. Understanding the nature of that gap and accounting for the variance in expectations and tolerance for misadventure (aka error) is the core of business governance.
A similar construct appears at the level of analytics governance, also known as model or AI governance, depending on the context. To be sure, the advent of multimodal and large language models (LLMs) has heightened the gap between our perception of a model’s capabilities and the reality. I would argue that breaking down the hype is important in and of itself. But it is also true that how the system is perceived is a practical issue at hand.
Someone who believes an LLM is inherently agentic, in the most rigorous sense of the word, is unlikely to invest in guardrails to ensure these all-too-fallible systems are appropriately robust, resilient, and responsive to failure. Furthermore, they may see such design strictures as not just unnecessary but as undermining the system’s scope. If a user believes the output of an LLM is factual because it just sounds so darn right, then they are less likely to critically interrogate the results. None of that is a dig at the user. We confront the same predilections with simpler, so-called traditional analytics models as well.
When an old-school machine learning model makes a prediction, how is it be perceived? As an absolute truth, a possible but not predetermined outcome, or merely food for thought? How confident is the user in making that determination, and is their confidence justified? Do they understand the logical mechanics of the model itself? Have they calibrated their decision making to align with the degree of uncertainty inherent in the output? The questions are many, but the core problem is the same: perception.
Ditto for data governance. We believe dataset A tells us something about problem B. But does it? Under what conditions is this a reasonable representation? When is it not? Is it comprehensive? Does it include all the required cohorts or subject populations? Does this data reflect real world realities? When? How?
And last, but not least, operational governance: Is the system delivering the outcomes we expected? Not are the model outputs what we expect, but is it delivering the experience, sales, efficiency expected? Is our perception of the system’s usefulness shared by the users themselves? Is our perception of why users engage, and will continue to engage, with us (they love our brand, product, etc.) in line with the user’s motivations (any of the prior and/or only hanging on until a better option arrives)?
All of which is to say that to govern is to manage perception, which is not a denigration of the activity. Governance is complex precisely because perception drives behavior at every level. The mechanisms we employ—such as quantifying data quality and deliberately injecting friction into a user’s workflow—work together to calibrate trust with trustworthiness. All of this is done to answer one simple question: Are we using the right technique with the right data to solve the right problem in a manner that will create the right business outcomes? The answer, it turns out, is a complex matter of perception.