There is, to the best of my knowledge, no way to constrain base AI technologies in ways that preclude use by bad actors—short, of course, of putting the genie back in the bottle, which has never been a winning strategy in fantasy or real life. Nor is it possible for poor outcomes by well-intentioned individuals to be entirely prevented. We can, however, reduce the prospect of such situations. Doing so requires getting up close and comfortable with uncertainty and risk. It involves lowering the barriers to discussing uneasy topics openly, honestly, and without prevarication, and last, but not least, making it not only acceptable but expected for teams to actively seek to disprove their own theories and criticize their creations. In this regard, the single most important output of your AI (or any other) governance program may be the capacity for critical thinking.
Understanding the Risks
Deploying AI fairly, safely, and responsibly requires clarity about the risks and rewards of an imperfect solution, not the attainment of perfection. An AI algorithm will make mistakes. The error rate may be equal to or lower than that of a human. Regardless, until data perfectly representing every potential state—past, current, and future—exists, even a perfectly prescient algorithm will err. Given that neither perfect data nor perfect algorithms exist, the question isn’t whether errors will happen but instead: When, under what conditions, and at what frequency are mistakes likely?
Enter the premortem. This concept—which Steven Johnson highlights in Farsighted: How We Make the Decisions That Matter the Most—is particularly important in environments that are inherently complex, where uncertainty abounds and the stakes are high. Outside of rote task automation, this description applies to most AI solutions.
Rigorous premortems confront uncertainty head-on by forcing teams to explicitly consider questions such as the following:
- What are the real mistakes this solution could make and their quantifiable impact? In healthcare, a system could underestimate the need for nursing staff, apply an incorrect diagnosis code, or generate a false-negative diagnosis.
- Are there inherent limitations, including biases, that may be reflected in the solution? If an algorithm is trained on data from a predominantly elderly population, for example, the algorithm may not be as accurate when applied to younger cohorts.
- What other perceptions or factors may impact how the solution is received? Despite their increasing sophistication, AI chatbots are not known for their bedside manner.
- What happens next? When it comes to clinical decision making, a differential diagnosis or predicted outcome is not the end of the story. Rather, it’s the beginning of a complex, ongoing dialogue that must marry clinical insight with empathetic human understanding.
- What is the potential harm and error tolerance of parties impacted by the solution? Consider that patients report they are more inclined to forgive a mistake made by a human doctor than a machine—even if the error is one the human clinician may make more frequently. Paradoxically, patients are also receptive to and, in many cases, welcoming of AI, for instance, in robotic surgery—if they perceive a positive benefit to their long-term health.
- What are potential ramifications of this solution today and tomorrow—even in the absence of errors?
- What other options exist to solve this problem?
Mitigating Harm and Course-Correcting
Of course, merely posing these questions is not enough. Mindful debate and out-of-the-box viewpoints should be encouraged. Such deliberations can be advanced through the use of tools such as scenario planning, decision mapping, simulation, adversarial game-playing, value modeling, and storytelling.
Done well, premortems allow diverse stakeholders to consider the impact of a given AI solution in the context of what is known and unknown. This, in turn, allows informed decision making regarding both if and how a given AI solution should be implemented, and it encourages teams to preemptively eliminate or mitigate harm and rapidly course-correct before and after a solution is deployed.
So, while we often think of governance as dictating rules, we are better served if governance promotes critical thinking. Indeed, the extent to which your AI governance program creates the capacity for continuous, constructive critique may be the extent to which your AI program does more good than harm.