Your AI Governance Program Can’t Stop Anything

Key quote:

A fire alarm is not a fire department. You can wire every room in your house, dutifully change the battery of every smoke detector annually, and route every alert to a beautiful dashboard — but still have your house burn down because nobody picked up the hose.

Why it matters:

It’s a slow news week because of holidays in Canada and the United States, which means it’s a good time for long thought pieces. Joseph Wallace’s latest, The Real Question to Ask About AI Governance, is a good read, but he didn’t go far enough, and it feels a smidge like a promotional piece for the good work he’s doing at Adobe.

Wallace is right that the governor is missing, but he’s less clear about why. But the reason isn’t mysterious: it’s incentives, and they’ve been working against governance since before AI was the buzzword of the decade. What got me thinking about this is that I’m working with my publisher on final proofs for my upcoming book, and I get into incentive alignment in Chapter 8.

Think about how performance metrics actually function inside most organizations, where product teams get rewarded for shipping, engineering for velocity, and sales for revenue. Nobody’s quarterly bonus depends on catching a model that’s quietly discriminating in hiring decisions or generating harmful content. The governance team that flags the problem gets called a “blocker” in the next all-hands meeting.

Wallace notes that actual decision authority sits with “someone whose primary job is shipping products and hitting revenue targets.” That’s not an accident but the natural consequence of how companies have structured their incentives. When you reward speed and penalize friction, the friction-reducers win. Governance, by definition, is friction, so governance loses.

The SolarWinds case shows where this might lead, long before anyone heard the term “vibe coding.” SolarWinds was treating security controls like a compliance checklist rather than an operational reality. The company maintained a public Security Statement claiming NIST compliance, while internally their update server password was solarwinds123. They’d implemented 21 of 325 NIST controls, and a VP warned leadership they were “in a very vulnerable state,” but nobody with authority acted. The SUNBURST attack followed, compromising 18,000 customers, including the SEC, who got rather shouty about it. We haven’t seen the vibe-coding equivalent of this disaster, but give it six months. Someone will ship an AI system based on vibes instead of operational controls. If the organizational guardrails are fiction (in favor of shipping faster), it won’t matter whether the model works perfectly.

At the board level, the picture is messy. Boards are under greater scrutiny on cybersecurity and AI, which has led some executives to complain that directors are “overstepping into management’s role.” In fact, 32% of executives say boards are overstepping, partly because of higher expectations in areas like cybersecurity and AI. That tension helps show the core problem: when boards demand accountability without understanding the mechanics, managers treat governance as theater. Neither side engages with the boring question that actually matters, which is who holds the button labelled “stop”.

So far, the instinctive answer to AI governance is to put a human in the loop, but that’s the wrong answer. Human-in-the-loop is the feel-good answer that lets everyone nod and move on. In practice, the human in the loop is often a safety driver who’s watching Netflix.

Look at the Uber autonomous vehicle fatality in Tempe, Arizona, where the AI detected the pedestrian but the human safety driver was streaming a video on her phone. The system wasn’t designed to brake autonomously for the classification it assigned. The “human in the loop” was supposed to be the failsafe, and she wasn’t.

Or look at what happened with Grok on X in January 2026, where the chatbot could be manipulated into generating non-consensual sexual images, including of minors. That capability should never have existed in production, but no amount of human review after deployment fixes a design-stage governance failure. A human reviewer who can flag the problem but can’t actually pull the plug isn’t a control; they’re a witness.

Regular readers know I’m a big fan of the Delaware Court of Chancery, and might see an out with their ruling on Marriott, when the court dismissed the derivative suit. The board had systems, yes, but those systems were imperfect. That ruling suggests courts will accept “flawed efforts” as long as there’s something in place, and that’s problematic for governance advocates, as well as vendors selling governance “solutions”. It means companies can check boxes and face no liability if catastrophe strikes, at least at the board level.

What companies need is automated governance with teeth, not dashboards or advisory councils, but technical controls that can stop production systems.

• First, define hard thresholds where an AI system automatically suspends: anomalous output rates spike, bias detection exceeds defined limits, or user harm reports hit a frequency threshold. The system shuts itself down or throttles to safe mode without requiring a human to convene a meeting.
• Second, name a human with technical kill-switch authority and a reporting line independent of product. Not advisory, not consulted, but the person who pulls the plug and reports to trust and security or the CRO, not to the VP of product. This isn’t a committee vote or a consensus process; it’s a single accountable owner. If it makes you feel better, add it to your RACI chart.
• Third, build rollback capability into every AI deployment. If you’re deploying AI agents that take real-world actions, you need the technical equivalent of an undo button, and someone with the authority to press it. Without that, you’re shipping irreversible actions into production with no way back.

The decision path for shutdown should be simple enough to explain on one slide: threshold breached, system auto-suspends, named governor reviews, and governor decides restart or permanent withdrawal. No committee, no consensus process, no three-week risk assessment. Wallace asks who in your organization can say no and have it mean something. That’s the right question. But the answer needs technical controls that execute the decision automatically, not a human who might be watching YouTube when the alarm goes off.