Useful Is Not Governable
What a $460 million trading failure can teach us about AI agents
At 9:30 a.m. on August 1, 2012, the market opened. Inside Knight Capital, an automated equity router began sending orders into the market at extraordinary speed.
The system was not trying to destroy the firm. It was not malicious. It was not “intelligent” in the way people now talk about AI agents. It was software doing what its environment allowed it to do.
A botched deployment had activated defective code in Knight’s order router. Within about 45 minutes, the system generated millions of executions across more than 150 stocks. By the time Knight stopped it, the firm had built up billions of dollars in unwanted positions and lost more than $460 million. Knight survived only after emergency financing. Its independence did not.
The most important detail is that there were warning signs before the market opened. Automated emails referenced the problematic router and identified an error, but those messages were not treated as meaningful alerts. The organization did not respond in time.
That is what makes the incident useful for thinking about AI. The problem was not simply bad code. It was a failure between the machine and the institution around it. The system could act faster than the organization could understand, constrain, or stop it.
A useful AI system is not the same thing as a governable one. Once systems move from producing outputs to taking actions, governance shifts from approving each action to designing the conditions under which action is possible.
The finance-to-AI analogy is not new. People have already drawn parallels around systemic risk, incident reporting, market-style circuit breakers, and high-frequency trading controls for AI agents. But the useful lesson from finance is not that AI needs a literal circuit breaker. It is that fast, automated, high-stakes systems need operational controls before failure.
Financial markets learned this through public mistakes.
The 2010 Flash Crash showed how automated systems can interact under stress in ways no single actor fully intends. Liquidity can disappear. Feedback loops can accelerate. Rules that make sense locally can produce instability at the market level. Later mechanisms like market-wide circuit breakers and Limit Up-Limit Down rules were not grand theories of safety. They were practical attempts to pause disorderly processes before they fed on themselves.
The 2012 Facebook IPO glitch showed a different kind of failure. Nasdaq’s system had a design limitation. Orders got stuck. Decisions were made under pressure before the root cause was fully understood. The lesson was not just that technology breaks. It was that system design, contingency planning, and decision-making break together.
Even cases that are not mainly about software, like LTCM in 1998 or Archegos in 2021, point to the same deeper pattern. Institutions can have information, models, reports, and smart people, yet still fail to control concentrated risk until it is too late. Finance is useful for AI governance because its failures repeatedly expose the gap between knowing something is risky and having a system that actually constrains it.
That gap is where AI agents will live.
A model with no tools can be wrong, and that can still matter. But an agent with access to email, databases, payments, production code, customer accounts, or external APIs can create consequences beyond speech. Risk changes when output becomes action.
In regulated markets, access is supposed to be a privileged condition. A system does not get to submit orders just because someone finds it useful. It has to pass through controls: order-size limits, credit limits, capital thresholds, duplicate-order checks, position constraints, and supervisory procedures. These are engineered boundaries, not suggestions.
AI systems need the same basic mindset. An internal research assistant should not have the same permissions as an agent that can send customer communications. A coding agent that suggests a patch should not automatically be able to deploy it. A procurement agent that compares vendors should not be able to sign contracts or release payment by default.
The important distinction is between what a system can propose, what it can prepare, what it can execute with approval, and what it should not be able to do at all.
This is where many AI deployments will drift. They will begin as assistants and slowly collect powers. Someone adds a plugin. Someone connects a database. Someone automates the final click because the manual step feels inefficient. No single change feels reckless. Each one makes the system more useful. Together, they may make it less governable.
Monitoring has the same problem. Knight had signals. Nasdaq had people on calls. Archegos’s counterparties had reports showing unusual exposure. Information existed, but it did not become control.
AI systems will make this harder because they can produce a lot of plausible noise: logs, traces, confidence scores, rationales, summaries, explanations. Some of this will be useful. Some of it will be decoration. The operational question is simpler: when the system behaves strangely, who can stop it now? What are they seeing? What authority do they have? What happens to actions already in motion?
“Turn it off” sounds simple until the system is everywhere. A real shutdown plan requires knowing where the agent is running, what tools it can call, what credentials it holds, what state it has changed, and what downstream systems need to be frozen or reversed. A kill switch is not just a red button. It is a design property.
Human oversight also needs to be treated carefully. It can mean a careful expert reviewing a high-stakes recommendation with full context and the authority to reject it. It can also mean an exhausted employee clicking approve because the queue is too long and the system is usually right. Both get described as human-in-the-loop. Only one deserves the name.
Finance clarifies this because in fast markets, humans cannot approve every order. Meaningful human control often happens before and around execution: setting limits, approving strategies, monitoring exceptions, responding to alerts, reviewing incidents, and changing the system after failures. AI governance should be just as precise. The question is not whether a human is somewhere near the process, but what function that human actually serves.
For some AI systems, approval of every action makes sense. For others, it becomes theater. If a customer-service agent drafts a refund email, review may be manageable. If a security agent triages thousands of alerts, the human probably cannot inspect every step. If a coding agent proposes a production change, review should focus on diffs, tests, permissions, and deployment gates rather than the vague reassurance that “a human looked at it.”
Where the finance analogy breaks
The financial-market analogy is useful, but it has limits. Ignoring those limits would be a mistake.
The first difference is generality. A trading algorithm usually operates in a bounded environment. It submits orders, cancels orders, reads market data, and manages positions. An AI agent may write code, search documents, communicate with people, call tools, reason across domains, and adapt its behavior based on context. That broader action space makes simple limits harder. In trading, a position limit has a clear meaning. For an AI agent, the equivalent depends on the setting: external messages, API calls, spending, sensitive data access, irreversible actions, or something else entirely.
The second difference is opacity. Many automated trading systems are complex, but their objectives and outputs are still fairly legible. Did the system buy or sell? At what price? In what quantity? Was the position within limits? AI systems can fail in more ambiguous ways. They can follow instructions in an unintended way, use tools correctly while pursuing the wrong goal, or give a plausible explanation for a bad action. Logs help, but they may not fully explain why a system behaved as it did.
The third difference is stability. Financial markets change, but they are structured environments with recurring rules, instruments, and data types. AI agents may be deployed into open-ended human environments where users phrase requests differently, combine tools in new ways, or introduce adversarial prompts. Their behavior can also shift when the underlying model changes, a tool is added, a retrieval system is updated, or a vendor modifies a product. Approval at one point in time may not be enough because the system being approved may not stay the same.
These differences do not make the finance analogy useless. They make it dangerous to apply lazily. The right question is not “what is the AI version of a circuit breaker?” The right question is when a circuit-breaker-like mechanism actually reduces risk for a specific AI system. Sometimes the answer will be a rollback mechanism. Sometimes it will be rate limits, tool permissioning, mandatory approval, or a decision not to deploy the system in that context.
The strongest counterargument is that finance may be the wrong model. Markets are unusually measurable. Prices update continuously. Trades are logged. Failures show up in dollars. AI systems may operate in domains where the harm is slow, diffuse, and hard to price. A bad benefits decision, a biased hiring screen, a misleading medical summary, or a flawed legal recommendation may not look like a market crash. There may be no obvious “price band” to trip.
That objection is right, but it does not defeat the analogy. It shows how the analogy should be used. Finance is not a master template for AI governance. It is a discipline of questions: what can the system do, where are the hard limits, who owns those limits, what is logged, what happens when something goes wrong, and which failures are unacceptable even if they are rare?
Regulation SCI is useful here because it does not pretend that resilient markets come from one magic switch. It focuses on systems integrity, corrective action, reporting, reviews, business continuity, and accountability for important market infrastructure. AI governance needs a similar control stack: permissions, hard limits, monitoring, escalation, incident learning, and clear ownership of the deployment decision. Without that, governance becomes hope with a dashboard.
Knight Capital is often told as a software deployment horror story. That is true, but incomplete. The deeper warning is that catastrophic failures do not require every layer to fail dramatically. Sometimes the code is stale, the deployment is inconsistent, the alert is not treated as an alert, the risk control is connected to the wrong thing, and the procedure exists only on paper. Each flaw looks survivable on its own. Together, they create autonomy without governability.
AI agents will create many chances for this kind of failure. A model update changes behavior. A tool permission is broader than intended. A human approval step becomes automatic. A monitoring system captures traces nobody reviews. A rollback plan stops the model but not the actions already triggered. Each change feels small because each one makes the system more useful.
The danger is not one machine. It is the architecture around it: the bridges, stairs, shadows, and hidden passages that make the whole structure difficult to see from inside.
The goal is not to make AI systems timid. Institutions adopt AI because they want capability, speed, leverage, and scale. Those are real benefits. Finance did not respond to automation by abandoning electronic markets. It built controls around the fact that electronic markets were here to stay.
AI governance should be equally practical. The next frontier is not only smarter models. It is systems whose autonomy is conditional, observable, interruptible, and owned.
Useful systems can impress in a demo. Governable systems can survive contact with reality.