Solving AI risk is more about people and organizations than engineering, and it’s not an imagined problem for tomorrow.
Washington's argument about artificial intelligence has settled into a familiar shape. According to the Foreign Policy article, Anthropic chief executive Dario Amodei has called for slowing frontier development, OpenAI's Sam Altman warns that humanity could lose control of the technology, and President Donald Trump has dismissed calls for restrictions as a conspiracy. The debate, the authors write, centres on guardrails for what laboratories build and on doomsday scenarios that capture the public imagination.
That framing, they argue, distracts from the complicated reasons real-world accidents occur. Accidents tend to emerge from the interaction of people, organisations and technologies, and the instinct outside the AI field is to blame the individual operator rather than the system around them. The article cites airline pilots who trust autopilot over their instruments, drivers who follow navigation software onto closed roads, and the 2003 case in Iraq in which Patriot missile systems classified friendly aircraft as incoming missiles and operators approved engagements that killed British and American troops. The authors point to a reported near-miss involving a Chinese ship, in which an analyst reportedly used AI to query intelligence reports and package the findings without disclosing that use, as evidence that the risk is present rather than hypothetical.
The organisational backdrop is the Artificial Intelligence Acceleration Strategy released in January by US Defense Secretary Pete Hegseth, which is designed to put AI tools in the hands of three million military and civilian personnel at every classification level. The authors accept that scaling adoption is essential to American military power, but note that the strategy is conspicuously missing a training component to ensure commanders and operators understand the advantages and risks of the systems they use.
Their first proposed remedy is procedural: standard operating procedures for managing AI outputs. Any finished intelligence product that used a generative AI tool should say so visibly to every reader, and AI inferences behind high-risk decisions should require confirmation beyond the model. Boarding a Chinese vessel during an ongoing war would have been extraordinarily risky, they note, and speed that produces operational failure is not useful. One additional check, in their view, need not slow matters substantially.
The second is training. In co-authored research, the authors found that people with the least experience of AI models were the most likely to defer to them, a version of the Dunning-Kruger effect applied to machines. More recent work found automation bias among West Point cadets in 3.9 per cent of trials, against 9 per cent in a matched sample of the American public, suggesting that military education can produce personnel who interrogate algorithmic outputs rather than merely accept them.
The authors' conclusion is that automation bias will not be engineered away by better models, because the failure lies in how people use them. Solving AI risk, they argue, is more about people and organisations than engineering, and it is not an imagined problem for tomorrow. The piece is an opinion column by Michael C. Horowitz of the University of Pennsylvania and Lauren Kahn of Georgetown's Center for Security and Emerging Technology, and its claims rest on their own research and on reported incidents whose full details are not set out in the article.
